Evaluation of Generation XP Publications
Publication Details
The progress of the Generation XP Digital Opportunities Pilot Project is reported for the period January 2002 to December 2003. During this time, eight low decile secondary schools in Auckland and Gisborne piloted running Microsoft Office Specialist and other industry level IT courses with their senior students. The pilot proved to be a successful ‘proof of concept’ with nearly 1000 students studying the MOS courses during the two years. By the end of 2003, the evaluator had been notified of a total of 107 students passing one or more component units. Between them they gained over 200 passes.
Author(s): Michael Winter, Christchurch College of Education. Report prepared for the Ministry of Education.
Date Published: August 2005
Executive Summary
One of the research questions was to identify 'lessons which have been learned from the pilot which could usefully inform any roll out of the scheme to other schools and to other school districts'. This summary begins by listing the evaluator's suggestions regarding these lessons and suggestions. These are followed by a summary of findings and lessons categorised under headings corresponding to the main foci of the research.
Key Lessons and Suggestions
Most schools did not perceive the project as being a true ongoing partnership between schools, businesses and the Ministry. They felt 'dumped on'. True partnerships would involve ongoing cooperation and communication between the partners for the duration of the project. Schools and students would benefit from a greater awareness of the business world and its opportunities. Businesses could get greater exposure and recognition of the social good of their involvement.
In future roll outs and similar projects, volunteering by committed schools and their full involvement in the partnership is most likely to guarantee project success. It is also recommended to establish realistic and meaningful performance criteria for schools to work to, and encourage schools to establish effective monitoring and control systems for the project. Such systems could be designed by schools and other stakeholders as part of the establishment of a partnership.
At the end of 2003, the MOS training was being extended to a further 20 non pilot schools.
There were software problems especially in the initial stages of the project. These difficulties were in part caused by schools needing to integrate the machines with school systems rather than running the Generation XP machines as a stand alone facility. Set-up was delayed by the variability of schools' computer systems. Working towards common school systems, or at least creating inventories of school systems was recommended. There were also problems setting up the testing software, both in the initial stages and periodically as a result of upgrades to the testing software. Future roll outs should consider funding schools for sufficient technical support to ensure that their systems are adapted quickly to cope with upgrades.
Several schools did not find it straightforward to integrate MOS with other courses. MOS was often seen as a useful extension for more able students.
Teachers had mixed feelings about the training offered at the start of the project. Most were not satisfied with the training, regarding it as inappropriate and poorly planned. Teacher selection, initial training and ongoing professional development were seen as crucial to the success of the programme. Professional development was regarded as necessary to cope with staff attrition from the MOS courses, and to help teachers cope with technology, course and assessment upgrades. Staff training and professional development would need to be carefully targeted to the staff who teach MOS courses. Using unqualified teachers in MOS classes has proved unsuccessful and should be discouraged in future. MOS teachers would need encouragement to reflect on their performance.
Most coordinators reported dissatisfaction with the pedagogical approach of the courses and with the course books. Some prepared supplementary resources which they thought more suitable, and others experimented to make the courses more appealing. However, teachers frequently commented about lack of time to prepare for the courses. Creation of teaching resources relevant to New Zealand school students should be encouraged, as should provision of sufficient resources for the numbers of students enrolled.
Teachers in schools that made the courses compulsory were least satisfied with the courses as they taught them. There is evidence of student resistance to the compulsory nature of the courses in these schools. Making the courses compulsory had a negative effect on student candidacy and success in MOS examinations.
In order to increase numbers of Maori and Pacific students taking the courses, it is suggested that target levels are set and that local iwi and Pacific community leaders should be involved in planning roll outs of the project. The project could be located in marae and other community centres as well as schools. In order to improve accessibility of the qualifications for Maori and Pacific students, the learning environment and courses need tailoring to their needs. This could include greater teacher encouragement of students, modified course materials and teaching approaches.
There appeared to be no barrier to out of school access to computers or to MOS programs for the vast majority of the students.
Collected Findings and Lessons Related to Research Foci
Why did students chose to do the Microsoft qualifications courses?
In 2003, increased overall percentages of students cited qualifications or value of the courses to their future or careers, (39% compared with 25% in 2002) and less mentioned increased skills or knowledge as reasons for studying MOS (9% compared with 15%). Within these findings, there was considerable variation between schools. One of the leading MOS schools had a threefold increase in the percentage of students citing qualifications in 2003, and a 50% increase in those citing career or future reasons.
Two schools made the courses compulsory. This resulted in compulsion being given as the most common reason for taking the courses. In these schools in 2002 there was no mention of qualifications or future benefits as reasons, and only minimal mention in 2003.
There is evidence of student resistance to the compulsory nature of the courses in some schools.
How successful were students in obtaining the relevant skills and gaining the qualifications?
80% of students surveyed claimed competence or expertise with Microsoft Word and 65% made the same claim with Excel after the MOS courses. Students also gained good skills with Access, Power Point and Outlook.
In 2002, 71 students (10% of enrolments) passed one or more MOS examinations. The figure for 2003, notified to the evaluator late in the year, was 36 (13%).
Making the courses compulsory had a negative effect on student candidacy and success in MOS examinations.
Which groups of students tended to enrol in the courses, and gain the qualifications?
Gender distribution: In 2002, 54% of the Generation XP students were female, whilst in 2003, this percentage had increased to 61%. This is largely accounted for by a decrease in the percentage of total responses from one boys' school, and an increase in that from one girls' school.
Age distribution: The age distribution of Generation XP students in both years was similar with a mean age of 16.5 years in both years, and a standard deviation of 0.7 years in 2003 and 1.2 years in 2002.
Ethnicity: The overall percentage of European students enrolled in both years was less than the national average for 16 year old students, reflecting the diversity within the communities of the participating schools. However, within the participating schools, it appears that European students were over represented compared with Maori and Pacific students. A higher percentage of European students enrolled overall in 2003 compared with 2002. The increase coincided with a decrease in the percentage of Pacific students. Percentages of Maori and Pacific students passing the examinations were very low. Passes were dominated by European and Asian students, with Asian students performing best in proportion to their numbers within the schools.
Academic achievements: The distribution of 2001 school certificate results amongst Generation XP students in most schools was similar to the distribution amongst the school as a whole. However, in all schools, the more academically able students dominated MOS examination passes.
There appeared to be no barrier to home access to computers or to MOS programs for the vast majority of the students.
Did participation in the courses influence students' career interests and ambitions?
There was no difference in students' interest in IT as a career between the entry and exit surveys in 2002.
On entry, a higher percentage of students in 2003 responded that they were 'very interested' in IT as a career. This percentage increased noticeably in the 2003 exit survey, with a concurrent decrease in the 'interested' category.
The percentage of students who thought that IT skills were 'very important' for employment also increased during 2003.
Percentages of respondents who were 'not interested' in IT careers; or who thought that IT skills were 'not important' for employment remained substantially constant over the four surveys at between 30 and 40%.
Generation XP students showed a high degree of interest in further and tertiary education after leaving school. In all four surveys 57% or more respondents were 'very interested' in university education.
Did they seek or find employment on leaving school which used the ICT skills gained during the courses?
It proved difficult to follow up MOS students after they had left school. In the event it proved possible to interview only 17 ex-students.
Only one of the interviewees was in employment in an IT based occupation.
Thirteen were involved in tertiary study leading to computer or business qualifications.
Most interviewees reported using MOS skills for a range of purposes including preparing assignments, and personally.
Some had mentioned the MOS courses in CVs or at interview, or thought that Microsoft skills had helped them to obtain jobs or entry to tertiary study.
Students from Gisborne schools undertook a range of part time work which used their MOS skills.
Did students enrolled in the Year 12 ICT courses come back to do ICT qualifications in Year 13?
Nearly 30% of the students studying MOS courses in 2003 had been taking the courses the previous year.
Over 60% of students in each year rated Microsoft qualifications as 'very important' or 'important' in their decisions to stay on at school.
In what ways and how much did students apply learning from the Microsoft qualifications courses to their learning in other subject areas?
90% of students anticipated that MOS courses would help their learning in other subjects. After the courses, the same percentage of students claimed that MOS courses had been helpful.
Students consistently reported that they used Word for assignments, Excel for mathematics, accounting and data processing and Power Point for presentations.
Students increasingly regarded the MOS applications as useful everyday tools.
What was done during the pilot project in order to get the Microsoft Qualifications registered on the New Zealand Register of Quality Assured Qualifications and accepted for credit inclusion towards the National Certificate of Educational Achievement? How easy was it to register the Microsoft Qualifications on the New Zealand Register of Quality Assured Qualifications and to have them accepted for credit inclusion towards the National Certificate of Educational Achievement?
A registered provider was chosen to sponsor the process of registration and credit inclusion of the MOS courses.
The provider compared the MOS qualifications with qualifications which were already on the NZQA Register to establish comparability of levels.
Although the MOS qualifications presented a novel case to NZQA, being foreign owned commercial qualifications, the process of accreditation and approval followed a standard course.
The process was complicated and hence lengthened by involvement of several stakeholders in the process.
Legislation has now been changed to allow non providers to submit courses for registration.
How was the qualifications course regarded by other students in the schools, by parents and by prospective employers?
Year 11 students at Gisborne schools were knowledgeable and positive about the courses. Many Year 11 students at Auckland schools had little awareness of the courses.
Student attitudes and knowledge appeared to reflect teacher enthusiasm and publicity about the courses within the schools.
Most parents of MOS students in schools where participation was voluntary were very positive about the courses. Very few parents were very negative about them.
A small number of parents expressed misgivings about the course pedagogy.
A limited amount of data was obtained on employers' attitudes to the MOS courses. The most reliable data, from a tertiary institution which had employed some MOS students part time, was overwhelmingly positive about the students' technical and on-the-job skills. The courses had led to a 'big breadth and depth of knowledge.'
How technically reliable and robust, over time, were the ICT systems put in place by the business partners?
All schools experienced difficulties with the IBM computers during the project. These difficulties were caused by a batch of faulty hard drives and by overheating.
There were software problems especially in the initial stages of the project. These difficulties were in part caused by schools needing to integrate the machines with school systems rather than running the Generation XP machines as a stand alone facility.
Set up was delayed by the variability of schools' computer systems. Working towards common school systems, or at least creating inventories of school systems was recommended.
There were also problems setting up the testing software, both in the initial stages and periodically as a result of upgrades to the testing software.
Infrastructural problems were much less of an issue in 2003 than in 2002.
The NZ agent for the MOS qualifications developed a trouble free procedure for getting new centres established quickly to run MOS courses.
Most schools worked to integrate the MOS computers into the wider school network in order to make fullest use of them.
The computers in all schools were in almost constant use throughout the school day. Apart from the MOS courses, they were used for a wide variety of applications including other ICT courses, web based research, art and design, Power Point presentations, word processing reports and spread sheeting for mathematics and science. Teachers at one school used them for writing student reports.
Some schools made the computers available outside classroom hours, one running an internet café for students. Other schools made the resources available to the wider community.
The Generation XP website was used in the early stages of the project to pass on information from the lead business partner to schools and to post periodic reports from schools.
After the set up, the web site was not used as a communication tool or resource for the project.
To what extent, and with what effects, were the schools able to integrate or separate their teaching of the Microsoft Qualifications with the teaching of other specialist ICT courses for other qualifications?
Three of the eight pilot schools, all single sex, taught the MOS material in stand alone courses. Two of these schools made the courses compulsory, and had among the lowest pass rates of the pilot schools.
The remaining schools integrated MOS teaching with that of other ICT based courses.
Most of these schools did not find it straightforward to integrate MOS with other courses. MOS was often seen as a useful extension for more able students.
Teachers commented on the difference in pedagogy and requirements between the MOS qualifications and the common New Zealand qualifications.
Teachers had mixed feelings about the training offered at the start of the project. Most were not satisfied with the training, regarding it as inappropriate and poorly planned.
Teachers appreciated the opportunity to attend the Navcon ICT conference in Melbourne.
Ongoing professional development was considered necessary, especially to cope with staff attrition from the MOS courses.
Professional development provided by Computer Press during visits to schools was seen as very helpful.
Teachers in schools that made the courses compulsory were least satisfied.
Most coordinators reported dissatisfaction with the pedagogical approach of the courses and with the course books. Some prepared supplementary resources which they thought more suitable, and others experimented to make the courses more appealing.
Teachers frequently commented about lack of time to prepare for the courses.
Schools sent staff, including non-Generation XP staff on the initial training course. This had the effect of raising the ICT profile within schools.
A few teachers gained MOS examination passes, and others studied the courses in 2002. This professional development was constrained by time pressures.
In 2003, little use was made of MOS courses for professional development in the schools.
One school planned to incorporate MOS and IC 3 in staff training in 2004.
Supplementary Research Question. In view of concerns expressed by MOS teachers concerning readability of the MOS 'Step by Step' course books, how does the readability of these resources compare with other MOS texts and with New Zealand produced ICT resources?
The Step by Step resources provided to support the programme were found to have reading ages up to four years higher than those of texts commonly used in New Zealand schools with this age group.
A new resource is being produced locally, by a respected author of computer teaching resources, to support teaching of MOS courses in New Zealand schools.
Schools and businesses regarded the programme as being sustainable in the future.
The fact that the courses had achieved NZQA approval and credit inclusion was regarded as important.
Many of schools' concerns were addressed with the Ministry's Memorandum of Understanding regarding continued support in 2004.
Schools continued to voice concerns regarding ongoing costs for maintenance, upgrading and being an accredited centre.
Examination costs even at subsidized rates may prove to be a barrier for students.
At the end of 2003, the MOS training was being extended to a further 20 schools which were not part of the pilot project.
What lessons were learned from the pilot which could usefully inform any roll out of the scheme out to other schools and other school districts?
In future roll outs and similar projects, identification of committed schools and their full involvement in the partnership is more likely to guarantee project success.
It will be helpful to establish realistic and meaningful performance criteria for schools to work to, and encourage schools to establish effective monitoring and control systems.
Teacher selection, initial and ongoing training was seen as crucial to the success of the programme. Teachers need encouragement to reflect on their performance. Using unqualified teachers in MOS classes was not successful and should be discouraged.
If the aim is to be to improve the qualifications of Maori and Pacific students, the environment and courses would need tailoring to their needs. This could include greater teacher encouragement, modified course materials and teaching approaches.
Courses and presentation would need to be adjusted to be attractive to Maori and Pacific students. This could be difficult given the nature of the content and assessment regime.
In order to increase the numbers of Maori and Pacific students taking the courses, it is suggested that target levels be set.
Local iwi and Pacific community leaders should be involved in planning roll outs of the project.
Publicising the courses among the target communities was seen as vital.
The project could be located in marae and other community centres as well as in schools.
The need was highlighted to create teaching resources relevant to New Zealand school students.
Schools were provided with too few course books for the numbers of students taking the courses. In future roll outs, it would be important to provide sufficient teaching resources for numbers of students enrolled.
Students would be helped by sourcing or producing relevant formative assessment materials with the same level and format as the qualifying examinations.
Upgrading the programme to accommodate new versions of the course content and testing software would involve extra costs unique to the programme. Schools would need funding for sufficient technical support to ensure that their computer systems are adapted quickly to cope with these changes.
Staff training and professional development would need to be carefully targeted to the staff who teach MOS courses.
New MOS teachers would need training, and ongoing professional development of MOS teachers would be needed to enable them to cope with upgrades to the curriculum and technology.
Initial training was seen as desirable to help new schools to set up MOS training courses.
Most schools did not perceive the project as being a true ongoing partnership between schools, businesses and the Ministry. They felt 'dumped on'.
True partnerships would involve ongoing cooperation and communication for the duration of the project.
Schools and students would benefit from a greater awareness of the business world and its opportunities. Businesses could get greater exposure and recognition of the social good of their involvement.
Communication could be fostered by periodic newsletters reporting progress with the project, and by business representatives visiting schools once or twice a year.
The Digital Opportunities Manager or the Generation XP Project Coordinator could facilitate communication.
Navigation
Contact Us
Education Data Requests
If you have any questions about education data then please contact us at:
Email:
Requests EDK
Phone:
+64 4 463 8065