Outcomes for teachers and students in the ICTPD School Clusters Programme 2005-2007: A national overview
This report focuses on the effectiveness of the 2005-2007 Information and Communication Technologies Professional Development (ICT PD) School Clusters programmes and supplements previous evaluations of the first five ICT PD programmes. It is part of an ongoing evaluation of the ICT PD teacher professional development initiative, which has been implemented in New Zealand since 1999.
Author(s): Vince Ham, CORE Education. Report prepared for the Ministry of Education.
Date Published: July 2009
This report is available as a download (please refer to the 'Downloads' inset box). For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.
This report is submitted to the Ministry of Education as part of an ongoing evaluation of the Information and Communication Technologies Professional Development (ICT PD) teacher professional development initiative, which has been implemented in New Zealand since 1999. The focus of this particular report is on the effectiveness of the 2005-2007 cohort of ICT PD School Clusters programmes and as such supplements previous evaluations of the first five ICT PD School Clusters Programmes submitted to the Ministry in 2002, 2004, 2005, 2006 and 2007.
As outlined in the performance agreements between the Ministry and the ICT PD clusters, the ICT PD School Cluster programmes in New Zealand are aimed at increasing teachers' ICT confidence, skills and pedagogical understandings of ICTs, fostering quality learning communities, and increasing the frequency and quality of the integration of ICTs to support effective classroom teaching and learning. The research reported here takes these performance criteria as its starting point to provide a national overview of the programme's impact on teachers, teaching and classroom learning in New Zealand, by means of an analysis of the results of the End of Project surveys of c.1250 participating teachers.
The overall finding of the study is that the 2005-2007 ICT PD programme had a marked and significant effect on the teachers and students in cluster schools with respect to all of its key goals. There were very high levels of goal achievement reported by participants, and marked increases or changes with respect to all of the relevant Ministry's objectives as outlined in cluster performance agreements. At the national level, the programme achieved its overall goals of: significantly increasing teachers' skills and confidence with ICTs, improving teachers' understandings of the role of ICTs in teaching and learning, and providing quality ICT-mediated learning experiences for students. The programme for the 2005-2007 cohort increased teachers' ICT skills, confidence and understandings about ICTs, and significantly increased routine student use of a range of ICTs for learning in classes. The 2005-2007 programme achieved levels of participant satisfaction that, though high, were somewhat lower than the preceding (2004-2006) cohort, and achieved overall increases in skills, confidence, understanding and classroom/student usage of ICTs for learning in orders of magnitude at least similar to those of earlier cohorts.
We note also that overall the ICT PD programme continues to have a significantly greater impact among primary teachers and schools than among their secondary counterparts.
Specifically, we found that:
The great majority of teachers expressed high levels of satisfaction and goal achievement at the end of the programme. Participant satisfaction, while high across the board, varied somewhat by sector and length of time in the programme, with primary teachers and those who had been in the PD programme for most of the 3 years stating higher levels of satisfaction and goal achievement than secondary teachers and those who were in the programme for only a few months or up to a year. The programmes were seen by the majority (66%) of participants as having been a 'significant' event in their overall development as teachers, which for many contributed well beyond any ICT-specific issues of increased technical skill, to encompass improved understandings in relation to teaching and learning more generally. Teachers generally appreciated all of the ways the programmes were delivered and the knowledge and expertise of particular facilitators. The content of the programmes, being for most programmes a mix of skills development, classroom ideas and principles of effective teaching and learning generally, was seen by more teachers than not as relevant and effective. Prominent among the most appreciated aspects of the programmes, were the various opportunities provided to share ideas and problems and reflect together on their use of ICTs, both within clusters and through the various regional and national conferences, suggesting that a learning community focused on critical reflection of classroom ICT practice is continuing to develop both regionally and nationally. At the end of the programme the greatest persisting concerns for teachers with regard to their use of ICTs were a lack of student access to equipment, a perceived lack of time to keep up to date, and technical reliability, and some concern about the continuing need for PD to continue after the programme's formal end point. There was a marked and significant increase in teachers' ICT skills over the period of the programme. Teachers' reported skill levels on entry to the programme were generally moderate to high already, though somewhat variable across different ICTs. By the end of the programme the great majority of teachers reported moderate or high skill levels across the whole range of educationally useful ICTs measured. There was also a marked and significant increase in teachers' confidence about their professional use of ICTs over the period of the programme, both in terms of their confidence as personal users and in relation to students using ICTs in their classes. Teachers reported that on entry the great majority of them had been either 'anxious', 'not confident' or 'neutral' about their professional use of ICTs, female and primary teachers being less confident than male and secondary teachers. By the end of the programme over three quarters of all the demographic groups studied were reporting moderate or even high levels of such confidence. Moreover, the longer teachers were in the programme, the greater was the extent of their gain in confidence. Throughout the programme levels of confidence as personal users remained higher than those related to classroom use of ICTs. The effect of the ICT PD programme on teachers' classroom practices was variable from teacher to teacher and from school to school, but substantial overall. Over a third of teachers acknowledged that over the programme period their classroom practices had changed to 'a large extent' or 'completely', while 86% felt their practices had changed at least to some extent. Female and primary teachers reported greater change in this regard than male or secondary teachers. Participants reported that on entry to the programme , they were already generally positive about the value of using ICTs for teaching and learning. At the end of the programme they showed a similarly positive disposition towards the value of ICTs in the teaching and learning process, but many stated that they now had a clearer conception of how its educational value might be judged.
- The most frequently reported effects of using ICTs with classes on their practice included: teaching with increased confidence and enthusiasm, expanding their repertoire of teaching techniques, using a wider range of activities, and catering for a greater range of student need across a broader range of curriculum objectives.
- The most prominent effects of the programme in terms of developing teacher understandings about learning were expressed as: a better understanding of student-centred teaching and learning, getting new ideas about establishing a resource-rich learning environment, increased knowledge of teaching and learning theories, styles and approaches, challenging pedagogical perspectives through sharing and discussion.
There was a marked and significant increase in teachers' use of ICTs with their classes as a result of the programme. Two thirds of teachers had either never used ICTs with classes prior to the programme, or had only used them once or twice a year. By the final year of the programme three quarters of participating teachers reported that their students were using ICTs on a routine basis (i.e.: in most or all units of work over the year). Primary teachers reported using ICTs in a greater proportion of their units of work than secondary teachers. The other significant predictors of increased classroom usage of ICTs by teachers in the clusters were their rising levels of confidence with and about the technology, and the length of time they were actively engaged in the PD programme. The surveys provide some proxy evidence of the conscious alignment of ICT-based classroom activities to identified student learning outcomes.
- Teachers tended to report the learning outcomes from student use of ICTs in terms of: increased student-centredness in lessons, increased student motivation, coverage of a wide range of curriculum topics and objectives, student acquisition of ICT skills, and increased opportunity for learning activities which promoted communications skills, enquiry skills, high order thinking, creativity and a range of social skills.
- There were no statistically significant sector differences in relation to increased student use of ICTs for higher order thinking and critical thinking skills, but primary students were more likely to engage in frequent use of ICTs with classes than secondary students for all of: creative activity, curriculum content acquisition, information gathering or processing, collaborative learning, motivation and technical skill acquisition.
- The largest proportion of ICT-based student activities reported by teachers related to the Languages Essential Learning Area (26%), followed by Mathematics (15%), Science (13%), Social Studies (12%).
In most respects the effects and trends listed above for the 2005-7 ICT PD national cohort are similar in both nature and magnitude to those reported for the earlier cluster cohorts. In general, the same effects, of a similar size, have been identified in all cohorts. The only notable exceptions or differences between the 2005-7 cohort and earlier cohorts in terms of programme effect or effect size were:
- The skill levels of teachers in the 2005-7 cohort on entering the programme, and also on exiting it, were generally higher than those of the earliest cohorts but similar to those reported for the 2003-5 and 2004-6 cohorts that immediately preceded it. Skill levels, both on entry and exit, as well as programme impacts on skills, are 'flattening off' as successive cohorts begin and finish their programme with higher skill levels across the range of ICTs used in education.
- There were some cohort differences in terms of the learning outcomes of ICT use by students in the 2003-5 and 2004-6 cohorts compared to earlier cohorts. There is a clear tendency for students in the 2005-7 cohorts to show higher frequencies of use of ICTs than previous cohorts with respect to all of: static presentation (mostly word processing and slide shows), problem solving activities (mostly through spreadsheet use), information processing activities (mostly through Internet use), online communication (email, social software), and curriculum practice activities (mostly games, Drill and Practice or interactive/multimedia tutorials).
Finally, we note that levels of goal achievement and meeting of expectations were lower than in the 2004-6 cohort but still much higher than in the 2003-5 cohort, across all of the groups of goals identified.