ICTPD through three lenses: An Evaluation of the ICTPD School Clusters Programme 2001-2003

Publication Details

This research report is submitted to the Ministry of Education as part of an evaluation of the initiatives announced in the documents Interactive Education: An Information and Communication Technologies Strategy for Schools (Ministry of Education, 1998) and Digital Horizons: Learning Through ICT: A Strategy for Schools (Ministry of Education, 2001).

Author(s): Vince Ham, Peter Moeau, Sandra Williamson-Leadley, Hasan Toubat and Michael Winter, Christchurch College of Education.

Date Published: 2006

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Executive Summary

The purpose of the evaluation reported here was to assess the overall impact of the ICTPD School Clusters programme of teacher professional development (PD) as implemented in the period from January 2001 to the end of 2003.

The report covers the impact of the ICTPD Clusters Programme from three perspectives:

  1. National Trends 2001-2003
  2. Two Kura Kaupapa Māori Cluster Case Studies, 1999-2003
  3. Teacher and Facilitator Action Research Projects, 2001-2003

The research questions guiding the evaluation were:

  1. How effective were the PD strategies implemented by various clusters in meeting stakeholder and participant goals?
  2. How effective were the ICTPD cluster programmes in participating schools in terms of promoting effective teacher use of ICTs for lesson preparation and planning and school administration?
  3. How effective were the ICTPD cluster programmes in increasing teachers' skills and knowledge related to the educational applications of ICTs?
  4. How effective were the ICTPD cluster programmes in increasing effective classroom usage of ICTs by students?
  5. How much and in what respects have these effects changed as the model has rolled out over time and to new cohorts?

The main findings of the study with regard to these questions follow.

The ICTPD Clusters programme of teacher professsional development 2001-2003 had a significant effect on the national profile of teacher use of ICTs in New Zealand. Teachers in the programmes tended to have goals related to increased ICT skills and more effective use of ICTs with classes, and report high levels of achievement of these goals as a result of the programmes. Teacher use of ICTs for school administration and lesson preparation also increased significantly over the period of the cluster programme, although the programme itself was only one of several factors encouraging this increase in usage. The most striking teacher effects of the programme were to increase significantly teachers' skills in using ICTs, and, perhaps more importantly, to significantly increase their confidence about and actual usage of, a range of ICTs for classroom learning activities. At the end of the project most teachers were using more ICTs, more confidently, more often and for a wider range of curriculum objectives with their classes than before the programme. The key factors which statistically correlated with these increases in classroom usage were teacher confidence with and about ICTs, and the length of time that individual teachers were actively engaged in the PD process.

The Kura cluster case studies provide some insight into how the cluster programme was implemented in Māori immersion contexts in particular. In most respects the Kura teachers' experiences reflect very similar concerns and patterns of progress to those found among non-Kura clusters in this and previous studies (Ham et al 2002), though there seem to be some differences in relation to how the programme is implemented in Kura comapred to mainstream schools occasioned by the shortage of human and teaching resources in those schools and, for many, their geographical isolation.

Like the Kura case studies, the teacher action research reports included in this report also suggest that for many teachers the programme has had a major impact in terms of stimulating deeper reflective practice about teaching and learning generally, and in improving their professional understandings around what might constitute educational 'value' and 'effectiveness' in terms of student learning when using ICTs.

In virtually all key respects the effects of the cluster programme on the 2001-2003 cohort were similar in scope and scale to those found for the previous 1999-2001 cohort.

National Trends in the ICTPD Clusters, 2001-2003

Effective strategies for professional development in ICT

  • Across the clusters as a whole, teachers tended to concentrate on PD goals related to increasing their ICT skills and increasing their use of ICTs with students in classrooms, with secondary teachers tending to report relatively more goals related to skill development and primary teachers relatively more goals related to classroom usage.
  • Overall, participants reported high levels of goal achievement as a result of the ICTPD programmes, and high levels of satisfaction with the programmes in terms of meeting their expectations. The overall proportions of objectives 'fully achieved' and expectations met were slightly lower for the 2001 cohort than for the 1999 cohort.
  • The extent to which goals and objectives were achieved varied by sector, length of time in the programme, and in some instances, gender. Primary teachers and those who had been in the programmes longest reported the highest levels of goal achievement and expectations met.
  • Most, but far from all, teachers tended to prefer one-to-one or small group modes of grouping for professional development, and to prefer sector-based groupings over mixed primary-secondary groups.
  • Female teachers showed a greater preference for working in groups of similar ICT skill level than male teachers.
  • A mixture of formal and informal, in-school and out of school, short term and longer term PD strategies were highly ranked by teachers. Most notably these involved release time, extended retreats, technology mentors and one-to-one tutorials. Teachers did not generally rank professional reading or listserves/online communities highly as effective PD strategies.
  • In almost all respects, what the 2001 cohort of teachers regarded as 'effective' in ICTPD strategies paralleled the preferences of the 1999 cohort before them.

Teachers' use of ICTs for lesson planning and preparation and school administration

  • There was a significant increase of the proportion of routine users of ICTs for both lesson planning/preparation and school administration among teachers over the period of the ICTPD programme. Both before and after the ICTPD programme, teachers tended to use ICTs for lesson-related purposes more often than for other administrative purposes.
  • Internet usage in particular, for both school administration and for lesson planning and preparation, increased significantly among both secondary and primary teachers over the period of the ICTPD programme. There is some evidence that this is part of a general increase in teachers' use of internet across the board, and is not confined or unique to the cluster schools.
  • There were significant differences between teachers of different school sectors in their use of ICTs for most lesson related and other administrative purposes, with primary teachers reporting greater increases in such usage compared to secondary teachers. This difference may be largely explained by the generally higher levels of entry point usage among secondary teachers compared to primary teachers.
  • Taking into account the higher entry level usage of the 2001 cohort of teachers, the increased use of ICTs for administration and planning reported for the 2001 cohort is comparable to that reported for the 1999 cohort. For both cohorts, moreover, the ICTPD programme seems to have been but one of several determinants of the reported increases in teachers' use of ICTs for various administrative and lesson planning/preparation purposes. Many school principals, whether in clusters or not, seem to be requiring administrative use of ICTs for reporting, roll checks etc, and it is likely that the laptops for teachers scheme is also having an impact on most teachers' use of ICTs for administrative and lesson planning functions.

Teacher effects: teachers' knowledge, attitudes, skills and confidence regarding ICTs

  • The teachers in the ICTPD programmes generally believed that ICTs could play a valuable role in teaching and learning, both in terms of helping to improve the efficiency of curriculum delivery, and as a pedagogically justifiable investment of school resources.
  • The perceived benefits of ICTs for teaching and learning were felt to relate most to increased student motivation, student skill acquisition (both ICT skills and the Essential Skills), and the achievement of specific curriculum objectives.
  • Primary teachers tended to be more focussed on student acquisition of ICT skills as a benefit of ICT usage, compared to secondary teachers.
  • Teachers' concerns about the use of ICTs for teaching and learning focussed most on issues of technical reliability, lack of time for themselves to become familiar with software and its possible uses, and lack of access for their students.
  • In all respects the reported benefits and concerns regarding the use of ICTs among the 2001 cohort of cluster teachers mirrored those of the 1999 cohort.
  • The majority of teachers in the 2001 cohort entered the programme already competent in a few ICT skills areas, most notably word processing and desktop publishing, but much less competent in most others. Overall, they entered the programme more competent than their predecessors in the 1999 cohort.
  • By the end of the programme the majority of teachers were at high levels of competence in several, though not necessarily all, of the ICTs commonly used in schools. The increases in competence were most notable with regard to graphics, internet applications, and basic computer systems. There were smaller increases in competence with regard to databases, spreadsheets, and, for secondary teachers, multimedia applications.
  • In terms of teacher confidence about ICTs, the overall trend was that teachers moved from being generally not confident before the ICTPD programme to being generally confident after participating in the programme, both in terms of personal confidence with ICTs and in terms of confidence about their use with students in classes.
  • Teachers were more confident about using ICTs personally than they were about using ICTs with their classes, both before and after the programme. The greater effect of the programme, however, was the increase in teacher confidence regarding classroom use.
  • Teachers' confidence with ICTs, both personally and with classes, was related to: gender (males reported higher confidence levels on entry to the programmes), length of teaching experience (teachers with more experience reported higher increases in confidence), sector (primary teachers reported higher increases in confidence than secondary), and length of time in the ICTPD programme (those who were longer in the programme reported higher increases in confidence). The 2001 cohort results also tended to confirm the major finding of the 1999 cohort evaluation that there was a correlation between increased teacher confidence about ICTs and increased classsroom usage of them.
  • In all key respects the trends identified in teachers' confidence in relation to using ICTs personally and with classes reflected those found for the 1999 cohort.

Classroom effects: teachers' classroom usage of ICTs for teaching and learning

  • There was a distinct tendency for teachers in the 2001 cohort to use a smaller range of ICTs with classes than those in the 1999 cohort.
  • Consistent with the findings for the 1999 cohort, secondary teachers in the 2001 cohort tended to use a smaller range of ICTs with classes than primary teachers. This is to be expected given primary teachers' need to teach all Essential Learning Areas, and secondary teachers' focus on one or two.
  • The predominant uses of ICTs in classes were for Language/English objectives, and to a lesser extent Social Studies, Maths and Science objectives.
  • Word processing for static print presentation continues to be the predominant student use of ICT in both sectors, followed by use of the internet for research and information gathering. Primary students seem to use a wider range of ICTs for a wider range of curriculum purposes compared to secondary students.
  • The pattern of curriculum coverage and pedagogical purpose for ICT use was substantially the same for the 2001 cohort as for the 1999 cohort, except for a tendency for more use of ICTs for problem solving, and less for content/skill practice, in the 2001 cohort.

Comparison of trends in the 2001 cohort with those in the 1999 cohort

  • In almost all key respects the trends identified in relation to the 2001 cohort of ICTPD teachers mirror those identified in the earlier study of the 1999 cohort.
  • The only areas in which the cohorts differed in any substantial way were :
  1. Demographically the 2001 cohort had proportionally more secondary teachers and proportionally more male teachers taking part than in the 1999 cohort, although both these groups were still under-represented in relation to their proportions in the teaching population generally.
  2. The 2001 cohort of teachers started with generally higher personal skill levels compared to the 1999 cohort. This trend is seen in subsequent cohorts as they enter the programme as well, and seems to indicate that skills based professional development is a continuing commitment in schools which are not part of a cluster programme (Ham, Graham & Toubat 2004).
  3. The 2001 cohort tended to make greater use of the internet than the 1999 cohort, both for general professional and administrative use as well as for teaching and learning activities with students.
  4. The 2001 cohort generally used a smaller range of ICTs with classes than the 1999 cohort.

ICTPD in Two Kura Kaupapa Māori Clusters

As was found in the cluster case studies in the 1999 cohort generally (Ham et al 2002), the ICTPD programmes in Te Urewera and Te Ara Rima clusters had a significant impact on the skill levels, and even more importantly, the confidence levels of most, though not all, teachers in the Kura. For the great majority of teachers in most of the Kura, also, it resulted in greater classroom use of ICTs, and the use of a wider range of ICTs in classrooms. However, the range of experience observed in these regards varied considerably from kaiako to kaiako, and from Kura to Kura. There seems to have been a group of kaiako, many older and often less experienced in teaching, for whom increased skills were achieved, but at the end of the project they were still hesitant about incorporating ICTs into their teaching. On the other hand, another group seemed to deal with the barriers they faced as a challenge to be overcome, both in a technical sense and in the sense of using the inclusion of ICTs into their teaching as an occasion for reviewing their core pedagogical assumptions.

The major issues reported by the kaiako in the Kura related to increasing the connectedness of ICT activities to other things happening in the classroom, practical strategies for organising the rotation of students through ICT activities given the small number of them available, and, above all perhaps, frustrations at the often low levels of technical reliability of the equipment.

The experiences of the kaiako in the Kura case study schools reflect most of the trends reported throughout the clusters generally, with two exceptions. The first is that the shortage of relievers and staffing movements in the Kura made it difficult to provide ongoing, long term PD programmes accessible to everyone equitably. The other exception is that the fragilities of the technical infrastructure in many of these small, low decile, isolated schools seem to have had a disproportionately diluting effect on the effectiveness of the PD compared to most of the other clusters.

The patterns of students' classroom usage of ICTs, curriculum coverage and learning outcomes observed in the two Kura case study clusters were little different from those reported for the cluster programmes in general. The predominant uses observed were language related activities using word processors or the internet, leading either to print-based presentation or, occasionally, multimedia slideshows. Most activities observed involved students on lower or middle order thinking skills, though there were some notable exceptions to this. Collaborative activities were common, and the general levels of technical skill shown by students were high.

The overall feeling in both the case study clusters was that their ICTPD programmes had been extremely effective in 'getting them off the ground' in terms of the effective uses of ICTs for teaching and learning. However, most at the end also expressed a sense of incompleteness in terms of meeting their full needs as immersion schools. As one participant put it: 'käore nawhe', not enough.

Teacher and Facilitator Action Research Projects

A range of action research or self-study projects was undertaken as part of the evaluation. In these, teachers or cluster facilitators volunteered to investigate their own 'best practice' in relation to some aspect of ICT use in the teaching and learning process, or some aspect of facilitating teacher professional development in that area.

A range of action research or self-study projects was undertaken as part of the evaluation. In these, teachers or cluster facilitators volunteered to investigate their own 'best practice' in relation to some aspect of ICT use in the teaching and learning process, or some aspect of facilitating teacher professional development in that area. Twelve action research reports are summarised in Part 3 of this report. The full reports themselves are included in an accompanying Supplement document.

Taken as a group, the action research reports provide richer case study evidence illustrative of some of the national trends identified in the first part of this report. They help exemplify, personalise and put into a more comprehensive context how some of these averaged trends actually played out in individual teachers' professional lives during the programmes, and in this they are in many ways the stories behind the statistics.

What is more striking in these reports, however, is that they bring to our attention an aspect of the ICTPD programme that is difficult to discern in the 'averaging effect' of national statistics and general trends. That is, the impact of the ICTPD programmes on teacher's and faciltators' understandings about teaching and learning. In this sense the research reports in this section need to be read not so much as stories about teachers' growing technical ability with ICTs, nor their growing confidence as ICT users, nor indeed their more frequent classroom use of ICTs, which tend to be the stories evoked by the statistical trends analysis. Rather they are best read as self-reflective stories of some individual teaching professionals coming to grips with what constitutes 'quality' and 'value' in teaching, and what it is that might produce 'productive learning', when students use ICTs in classes. The ICTs discussed might provide the context of the studies, but they are not the substance. The substance is denoted by the much more pedagogically oriented phenomena that they address. They are, at essence, studies of things like 'equity', 'integration', 'quality teaching', 'information literacy', 'peer tutoring', 'narrative', 'whole school development', and so on, and not studies of ICTs. In critically investigating such phenomena in the context of their own practices, the authors themselves seem to move beyond a preoccupation with ICTs per se, to a preoccupation with more fundamental aspects of what it is, for them, that makes for quality teaching and learning in general.

Contact Us

Education Data Requests
If you have any questions about education data then please contact us at:
Email:      Requests EDK
Phone:    +64 4 463 8065