Laptops for Teachers: An evaluation of the TELA scheme in schools (Years 1 to 3)

3. Laptops for teachers (TELA) evaluation

3.1. Evaluation focus

The focus of this evaluation was to monitor the impacts of the TELA scheme on teachers’ professional lives including the impacts on administration and management, lesson planning and preparation and, in particular, the impact on classroom teaching and learning. Consistent with Ministry of Education expectations for the TELA scheme, the evaluation sought to find out “what kind of professional tasks are undertaken using the laptop” and “patterns of use over time and what kind of professional tool the laptop becomes” (Ministry of Education, 2004). The goal was to understand the impacts of TELA so that the scheme might be adjusted to best support the integration of the laptops into school and teacher practices.

3.2. Evaluation framework

Policy support for change and sustainability

Research that has sought to explicate what contributes to the sustainable systemic use of ICTs has highlighted the role of national policy in shaping the context for ICT (see for example Kozma, 2005; Olson, 2000; Selwyn, 2002: Venezky, 2004). Hennessey, Ruthven and Deaney (2005) warn that policy approaches that ignore personal and professional beliefs and that expect teachers to adopt an innovation do not necessarily lead to classroom change. In New  Zealand, the TELA initiative affords all New Zealand teachers access to a personal laptop computer. Evaluation findings to date have found that teachers bring their own experiences, abilities and knowledge to laptop ownership. Policy expectations for the use of electronic data in association with professional development were unwritten to a large extent but have become essential activities in many schools. The evaluation has identified contextual factors such as leadership and school technological infrastructure as enabling or constraining teacher laptop use, but there are also the professional learning opportunities that other policy initiatives such as Numeracy, Literacy and the ICT PD cluster programme offer that can support schools as they undertake whole-school change. A recent policy document, the e-Learning Action Plan developed for New Zealand schools (Ministry of Education, 2006), promotes the use of communication and learning networks as a way to provide teachers with relevant and timely information about effective e-Learning teaching practice. e-Learning is now the term used to identify any learning that is facilitated by, or supported through, the smart use of information and communication technologies. This places learners and teachers at the centre of their own communication and information networks and is the key idea in a new action plan for schools that aspires to transform the way we learn (Ministry of Education, 2006). This evaluation describes examples of teacher laptop use that match e-Learning objectives for effective teaching, and considers other policy initiatives that have helped to make effective use of teachers’ TELA laptops.

The school context for change

Although teachers may work closely with one another in school groups or ‘learning communities’ (Lave & Wenger, 1991) where one might expect innovations involving ICT to be readily shared, in reality practice develops over time, and sharing information with colleagues is not enough to embed a new practice (Loveless, DeVoogd & Bohlin, 2001; Hennessey & Deaney, 2004). Research indicates that integrating ICT is a gradual, reflective process for most teachers and one that is influenced by a complex mix of factors. In particular, effective practice involves developing new forms of pedagogy (Loveless et al, 2001; Hennessey & Deaney, 2004). Loveless et al. discuss the differing perspectives of ‘old’ and ‘new’ pedagogies, but comment that with three or fewer computers in a classroom and with close to 30 students the pedagogical strategies are limited and that instructional methods will more than likely be substitutes for the chalkboard, or in the case of the student working at a computer screen would leave learning management up to the software program, so that the learner never has to manage his or her own learning. Many of the ‘new’ ways of knowing require teachers to establish new classroom routines and procedures that reflect evolving theories about knowledge, and have not diffused into the majority of classrooms because the pressures of such a change are too great – the new teacher, concerned with classroom management, is not likely to be adventurous enough, the established teacher is already comfortable teaching in a particular way, or there is no school priority given to ICT-supported learning. Hennessey and Deaney (2004) found that a supportive organisational culture can lead to further resources and expanding practice, which in turn leads to teachers’ increasing confidence, skills and motivation towards using ICT. Organisational factors or whole-school characteristics were found to have the biggest motivating influence on both sustainability/development and dissemination of ICT-supported practice, with access to technology being the most frequently mentioned factor in this group. Organisational change and prioritisation of ICT by the school had a significant impact on both development and dissemination. This evaluation looked at the contextual factors that support or constrain teacher use of laptops and described any change over the three-year period.

Teacher confidence, and pedagogic beliefs

In a study of British secondary schools where teachers used ICT in their teaching by Hennessey and Deaney (2004) there was a gradual but perceptible process of pedagogical evolution taking place. Their research found that teachers’ technical confidence and confidence in approach played a key role in change to ICT-supported learning, as did affinity, technology skills, resistance to change and teacher age. Hennessey and Deaney propose that while school and curriculum requirements may have some influence in teachers taking up ICT-supported practices, once practices are established and trialled, teacher confidence and enthusiasm for using ICT and their pedagogic beliefs may become more significant motivational factors underlying their sustainability over time and generalisability to further contexts (p. 6). This is reflected in Shulman and Shulman’s (2004) model for teacher learning that has individual reflection at the centre, supported by individual, community and policy factors. These studies make it apparent that any evaluation of teachers’ use of laptops must bear in mind the policies that support such use and the context in which teachers work. However, of prime importance for sustainability of ICT-supported teaching is the individual teacher’s voice that must be heard. This evaluation therefore sought to identify and portray both how teachers were using their laptops and the set of inter-related factors that affected the integration of the laptops into teachers’ professional lives, with the overall goal of developing an understanding of how and why Years 1 to 3 teachers came to use their laptops in different ways over time. The emphasis of the evaluation was on the immediate impacts of the TELA scheme on teachers’ professional practices and the factors that enabled and constrained these practices. Although this generated incidental data about changes in the school context and wider policy initiatives, these were not the direct focus of the study.

3.3. Evaluation methodology

The TELA evaluation design was to use three-yearly cycles of annual nationwide surveys via a written questionnaire, and regional focus groups. The different methods and how they relate to the evaluation focus is now given:

3.3.1. Questionnaires

Surveys are useful for generating prevalence data on first-hand experiences (Cohen, Manion & Morrison, 2000). In this evaluation the survey questionnaires were designed to provide prevalence data on different types of teacher use of the laptops and the kinds of support they had experienced for these uses. The Years 1 to 3 questionnaire built on some years of experience with other teachers (Years 4 to 13 teachers who had TELA laptops). Many of the same questions were used across the year groupings and across the three years of the study so that comparisons of frequency of use and patterns of use over time could be made. The impacts of the TELA scheme on teachers’ professional lives were monitored by the inclusion of questions on laptop use for teacher professional growth and collaboration, lesson planning and preparation, administration, access to, and quality production of teaching, learning and assessment resources, and classroom practice. Questions also prompted teachers to report on their self-assessment of the expertise and comfort with a range of tasks. Free response questions were included where categories had not been defined, so that teachers could describe more fully their experiences, such as what they had found useful about any laptop-based professional development and describing an example of laptop use in the classroom.

Questionnaire data were analysed using the statistical package SPSS. Frequencies and ratios were calculated; percentages were rounded up to the next whole number. Qualitative data from free-response questions were coded into categories. Frequencies and ratios were calculated, but reporting these data was designed to highlight particular trends and weightings given in teachers’ responses. The combination of quantitative and qualitative survey data provided for a deeper understanding of the ways teachers were making use of the TELA laptops, and the factors that contributed to these uses.

3.3.2. Focus groups

Focus groups can also be used as a method in their own right but they are also a useful complement to other data collection methods (Cohen, Manion & Morrison, 2000). In this study, the focus group discussions enabled researchers to understand more fully the impacts on teachers’ professional lives and to explore the factors that influenced these impacts, albeit for a very small group of teachers. The focus groups served as a check that the questionnaire addressed key concerns and practices as these evolved for Years 1 to 3 teachers over the three-year period of the study. The focus group discussions allowed the researchers to validate their interpretations of the qualitative questionnaire comments. They also allowed for the fuller exploration of some of the issues associated with teacher use of laptops. Topics for the focus group discussions included what kind of professional tasks were being undertaken using the laptop and how these uses had changed from one year to the next. Teachers were also asked to comment on factors that enabled and constrained their laptop use and their goals for the next year. Focus group discussions were taped and transcribed. In this report the focus group data have been included to illustrate points made in teacher written responses.

3.4 Participants

3.4.1. Questionnaire respondents

The procedure to recruit Years 1 to 3 teachers for the questionnaire began with creating a random sample from a list provided by the Ministry, as at August 2004, of schools involved in the laptop scheme. From a pool of 1,195 schools, a random sample of 120 schools was generated. The 76 Years 1 to 8 schools that had taken part in the Years 7 and 8 evaluation were added to bring the sample of schools to 196. The sample was stratified, in case of non-response, leading to a sample of 204 schools. In 2006 a further 126 schools were added, making the sample 330, so as to ensure wider coverage of schools and teachers.

The researchers contacted the principals of the schools in the sample, notifying them about the Ministry of Education Laptops for Teachers evaluation and inviting their school to take part in the evaluation. Principals were advised that questionnaires would be sent out in the second term of 2006, and then again in 2007 and 2008. The principal was asked to nominate one teacher who would accept responsibility for distributing, collecting and returning the completed paper questionnaires to the research team, and for forwarding the website address to teachers who chose to complete the questionnaire online.

Respondent teachers represented schools in all deciles, mostly in main urban areas, and all schools were co-educational. One hundred and two schools returned completed questionnaires in 2006 (54 contributing, 45 full primary, 2 composite/restricted composite, and 1 special school), 119 in 2007 (61 contributing, 56 full primary, 1 composite and 1 special school) and 100 in 2008 (49 contributing, 48 full primary, 1 composite/restricted composite, and 2 special schools).

The number of respondents was 271 in 2006, 340 in 2007 and 317 in 2008. Around 30% of Year 1-3 teachers in each year of the evaluation were under the age of 35 years; 23% between the ages of 35 and 44 years; and 47% were over the age of 45. Nearly all Years 1 to 3 teachers were female (2006-96%: 2007-96%: 2008-95%) and nearly all respondents had a teaching role in their school. Nearly three-quarters were classroom teachers (2006-69%: 2007-74%: 2008-73%). Around a sixth (2006-15%: 2007-12%: 2008-14%) were heads of department, syndicate leaders or senior teachers. Each year around a fifth (2006-18%: 2007-19%: 2008-20%) of questionnaire respondents had responsibility for ICT in their schools. Across the three years around a quarter had between 0-5 years teaching experience (2006-25%: 2007-26%: 2008-25%). Around a third had between 6 and 15 years experience (2006-33%: 2007-31%: 2008-33%). Just over two-fifths in each year of the evaluation had spent more than 15 years teaching (2006-42%: 2007-43%: 2007-42%).

It is important to note that it is impossible to know if the respondent teachers in this evaluation are representative of the Years 1 to 3 teaching population that have accessed TELA laptops. Because of this, and because of the self-report nature of the evaluation data, caution is needed interpreting the findings. The data reflect what teachers considered relevant in relation to the questions about their laptop use and what supported, and constrained, this use. No classroom observations were conducted and so the examples of classroom for teaching and learning are also teacher self-report. This said, teacher perceptions and beliefs are important because they have been linked to teacher use of ICTs.

3.4.2. Focus group respondents

There were three focus group meetings held in regional areas each year – Hutt Valley, Marlborough and Taranaki. Focus group schools were selected on their geographical location, with due regard to achieving a spread of school socio-economic status and size. Consideration was also given to a mix of state and integrated schools. In 2006, when the Years 1 to 3 evaluation began, all the schools were selected by a Ministry of Education representative were or had been involved in the ICTPD School Clusters Programme³. Having selected the schools, the researchers contacted the schools initially by phone followed by letter. Schools that declined to be part of the study were replaced by similar schools in relation to the variables identified above. Every effort was made to encourage teachers to attend a focus group by pointing out the benefits of participation.

Each year between 14 and 20 Years 1 to 3 teachers from between 20 and 26 schools took part in focus group discussions. Focus group discussions were held in non-school venues and lasted for up to three hours. Those attending commented on the positive experience of attending a focus group and on the professional development that it had given as a space to share ideas and examples of practice using ICT. Discussion was lively and positive.

3.5. Evaluation timetable, evaluation reports and dissemination

The evaluation timetable (Appendix A) shows that findings were presented in reports at six-monthly intervals, informing the ongoing thinking (about the TELA scheme) of the policy and programme manager stakeholders. There have been numerous formal and informal discussions with the TELA project manager about the findings and their implications for TELA policy. One research paper has been given at a national educational conference (Harlow, Cowie & Jones, 2006) and one at an international conference (Cowie, Harlow, Jones & Cooper, 2008). One article focusing on laptop/ICT use by Years 1 to 3 teachers⁴ has been published in a New Zealand educational journal – Computers in New Zealand Schools. This final report of the Years 1 to 3 findings should be viewed as one element in a total utilisation process.

Footnotes

3. Since 1999, The Ministry of Education has funded ICT professional development in schools through an ICT PD School Clusters Programme. By March 2004, approximately half of New Zealand schools were or had been involved in this programme (Ham, 2005).
4. Harlow, A., & Cowie, B. (2008). Will the teacher’s laptop transform learning? Computers in New Zealand Schools. (20)3.