TELA: Laptops for Teachers Evaluation—Final Report Years 9-13

The purpose of this evaluation was to investigate the impacts of the Laptops for Teachers Scheme: TELA (referred to from here as the TELA scheme) on teachers’ work over a period of four years (2003-2006) and to record emerging changes in laptop use.

Author: Bronwen Cowie, Alister Jones, Ann Harlow, Clive McGee, Bev Cooper, Mike Forret, Thelma Miller, & Ben Gardiner
Date Published: June 2008

---

3.1 EVALUATION FOCUS

The purpose of this evaluation was to monitor the impacts of teacher access to a TELA laptop. The goal was to understand the nature of the impacts on teachers’ professional lives with particular emphasis on the impacts on administration and management, lesson planning and preparation and classroom teaching and learning.

3.2 EVALUATION FRAMEWORK

Research on teacher and school integration of ICT indicates that an evaluation of teacher use of laptops needs to take into account both the immediate and wider setting in which teachers find themselves to understand how and why they come to use technology in different ways over time. As indicated earlier (Section 2), research on teacher use of ICT has shifted from a focus on the personal attributes of the individual teacher to a focus on the teacher in the context of the school and wider social, economic and policy setting (Lim, 2002; Selwyn, 2002; Zhao et al., 2002; Zhao & Frank, 2003). This setting is recognised as shaping and framing teacher opportunities and incentives to use ICT and as such is of concern to an evaluation of the impact of policy. Indeed, Olson (2000) suggests that policymakers need to ‘engage in conversations with teachers about their work culture, the technologies that sustain it and the implications of new approaches for those technologies’ (Olson, 2000, p. 6). Policy evaluation research, which that has attended to individual teacher responses, has highlighted the multiple demands on teachers’ working lives and the diversity in the ways they make sense of and accommodate these demands (Knapp, 2002). A focus on the teacher in context is consistent with research on teacher change/professional learning and with research on school change and improvement (Fullan, 2001, 2005; Senge, 1994; Spillane, Reiser, & Reimer, 2002; Wenger, McDermott, & Snyder, 2002). Teacher change/learning is now generally viewed just as much as a situated social process as it is a personal process (Putman & Borko, 2000). Institutional change as a process of organisational learning (Wenger et al., 2002) and sustainable systemic change is predicated on developing the social environment (Fullan, 2005; Spillane, 2006). Drawing together these aspects, the assumptions that underpin this evaluation of the impact of the TELA laptops are: 

• Valued teacher uses of laptops include all the professional tasks associated with being a teacher: administration, management, pastoral care, professional development and collaboration, communication, lesson planning and preparation, and classroom instruction.
• Change in teacher laptop use is just as likely to be incremental and accumulative (evolutionary) as it is transformative.
• There is a need to consider the individual teacher in context.
• It is important to consider what sustains and may accelerate change as well as what enables change. 

These assumptions, while they derive from consideration of research on ICT and teacher and school change, fit within a ‘systems’ perspective towards evaluation (Patton, 2002). In this case, changes in one part of the system are considered to lead to changes among all parts and the system as a whole. This evaluation model seeks to identify and portray the set of inter-related factors that affect the change process. It conceptualises these factors as a system in which the whole is greater than the sum of the parts.

A system is a whole that is both greater than and different from its parts. Indeed, a system cannot be validly divided into independent parts as discrete entities of inquiry because the effects of the behaviour of the parts on the whole depend on what is happening to the other parts. The parts are so interconnected and interdependent that any simple cause-effect analysis distorts more than it illuminates. Changes in one part lead to changes among all parts and the system itself. Nor can one simply add the parts in some linear fashion and get a useful sense of the whole. (Patton, 2002, p.120)

The analysis strategy then for the evaluation had a holistic perspective where the whole phenomenon under study was understood as a complex system that was more than the sum of its parts. The focus was on the complex interdependencies and system dynamics that could not meaningfully be reduced to a few discrete variables and linear, cause-effect relationships (Patton, 2002, pp.40-41). The TELA scheme specifications suggested, and the literature as briefly reviewed in Section 2, indicated the need to focus on a breadth of teacher professional uses for the laptop and the contextual factors (school culture and infrastructure, professional learning opportunities and wider policy factors) that may constrain and enable these uses.

3.3 EVALUATION METHODOLOGY

The TELA evaluation design was to use a mixed-methods approach incorporating three yearly cycles of annual nationwide questionnaires, regional focus groups, and school-based longitudinal case studies. The surveys provided prevalence data on teacher use of the laptops and the kinds of support they had experienced for these uses. An initial focus group of teachers informed a baseline questionnaire to provide an insight into the practitioner stakeholder viewpoint. Questions were reviewed and discussed with the Ministry and central issues in the evaluation were clarified. This process of question development was ongoing. The focus group component allowed for in-depth exploration of the issues associated with teacher use of laptops in a manner that allowed participants to build on each other’s ideas and introduce topics of interest to them (Morgan & Krueger, 1993). The case studies provided for a rich description and analysis of what was happening in a bounded system (Bassey, 1999) - in this case, the schools or classrooms where teachers were utilising their laptops. An initial focus group discussion contributed to the design of the questionnaire. Subsequent focus group discussions illuminated the questionnaire data and contributed to the adaptation of the questionnaire to reflect what was happening in schools over the three-year period. Building on this, the case studies provided insights into how particular school contexts interact with teacher personal factors to shape teacher use of laptops. A timeline is provided in Appendix A.

3.4 PARTICIPANTS

3.4.1 Questionnaire respondents

In 2003, the sampling process for the questionnaires involved contacting all schools involved in the STELA scheme. Randomised sampling was used to identify 20% of the available laptop schools, but modified to ensure a range of school types (urban/rural, co-educational and single sex) and decile. Schools agreeing to take part in the questionnaire nominated a contact person who was responsible for distribution and collection of questionnaires. Teachers who responded in 2003, 2004 and 2005 were drawn from the same sample of schools, although the teacher respondent sample was not necessarily identical. The demographic constitution of the respondent group was similar sample in each of the three years. 

Forty-nine secondary schools returned completed questionnaires in 2003 (48 in 2004; 50 in 2005). The number of respondents was 688 in 2003 (744 in 2004; 690 in 2005). Across the three years, the respondents were generally experienced teachers: half had been teaching at a secondary school for more than 15 years. There were roughly equal numbers of male and female teachers. Almost two thirds of teachers came from main urban centre secondary schools. Nearly all teachers had a teaching role in their school. Over a third were heads of department or faculty. Around 15% of teachers had responsibility for ICT in their schools. Each year around a fifth (2003-21%; 2004-21%; 2005-26%) of questionnaire respondents rated themselves as expert users of the laptop, two thirds as intermediate users (2003-65%; 2004-67%; 2005-65%) and around 10% as beginner users of the laptops (2003-14%; 2004-12%; 2005-9%). Nearly 80% had a computer at home and around 40% had had a computer at home for at least six years.

3.4.2 Focus group respondents

Each year six focus groups were conducted to ensure reasonable coverage of issues (Vaughan, Schumm, & Siaguh, 1996). Two groups were established in Auckland, one in the Waikato region, one in the Wellington region, one in the upper South Island, and one in the Christchurch region. Focus group schools were selected based on their reasonable proximity to a main centre with due regard given to school socio-economic status, gender, and the school roll. Each school was asked to nominate three teachers to attend, these teachers were to reflect a range of interests and abilities in their school and would not necessarily include the school ICT coordinator. 

Between six and ten teachers from three or four schools attended each group, a total of around forty-eight teachers from twenty schools. Focus group discussions were held in non-school venues and lasted for up to three hours. The focus group participants held positions in their schools ranging from that of principal to first-year teacher. Some described themselves as “computer freaks” and others said the laptop was the first computer they had used. While it was not always easy to recruit teachers, due mainly to time factors, those attending commented on the positive experience of attending a focus group and on the professional development that it had given as a space to share ideas and examples of practice using ICT. Discussion was lively and positive. Teachers were keen to maintain their involvement and with the exception of two teachers who shifted schools the same teachers attended the focus groups in each of the three years.

3.4.3 Case study sample

In selecting case study schools from the list of schools involved in the laptop scheme, factors of region, decile, school size, ICT experience, school type, area, implementation and professional development strategies were taken into account. Eight case studies were undertaken in a range of schools in the Auckland, Waikato, Wellington and Christchurch areas each year. The same schools participated in each of the three years. Between eight and fourteen teachers were interviewed at each school, some individually and some in departmental groups. When they were available the same teachers were interviewed in each of the three years.

3.5 RESEARCH REPORTS AND DISSEMINATION

The evaluation findings have been documented in the form of written and oral reports to the Ministry of Education at six monthly intervals (see Appendix A). This report synthesises the three years of findings. 

To ensure that the emergent findings of the evaluation have been available to a range of stakeholders formal feedback sessions have been conducted with stakeholders groups including the laptop providers and the TELA scheme school advisory group. There have been ongoing informal discussions with the TELA project manager. Research papers have been presented at national and international educational research conferences and teacher professional conferences. A paper directed towards principals has been posted on the TELA website. The following conference presentations and publications have taken place to date: 

The education research community

2007

Jones, A., Cowie, B., & Harlow, A., (2007). Distributed leadership in support of teacher use of laptops in New Zealand. Paper presented at AERA conference, Chicago. 9-13 April 2007.

2006

Cowie, B., Jones, A., & Harlow, A., (2006). The Digital Horizons: Laptops for teachers policy initiative: Impacts and consequences. New Zealand Annual Review of Education, 15:2005, 111-132. Wellington, N.Z: Victoria University. 

Jones, A., Cowie, B., & Harlow, A., (2006). The affordance of a laptop for teaching science: The case of Josh. Paper presented at NARST professional conference, San Francisco. 3 - 6 April 2006. 

Cowie, B., Jones, A., & Harlow, A., (2006). Infrastructure and information environments. Paper presented at NARST professional conference, San Francisco. 3 - 6 April 2006. 

Jones, A., Cowie, B., & Harlow, A., (2006). Laptops for teachers policy initiative in New Zealand: Impacts and consequences. Paper presented at APERA professional conference, Hong Kong. 27 November – 1 December 2006.

Cowie, B., Harlow, A. Jones, A., & Gardiner, B. (2006). Laptops as a fulcrum or lever: Changes in secondary teacher practices over the three years subsequent to their access to TELA laptops. Paper presented at NZARE professional conference, Rotorua. 5 - 8 December 2006. 

2005

Cowie, B., Jones, A., Harlow, A., & Miller, T. (2005). The Digital Horizons: Laptops for teachers’ policy initiative: Impacts and consequences. Paper presented at NZARE conference, Dunedin. 6 - 9 December 2005. 

2004

Cowie, B., Jones, A., Harlow, A., & McPherson, M. (2004). Teachers with laptops in New Zealand: Impacts on teachers and their practice. Paper presented at NZARE conference, Wellington. 24 – 26 November 2004. 

Teacher professional conferences and papers

2006

Cowie, B., Harlow, A., & Jones, A. (2006). How are science teachers using their laptop? Paper presented at SciCon professional conference, Hamilton. 2 - 6 July 2006. 

Cowie, B., Jones, A., & Harlow, A. (2006). Digital Horizons laptops for teachers: Practices and possibilities. Paper presented at PPTA professional conference, Wellington. 19 - 21 April 2006. 

2005

Cowie, B., & Jones, A. (2005). Digital Horizons: laptops for teachers evaluation study – update on secondary teachers’ experiences. Paper prepared as a report for school leaders. September 2005. Available on http://www.minedu.govt.nz (Waikato University TELA Research Report)