3.1 EVALUATION FOCUS

The Ministry of Education sought to find out “what kind of professional tasks are undertaken using the laptop” and “patterns of use over time and what kind of professional tool the laptop becomes” (Ministry of Education, 2004). The focus of this evaluation was to monitor the impacts of the TELA scheme on teachers’ professional lives with particular emphasis on the impacts on administration and management, lesson planning and preparation and classroom teaching and learning. The goal was to understand these impacts so that the scheme might be adjusted to best support the integration of the laptops into school and teacher practices.

3.2 EVALUATION METHODOLOGY

The TELA evaluation design was to use three yearly cycles of annual nationwide questionnaires, and regional focus groups, while fieldwork was taking place across two other project teams. The questionnaires provided prevalence data on different types of teacher use of the laptops and the kinds of support they had experienced for these uses. The focus group component allowed for in-depth exploration of the issues associated with teacher use of laptops in a manner that allowed participants to build on each other’s ideas and introduce topics of interest to them (Morgan & Krueger, 1993). Focus group discussions illuminated the questionnaire data and contributed to the adaptation of the questionnaire to reflect what was happening in schools over the three-year period.

3.3 PARTICIPANTS

3.3.1 Questionnaire respondents

For the questionnaire, the contract specified a sample of approximately 500 teachers including teachers with laptops from a stratified sample of ten percent of intermediate schools with laptops, the numbers being made up with Year 7 and 8 teachers from other types of schools. An investigation revealed just over half of Year 7 and 8 students attended intermediate and a third attended full primary schools and so the decision was to make the initial selection from the full primary schools. From a pool of 1097 schools, a random sample of every twelfth full primary school generated a sample of 76 schools and an estimated pool of 200 teachers with laptops. There were 80 intermediate schools from which a random sample of 21 was created. The final sample of schools with Year 7 and 8 students consisted of 89 non-intermediate schools (76 full primary schools, 12 composite schools and 1 restricted composite school) and 21 intermediate schools.

The researchers contacted the principals of the schools in the sample, notifying them about the Ministry of Education: Laptops for Teachers evaluation and inviting their school to take part in the evaluation. Principals were advised that questionnaires would be sent out in the third term of 2004, and then again towards the end of 2005 and 2006. The principal was asked to nominate one teacher who would accept responsibility for distributing, collecting and returning the completed paper questionnaires to the research team, and for forwarding the website address to teachers who chose to complete the questionnaire online.

The number of respondents was 175 in 2004, 153 in 2005, and 149 in 2006. Thirty-seven schools returned completed questionnaires in 2004 (14 intermediate, 21 full primary, 2 composite), 43 in 2005 (13 intermediate, 29 full primary, 1 composite), and 70 in 2006 (12 intermediate, 52 full primary, 2 composite, 1 special). Laptop teachers represented schools in all deciles, mostly in urban areas, and all schools were co-educational.

Nearly all teachers had a teaching role in their school. Around a fifth were heads of department, syndicate leaders or senior teachers. Each year around a quarter (2004-23%: 2005-26%: 2006-30%) of questionnaire respondents had responsibility for ICT in their schools. Across the three years there was a drop in teachers who had between 0-5 years of teaching experience (2004-38%: 2005-31%: 2006-25%), and an increase in those with 6-15 years of experience (2004-26%: 2005-32%: 2006-37%). Just over one-third in each year of the evaluation had spent more than 15 years teaching (2004-36%: 2005-35%: 2006-38%). The proportion of female respondents rose over the three years (2004-56%: 2005-67%: 2006-70%).

3.3.2 Focus group respondents

Focus group schools were selected on their geographical location and reasonable proximity to a main centre with due regard given to achieving a spread of school socio-economic status, gender, and size. Having selected the schools, the researchers contacted the schools initially by phone followed by documentation by letter. Where schools declined to be part of the study, they were replaced by other schools that were fairly similar in relation to the variables identified above. Every effort was made to encourage teachers to attend a focus group by pointing out the benefits of participation.

There were two focus group meetings each year – one for intermediate schools and one for full primary schools. Each year, six teachers from three intermediates and between four and six teachers from four to six full primary schools took part in focus group discussions. Focus group discussions were held in non-school venues and lasted for up to three hours. Those attending commented on the positive experience of attending a focus group and on the professional development that it had provided as a space to share ideas and examples of practice using ICT. Discussion was lively and positive.

3.4 EVALUATION FRAMEWORK

Initially, research on teacher adoption of ICT tended to discuss teacher personal, professional, and contextual factors as if they were independent (Zhao & Frank, 2003). This contrasts with recent research on teacher and organisational learning which construes it as much a situated social process as an individual process (Putman & Borko, 2000: Senge, 1994; Spillane, 2004). It also contrasts with current research on teacher use of ICT which positions ICT as a tool that shapes, and is shaped by, the immediate and wider school environment in which it is deployed (Lim, 2002; Zhao & Frank, 2003). Added support for the need to focus on teacher laptop use in context, comes from research that has sought to explicate what contributes to the sustainable systemic use of ICT, which has highlighted the role of national policy in shaping the context for ICT (see for example Kozma, 2005; Olson, 2000; Selwyn, 2002: Venezkey, 2004). Taken together, this research indicates that any evaluation of teacher use of laptops needs to take into account the setting in which teachers find themselves, along with their personal preferences and views, in order to understand how and why they come to use technology in different ways over time. This approach is consistent with what Patton (2002) has described as the “Interdependent System Relationship Maps” conceptualisation of evaluation.

In a ‘systems evaluation’ approach the phenomenon under study is understood as a complex system that is more than the sum of its parts. The focus is on the complex interdependencies and system dynamics that cannot meaningfully be reduced to a few discrete variables and linear, cause-effect relationships (Patton, 2002, pp.40-41). The main question to be answered is, ‘How and why does this system as a whole function as it does?’ In this case, the system is conceptualised as greater than the sum of the parts.

A system is a whole that is both greater than and different from its parts. Indeed, a system cannot be validly divided into independent parts as discrete entities of inquiry because the effects of the behaviour of the parts on the whole depend on what is happening to the other parts. The parts are so interconnected and interdependent that any simple cause-effect analysis distorts more than it illuminates. Changes in one part lead to changes among all parts and the system itself. Nor can one simply add the parts in some linear fashion and get a useful sense of the whole. (Patton, 2002, p.120)

The TELA evaluation therefore sought to identify and portray the system or set of inter-related factors that affected the integration of the laptops into teachers’ professional lives with the goal of developing an understanding of how and why teachers come to use technology in different ways over time.

3.5 EVALUATION TIMETABLE, EVALUATION REPORTS AND DISSEMINATION

The evaluation timetable may be found in Appendix A where it can be seen that evaluation findings were presented in the form of evaluation reports at six monthly intervals informing the ongoing thinking (about the TELA scheme) of the policy and programme manager stakeholders. There have been numerous formal and informal discussions with the TELA project manager about the findings and their implications for TELA policy. Interim findings have been presented to key stakeholders. One research paper has been given at a national educational conference on the Year 7 to 8 data (Harlow, Cowie & Jones, 2006), and as results from other primary levels come in there will be further papers that include findings from the Year 7 and 8 evaluation. This final report of the Year 7 and 8 findings should be viewed as one element in a total utilisation process.