Laptops for Teachers: An evaluation of the TELA scheme in schools (Years 4 to 6)

3. Laptops for teachers (TELA) evaluation

3.1 Evaluation focus

The focus of this evaluation was to monitor the impacts of the TELA scheme on teachers’ professional lives with particular emphasis on the impacts on administration and management, lesson planning and preparation and classroom teaching and learning. Consistent with Ministry of Education expectations for the TELA scheme the evaluation sought to find out “what kind of professional tasks are undertaken using the laptop” and “patterns of use over time and what kind of professional tool the laptop becomes” (Ministry of Education, 2004). The goal was to understand the impacts of TELA so that the scheme might be adjusted to best support the integration of the laptops into school and teacher practices.

3.2 Evaluation framework

Initially, research on teacher adoption of ICT tended to discuss teacher personal, professional and contextual factors as if they were independent (Zhao & Frank, 2003). This contrasts with recent research on teacher and organisational learning, which construes it is as much a situated social process as an individual process (Putman & Borko, 2000: Senge, 1994; Spillane, 2004). It also contrasts with current research on teacher ICT use, which positions ICT as a tool that shapes, and is shaped by, the immediate and wider school environment in which it is deployed (Lim, 2002: Zhao & Frank, 2003). Research that has sought to explicate what contributes to the sustainable systemic use of ICTs has highlighted the role of national policy in shaping the context for ICT (see for example, Kozma, 2005; Olson, 2000; Selwyn, 2002: Venezky, 2004). Taken together, this research indicates that any evaluation of teacher use of laptops needs to take into account the setting in which teachers find themselves, along with their personal preferences and views, in order to understand how and why they come to use technology in different ways over time. In line with this, Fisher, Higgins and Loveless (2006) found a nested and multi-dimensional approach, as suggested by Shulman and Shulman (2004), was useful in the analysis of the integration of ICT in schools in the United Kingdom. Shulman and Shulman’s (2004) model for teacher learning has individual reflection at the centre, this is supported by individual, community and policy factors. This multi-faceted approach is consistent with what Patton (2002) has described as the “Interdependent System Relationship Maps” conceptualisation of evaluation.

In a systems evaluation approach, the phenomenon under study is understood as a complex system that is more than the sum of its parts. The focus is on the complex interdependencies and system dynamics that cannot meaningfully be reduced to a few discrete variables and linear, cause/effect relationships (Patton, 2002, pp. 40-41). The main question to be answered is, “How and why does this system as a whole function as it does?”.

"A system is a whole that is both greater than and different from its parts. Indeed, a system cannot be validly divided into independent parts as discrete entities of inquiry because the effects of the behaviour of the parts on the whole depend on what is happening to the other parts. The parts are so interconnected and interdependent that any simple cause/effect analysis distorts more than it illuminates. Changes in one part lead to changes among all parts and the system itself. Nor can one simply add the parts in some linear fashion and get a useful sense of the whole." (Patton, 2002, p.120)

The TELA evaluation therefore sought to identify and portray both how teachers were using their laptops and the set of inter-related factors that affected the integration of the laptops into teachers’ professional lives with the overall goal of developing an understanding of how and why Years 4 to 6 teachers came to use their laptops in different ways over time. The emphasis of the evaluation was on the immediate impacts of the TELA scheme on teachers’ professional practices and the factors that enabled and constrained these practices. Although this generated incidental data about changes in the school context and wider policy initiatives, these were not the direct focus of the study.

3.3 Evaluation methodology

The TELA evaluation design was to use three-yearly cycles of annual nationwide surveys via a written questionnaire, and regional focus groups. The different methods and how they relate to the evaluation focus is now given:

Questionnaires

Surveys are useful for generating prevalence data on first-hand experiences (Cohen, Manion & Morrison, 2000). In this evaluation the survey questionnaires were designed to provide prevalence data on different types of teacher use of the laptops and the kinds of support they had experienced for these uses. The Year 4-6 questionnaire built on some years of experience with other teachers (Years 7-13 teachers who had TELA laptops). Many of the same questions were used across the year groupings and across the three years of the study so that comparisons of frequency of use and patterns of use over time could be made. The impacts of the TELA scheme on teachers’ professional lives were monitored by the inclusion of questions on laptop use for teacher professional growth and collaboration, lesson planning and preparation, administration, access to and quality production of teaching, learning and assessment resources and classroom practice. Questions also prompted teachers to report on their self-assessment of the expertise and comfort with a range of tasks. Free response questions were included where categories had not been defined, so that teachers could describe more fully their experiences, such as what they had found useful about any laptop-based professional development and describing an example of laptop use in the classroom.

Questionnaire data were analysed using the statistical package SPSS. Frequencies and ratios were calculated – percentage numbers have been rounded up to the next whole number. Qualitative data from free-response questions were coded into categories. Frequencies and ratios were calculated, but reporting these data was designed to highlight particular trends and weightings given in teachers’ responses. The combination of quantitative and qualitative survey data provided for a deeper understanding of the ways teachers were making use of the TELA laptops, and the factors that contributed to these uses.

Focus groups

Focus groups can also be used as a method in their own right but they are also a useful complement to other data collection methods (Cohen, Manion & Morrison, 2000). In this study, the focus group discussions enabled researchers to understand more fully the impacts on teachers’ professional lives and to explore the factors that influenced these impacts, albeit for a very small group of teachers. The focus groups served as a check that the questionnaire addressed key concerns and practices as these evolved for Years 4 to 6 teachers over the three-year period of the study. The focus group discussions allowed the researchers to validate their interpretations of the qualitative questionnaire comments. They also allowed for the fuller exploration of some of the issues associated with teacher use of laptops. Topics for the focus group discussions included what kind of professional tasks were being undertaken using the laptop and how these uses had changed from one year to the next. Teachers were also asked to comment on factors that enabled and constrained their laptop use and their goals for the next year. Focus group discussions were taped and transcribed. In this report the focus group data have been included to illustrate points made in teacher written responses.

3.4 Participants

3.4.1 Questionnaire respondents

For this evaluation the final sample of 330 schools with Years 4 to 6 students consisted of 179 full primary schools, 134 contributing schools, 10 composite schools and 7 special schools. In 2005, the procedure to recruit Years 4 to 6 teachers for the questionnaire began with creating a random sample from a list provided by the Ministry, as at August 2004, of schools involved in the laptop scheme. From a pool of 1195 schools, a random sample of 120 schools was generated. The 76 Years 1 to 8 schools that had taken part in the Years 7 and 8 evaluation were added to bring the sample of schools to 196 schools. The sample was stratified, in case of non-response, leading to a sample of 204 schools. In 2006 a further 126 schools were added, making the sample up to 330 schools, so as to ensure wider coverage of schools and teachers.

The researchers contacted the principals of the schools in the sample, notifying them about the Ministry of Education Laptops for Teachers evaluation and inviting their school to take part in the evaluation. Principals were advised that questionnaires would be sent out in the second term of 2005, and then again in 2006 and 2007. The principal was asked to nominate one teacher who would accept responsibility for distributing, collecting and returning the completed paper questionnaires to the research team, and for forwarding the website address to teachers who chose to complete the questionnaire online.

Respondent teachers represented schools in all deciles, mostly in main urban areas, and all schools were co-educational. Seventy-one schools returned completed questionnaires in 2005 (25 contributing, 44 full primary, 1 composite/restricted composite, and 1 special school), 112 in 2006 (66 contributing, 29 full primary, 1 composite) and 131 in 2007 (72 contributing, 56 full primary, 1 composite/restricted composite, and 2 special).

The number of respondents was 200 in 2005, 279 in 2006 and 353 in 2007. Nearly all respondents had a teaching role in their school. Around a fifth were heads of department, syndicate leaders or senior teachers. Each year around a quarter (2005–24%: 2006–28%: 2007–31%) of questionnaire respondents had responsibility for ICT in their schools. Across the three years around a fifth had between 0-5 years teaching experience (2005–22%: 2006–25%: 2007–22%). There was an increase in those with 6-15 years experience (2005–32%: 2006–31%: 2006–36%). Just over two-fifths in each year of the evaluation had spent more than 15 years teaching (2005–45%: 2006–43%: 2007–41%). Between three-quarters and four-fifths were female teachers (2005–79%: 2006–82%: 2007–76%).

It is important to note that it is impossible to know if the respondent teachers in this evaluation are representative of the Years 4 to 6 teaching population that have accessed TELA laptops. Because of this, and because of the self-report nature of the evaluation data, caution is needed interpreting the findings. The data reflect what teachers considered relevant in relation to the questions about their laptop use and what supported and constrained this use. No classroom observations were conducted and so the examples of laptops for teaching and learning are also teacher self-report. This said, teacher perceptions and beliefs are important because they have been linked to teacher use of ICTs.

3.4.2 Focus group respondents

There were two focus group meetings each year – one for urban schools and one for rural schools. Focus group schools were selected on their geographical location, with due regard to achieving a spread of school socio-economic status, and size. Consideration was also given to a mix of state and integrated schools. In 2005, at which time 46% of all schools had been involved in an ICT PD cluster, avoiding schools that had been involved, or were involved in an ICT professional development cluster, was not practical and so these schools were included when selecting the focus group schools. Having selected the schools, the researchers contacted them initially by phone followed by letter. Schools that declined to be part of the study were replaced by similar schools in relation to the variables identified above. Every effort was made to encourage teachers to attend a focus group by pointing out the benefits of participation.

Each year between six and seven teachers from urban schools and between five and eight teachers from rural schools took part in focus group discussions. Focus group discussions were held in non-school venues and lasted for up to three hours. Those attending commented on the positive experience of attending focus groups and on the professional development that it had given as a space to share ideas and examples of practice using ICT. Discussion was lively and positive.

3.5 Evaluation timetable, evaluation reports and dissemination

The evaluation timetable is in Appendix A where it can be seen that evaluation findings were presented in reports at six monthly intervals informing the ongoing thinking (about the TELA scheme) of the policy and programme manager stakeholders. There have been numerous formal and informal discussions with the TELA project manager about the findings and their implications for TELA policy. Data has been presented to key stakeholders. One research paper has been given at a national educational conference (Harlow, Cowie & Jones, 2006), and as results from other primary levels come in there will be further papers that include findings from the Years 4 to 6 evaluation. This final report of the Years 4 to 6 findings should be viewed as one element in a total utilisation process.