Laptops for Teachers: An evaluation of the TELA scheme in Auckland schools

2. Methodology

This evaluation utilised a range of data collection tools, including surveys, teacher logs, focus groups, interviews and classroom observations. It was undertaken over a four-year period, with the baseline data collection occurring early in 2005 and the final round in November 2008.

The evaluation consisted of two parts. The first was a longitudinal study, which aimed to focus on teacher use of their laptops over time. The surveys and teacher logs were the key sources of data for this part of the study.

In the second part, data were gathered through individual and group interviews and classroom observations. There was a different focus for each year of the study. Initially, the focus involved inquiry into values and beliefs and current use of technology. Then, in 2006 the focus was on barriers to use. Finally, in response to the data gathered in 2006 the foci for both 2007 and 2008 were changed to better reflect the situation found in the participant schools and to minimise the difficulties experienced in retaining school participation, particularly in relation to surveys. The focus became the place of laptops in the wider learning environment, and indepth studies of selected schools were undertaken. This shift in focus also enabled a consideration of the impact of the TELA project within the changing policy context as elaborated previously.

Participant schools

Participant schools were purposively selected from the Greater Auckland area with the intention of providing a cross-section of schools in terms of decile, size and type (contributing, full primary and intermediate). A list of 20 schools was compiled from information regarding the number of teachers who had initially taken up the Ministry offer for laptops at schools in the wider Auckland region.

The criteria for this shortlist were more than 10 teachers with laptops at the school and that the school was within relatively easy access for the evaluation team. A range from intermediate, contributing and full primaries was selected. An attempt was also made to select both urban and rural schools.

These schools were initially contacted by letter and phone. Participant information sheets and consent forms were supplied for the board of trustees at a school level and for individual participant teachers. Thirteen of these schools agreed to participate.

Baseline data were received from the 13 participating schools, through the initial survey. It was the intention of the evaluation team to continue to gather longitudinal survey data from these 13 schools in each year of the study. However, as is discussed subsequently, the reluctance of teachers to continue to participate, at the regular intervals intended, meant this did not occur. Rather, participation in some cases was sporadic, and no survey data were gathered in 2007.

In addition, in 2005, six case-study schools were selected to participate in focus group interviews and to complete the teacher logs. These were determined based on responses to the surveys, where teachers had been asked to indicate their willingness to be part of the case studies. Again, an effort was made to ensure a representative spread of schools.

In 2007, a different group of three schools was selected as case-study schools. Two of these three schools were selected based on earlier findings, which had indicated that ICT was being used extensively in teaching and learning. They were part of the initial six case-study schools. The third was selected based on its decile and its status as an ICTPD lead school, also suggesting the likelihood of extensive use of ICT. There was no comparable school amongst the initial six case-study schools so it was necessary to look further afield.

Table 2.1 provides a summary of the demographic factors of each of the thirteen participant schools in May 2005. The original case-study schools for the 2005 and 2006 foci are indicated with an asterisk. The 2007-08 indepth case-study schools are indicated by a plus sign.

Data collection

A range of data collection methods was used across the time of the evaluation. This included surveys, teacher logs, focus groups, observations and interviews. Table 2.2 summarises when these data were collected.

Longitudinal surveys

Surveys were distributed to the 13 participating schools in May 2005. The purpose of these surveys was to gather baseline data regarding teachers’ expectations of the impact of the laptops, their use of computers prior to receiving their laptops and their perceived skill and confidence level in using computers. A total of 188 surveys were returned and analysed from across 13 of the schools. This represented a return rate of 83% based on the reported number of laptops across the 13 participating schools. In terms of total staff numbers, the return rate was 54%.

A second survey was administered in November 2005. As is described in more detail in chapter four, some changes were made to the baseline survey for this longitudinal survey. This was the first of what was intended to be an annual survey considering teacher self-reported use of their laptops; their competence, or capability, and confidence in using the laptops; and the extent of impact on various areas of their professional lives. A total of 190 surveys were returned and analysed from 11 of the initial 13 schools. This represented a return rate of 54%, if one considers the total staff roll across the 13 initial participant schools. While this is the same return rate as for the earlier survey, it should be noted that more staff had laptops in these schools as a result of further roll-outs by the Ministry. Therefore, one could expect a higher return rate.

The same survey was administered in November 2006³.This allowed for a consideration of changes in competence, confidence and use over the preceding twelve-month period. Unfortunately, only 157 surveys were returned and analysed from ten schools, a return rate of only 45% if one considers the total staff numbers at the 13 participant schools in 2005.

This low return rate was problematic for the ongoing validity of the longitudinal study as it was originally designed. Not only had the overall participant levels dropped, the number of staff completing both November surveys was less than 70 suggesting any ongoing longitudinal data sample would be very small. In addition, informal feedback from schools completing both surveys suggested that a subsequent survey, in 2007, would be completed by even fewer participants. This feedback, along with the participant numbers and the data gathered, which showed little change between the two surveys, resulted in an agreement with the Ministry that no survey would be administered in 2007.

A final survey was administered in November 2008. This survey asked the same core set of questions as the previous two surveys but also considered other areas, which had arisen during the 2007 focus on classroom practice. A total of 195 surveys were returned from across 12 schools, a return rate of 56% if one considers staff numbers from 2005.

Table 2.3 summarises the total number of surveys returned and analysed from each of the schools across the four surveys administered. Table 2.4 summarises the total number of participants returning 1, 2, 3 or 4 surveys from each school. The total number of surveys returned by participants was 738⁴.

As Table 2.4 shows, only 25 teachers completed all four surveys. This reflects the difficulty in gaining ongoing participation in evaluations and the extent to which staff change schools over time.

Given the small number of participants completing more than one survey, the decision was made to compare results from November 2005 and November 2008 only. A total of 70 participants from across ten schools completed both of these surveys.

Teacher logs

Teacher logs were developed and distributed to teachers who taught in the first group of six case-study schools. The purpose of these logs was for teachers to detail their use of computers over a ‘typical’ school week. The types of use contained in the log (and their descriptors) were taken from a study by Ward (2006). The logs were distributed in 2005 and again in 2006 providing snapshots of use in each period.

There were two key methodological concerns with the logs. Firstly, only the school week was included. As became clear subsequently, a key advantage of the laptops was their portability and the enhanced ability they gave teachers to work outside school hours. While teachers may have included weekend use into their reports of use during the school week, it is likely that the reported use is much lower than actual in some instances. Secondly, the logs were not always completed by the same teachers in 2006 as in 2005. While the same schools were used, the issue of staff turnover and lack of regular, ongoing participation found in the surveys were also of concern here.

Given these two concerns and the relatively small shifts in use seen through an analysis of the teacher logs, these were not used in either 2007 or 2008⁵.

Qualitative data

Individual and group interviews, as well as classroom observations, provided a range of qualitative data across the four years of the study. The focus of the data collection and the questions posed varied each year.

2005 group interviews

In 2005 group interviews involving between three and eight teachers in each school were undertaken in six schools. Table 2.5 provides a summary of the number of teachers participating at each school.

The interviews were taped and transcribed in four of the schools. Detailed notes were taken at the other two, where the interview situation was not conducive to taping owing to background noise in one instance and teacher reluctance in the other. All the interviews were conducted after school and were semi-structured in nature. The teachers who were interviewed predominantly self-selected, others were asked by their principals to participate.

In 2005 the questions asked focused on:

• the perceived value of the laptops
• innovative and/or ‘cutting edge’ use in the classroom
• current skill and confidence levels
• what differentiated a highly confident user from other teachers.

2006 group interviews

Two series of group interviews were undertaken in 2006. Four schools participated in both sets of interviews, while one participated in the first set only. Teachers at the sixth school declined to participate further in the study in 2006.

The first set of group interviews focused on shifts between 2005 and 2006. Questions again centred on participants’ use of laptops, their skill and confidence levels and the perceived value of the laptops.

The second set of group interviews considered barriers to laptop use. Schools were asked to select teachers whose own laptop use was limited either in terms of the level of use or the breadth of the nature of use. However, in one school there were no teachers in this category as the teachers in this school all appeared to be relatively high users of their laptops.

2007/2008 case studies

Data gathered during 2005 and 2006 suggested teachers were regularly using their laptops and that little would be gained from continuing with the initial evaluation design. In consultation with the Ministry the focus was shifted for the last two years of the evaluation. As a result three of the original 13 schools were purposively selected in 2007 as case studies for a consideration of the place of the laptops in learning.

Over a two- to three-day period the evaluation team visited, in turn, each of the three schools selected. We interviewed individual teachers, senior managers and students at each school. Teacher participants were nominated by the schools and the students came from their classrooms. In this instance, we had outlined our intention to map effective use of technology in the classroom; thus we required the teachers be relatively high users. Table 2.6 details the interviews undertaken at each school. We also observed classroom practice.

The focus of these visits was to consider more broadly the place of the laptops in teaching and learning. The hypothesis was that the impact of the laptops was more far reaching than their direct use might suggest. Rather than asking teachers what they used their laptops for, we asked them about the teaching and learning that was occurring in their classrooms. We then looked to see where the laptop sat in relation to these wider activities.

In the subsequent sections of this report, data from each of these data collection methods are reported under timeframe headings. Not all the data gathered and reported in the relevant interim reports have been repeated in this final report. In particular, the data gathered in 2006 from the surveys, interviews and logs are not included. These data showed that little change had occurred from 2005 and, as such, add nothing to the overall evaluative findings.