Executive Summary

The focus of this evaluation was to investigate the impacts of these programmes upon student achievement (and to look particularly at Māori and Pasifika student achievement) teaching approaches and how the programmes influenced school-wide teaching and learning. The evaluation was from June to December 2011 and, during the project, researchers and Ministry research and curriculum personnel worked together to shape the evaluation.

The programmes

The 2011 literacy and mathematics programmes for students were:

1. Reading in Year 1: Shared book/language experience to supplement guided reading for students at risk of not achieving National Standards after one year at school.
2. Reading in Year 2: Intensive and responsive teaching of reading for students at risk of not achieving the National Standards in reading.
3. Writing in Years 4 to 8 for students who were 'below' or 'well below' National Standards in writing.
4. Reading and writing for English language learners.
5. Mathematics for Years 3 to 8 students who were 'below' or 'well below' National Standards (Accelerating Achievement in Mathematics – ALiM).
6. Mathematics for Years 3 to 8 students who were 'below' or 'well below' National Standards and were taught by Specialist Mathematics Teachers (SMT).

Schools selected the programme related to the area of focus for their students. Some schools took part in more than one programme. The first five programmes were taught for 10 weeks and the SMT programme was taught for 15 weeks.

The main aim of the programmes was to accelerate learning for a small group of students who were judged as 'below' or 'well below' National Standards. Accelerated learning was defined as occurring when the amount of a student's learning in reading, writing or mathematics increased by more than the amount expected in the equivalent time in the regular classroom programme. That is, if reading, writing and ALiM (maths) 10-week programmes (or the SMT maths programme of 15 weeks) resulted in greater progress than would be expected, learning was assessed as accelerated.

Participating schools

Schools were selected by regional Ministry of Education offices, with input from mathematics facilitators, literacy advisors and professional learning development (PLD) providers. Priority was given to schools with high percentages of Māori, Pasifika or English for speakers of other languages (ESOL) students who were achieving below expectations. Schools selected for the programmes were expected to have:/p>*

• effective classroom teaching and learning occurring
• students identified as operating 'below' or 'well below' National Standards
• school leadership actively supporting teacher learning and innovation
• effective classroom teachers.

Within the schools principals were asked to select teachers for their expertise in working with low achieving students and for their willingness and ability to respond to student learning needs.

Students who were 'below' or 'well below' National Standards were selected for the programmes. Consideration was also given to the likelihood the student would benefit from the intensive programme and from working in a small group. Other selection criteria applied by schools included regular attendance and ability to work in a group.

Programmes were run throughout 2011. Planning meetings were held prior to the start of the programmes by the Ministry of Education to brief teachers and principals on the intentions of the programmes.

Evaluation methodology

The evaluation sought to determine the extent to which student learning was 'accelerated' as a result of their participation in the programmes and what pedagogical practices had contributed to any gains.

The research questions for the evaluation were:

1. To what extent have the programmes in reading, writing, mathematics and English language learning accelerated learning for programme students?
2. To what extent have the programmes accelerated learning specifically by Māori, Pasifika and students with special learning needs?
3. What factors contributed to the success of programmes in terms of student achievement and teaching approaches?
4. How responsive to the needs of students was the teaching, including any differences across the targeted groups?
5. How well did teachers of the different programmes design and deliver learning plans to students?
6. To what extent did a more responsive pedagogical approach better meet the needs of students and did teachers change their usual teaching practices during the programmes?
7. Have reading, writing and mathematics programmes been influenced by other current or recent Ministry activities?

Data collection

A mixed methods approach to data collection was used. Data sources were: online surveys, student achievement data, interviews and case studies.
Data were sought for this evaluation from all schools involved in programmes during Term 3 2011 (n=240 schools: 94 involved in literacy programmes, 146 in mathematics).

• Principals and teachers: Online surveys were emailed to principals and teachers at the start and at the end of the programmes.
• Students: Online attitude surveys were completed with students at the start and at the end of the programme.
• Case study schools: There were 25 case study schools, among these: 11 had mathematics programmes (nine ALiM and two SMT schools); 15 schools had literacy programmes: four had Year 1 reading; five had Year 2 reading; seven had writing in Years 4-8, and there was one English Language Learning school.

Student achievement data: collected pre and post programme

• Mathematics: Progressive Achievement Test (PAT): Mathematics test at the appropriate level and Numeracy Project Assessment – Diagnostic Interview (NumPA)
• Reading: Observation Survey, instructional text reading levels and Overall Teacher Judgement (OTJ)
• Writing: asTTle writing and Overall Teacher Judgement.

Data were analysed for the students in the study and three ethnic groups of students—New Zealand European, Māori and Pasifika. Reading data were submitted by 52 per cent of schools (122 students). Writing data were submitted by 63 per cent of schools (320 students). For ALiM maths schools, PAT data was available for 559 students and 642 students for NumPA. For SMT maths schools, PAT data was available for 252 students and Numpa data for 371 students.

Measuring student progress from short-term interventions is complex and challenging, particularly when using the assessment tools currently available. In this evaluation a variety of measures were used to estimate whether progress for students was accelerated. In some cases the sample sizes are small and some findings, particularly for the ethnic groups, should be considered as indicative only. Further work is needed to develop better measures of progress and acceleration in short-term interventions.

Evaluation findings

Key findings

1. The majority of mathematics, reading and writing students accelerated their learning during the programmes, that is, their achievement increased more in the programmes than would be expected in the equivalent classroom time. In reading, nearly two-thirds of the students improved in relation to the National Standards. In writing, students, on average, improved by at least one level on the asTTle test. In the ALiM maths programme the majority of students gained at least one framework stage over the 10 weeks.
2. Māori, Pasifika and New Zealand European student groups increased their learning in reading and writing. In mathematics, Māori students recorded a significantly higher increase in the scale score than the other two groups for PAT, and Pasifika students recorded the greatest stage increase across all the Numeracy Framework domains.
3. Key elements in the organisation of the programmes were identified and should inform future programmes, eg, small groups, targeted teaching, sessions longer than 30 minutes, at least four lessons a week, responsive teaching, planning and reflection, focus on and use of data.
4. Teacher reflection, planning and preparation was iterative and adaptive to respond to student learning as it evolved. These factors were linked to increases in student achievement in reading, writing and mathematics.
5. Leadership from principals was essential to the programme. Principals played an important role in ensuring there was support for the programme teacher, commitment to the programme and for spreading positive programme outcomes more widely in the school during the programme and beyond it.
6. Principals and programme teachers of the mathematics, reading and writing had high expectations that they would accelerate the learning of most programme students.
7. The small group design of the programmes fostered and enabled safe and more frequent interaction amongst students and with the teacher.
8. Ministry planning and follow-up days for teachers played an important role in setting expectations and accountability for the programmes, disseminating successful outcomes and initiating a community of interest focused around raising the achievement of students who were achieving 'below' or 'well below' expectations.

School organisation for the programmes

In schools there were two main organisational forms or models developed for the programmes; either groups were withdrawn to a separate space or groups were taught in the classroom. The particular teaching arrangements varied; the teacher of the intervention was sometimes the students' regular teacher and sometimes not. In some schools, more than one teacher was involved in the programme. This was more likely to be the case in the literacy schools.

Programmes were organised and run differently across the schools. In maths, three-quarters of the SMT and ALiM programmes were taught four or five times a week. Half of the SMT teachers ran lessons of more that 30 minutes and three-quarters of ALiM teachers' lessons were 30 minutes or more. Over 80 per cent of the reading programmes and 79 per cent of writing programmes had at least four or five lessons per week; 62 per cent of the writing lessons and 35 per cent of the reading lessons were over 30 minutes in duration and half the reading teachers ran 16-30 minute lessons.

It appeared that frequent teaching in lessons over 30 minutes was linked to improved learning outcomes because students could engage in clearly linked, cumulative and focused learning activities over a longer time.

Student achievement

A key aspect of this evaluation was to measure the extent to which the programmes in reading, writing, and mathematics accelerated learning for all students and for Māori and Pasifika students. Data were analysed for the whole student group and three ethnic groups of students—New Zealand European, Māori and Pasifika. Group means showed that all three ethnic groups had comparable levels of achievement change on the various assessment measures for reading, writing, and mathematics.

Reading and writing

The results showed that overall, students achievement in reading and writing improved and this was accelerated. Māori, Pasifika and New Zealand European students all made accelerated progress in terms of group mean scores, that is, they made more progress than they usually would have in the same time.

Reading

In reading, student achievement data were gathered using the Observation Survey, Running Records and an Overall Teacher Judgement against the National Standard.

At the beginning of the programmes almost all reading students (92 out of 95) were rated 'well below' or 'below' the National Standard for reading. At the end of the programmes nearly two-thirds of the ratings teachers gave students from the three ethnic groups improved in relation to the National Standard, eg, improved from 'well below' to 'below' or 'below' to 'at'. At the end of the programmes over a third (n=27) of students were rated 'at' or 'above' the National Standard.
Sixty-one per cent of Year 1 and 68 per cent of Year 2 students improved by two or more colour levels on the Ready to Read colour wheel indicating accelerated learning in the 10-week programmes.

Writing

In writing, pre and post student achievement data was collected using asTTle writing and Overall Teacher Judgement. Analysis of achievement scores showed that the majority of students accelerated their learning over the programmes.

At the end of the programmes, writing achievement of New Zealand European, Māori, Pasifika students improved, on average, by at least one level on the asTTle test.

At the beginning of the programmes almost all students were rated 'well below' or 'below' the National Standard for writing. At the end over half of the programme teachers' ratings of the students in relation to the National Standard had improved eg, ratings had improved from 'well below' to 'below' or 'below' to 'at'.

Mathematics

In mathematics progress and acceleration for student achievement data was assessed using PAT scale scores and changes in group means in the Numeracy Framework domains using NumPA. Overall, across the measures the change in group means were such that student learning can be considered accelerated.

In ALiM and SMT, the PAT scale score and percentage yearly growth group means for the whole group and each of the Māori, Pasifika and New Zealand European groups significantly increased from pre to post programme.
NumPA results showed a significant increase in group means in all domains for both ALiM and SMT from pre to post assessment. The majority of students made a gain of at least one framework stage over the 10 weeks of the ALiM programmes. This compares with the findings from the Numeracy Development Project (Young-Loveridge 2010) where students at stages 4 or 5 would be likely to move a stage in a year. For SMT participants, growth over the 15-week programme was similar to what would be expected for 27 weeks.
Students who began the programmes at a lower stage of the Number Framework tended to make more progress, on average, than those who started at higher stages. This might be expected as steps at the lower stages are smaller and easier to progress through.

The effect sizes for Māori and Pasifika students were higher than that for New Zealand European students.  Pasifika students recorded the greatest stage change growth across all domains in the Numeracy Framework.
The PAT data showed all three ethnic groups made significant gains in PAT scores from beginning to end of the programmes. The average change for each group was comparable and progress was similar, irrespective of starting point across the three ethnicities. Year 4 students showed the average greatest growth in PAT scale score, suggesting the programme may be of particular value to this group.

In the SMT programme, the Year 4 students showed the greatest change as measured in average scale score in PAT mathematics scale score. Growth over the 15-week programme was similar to what would be expected for 27 weeks.

To get an estimate of progress for students participating in ALiM compared to their progress prior to ALiM, PAT data was collected from a small number of schools. Student progress over Terms 1 and 2 was compared to progress over the ALiM programme (Term 3). Data from the 32 students showed significantly greater increases in PAT scale scores over the ALiM programme compared to progress in the two terms prior to the programme. While the results should be interpreted cautiously because of the small numbers, they suggest the programmes did accelerate students' learning at a greater rate than previously.

Māori students

On all of the test and assessment measures Māori students as a group made accelerated progress at a similar rate to Pasifika and New Zealand European students in the reading, writing, and ALiM and SMT mathematics programmes.

In ALiM, the effect sizes for PAT measures for Māori and Pasifika students were comparable, and greater than for the New Zealand European student group. The ALiM Māori student group recorded the highest movement in PAT scale score mean and experienced significantly greater progress than New Zealand European students (not Pasifika) when calculated by percentage of usual yearly growth.

Pasifika students

On all of the assessment measures Pasifika students as a group made accelerated progress in reading, writing, ALiM and SMT programmes. Half of the Pasifika students reached the expected reading level on the National Standard, although the sample was small. Teachers also judged Pasifika students to have improved in relation to the National Standard in writing, eg, ratings increased from 'well below' to 'below' and for some from 'well below' to 'at' the National Standard. Pasifika students recorded the greatest stage change growth across all the domains of the Numeracy Framework.

New Zealand European students

The New Zealand European students as a group accelerated their learning on all of the assessment average change measures.

Programme design elements

These literacy and mathematics achievement results show that, on average, students had accelerated their learning in reading, writing and mathematics by the end of the programmes. The results are consistent with: high expectations and accountability, planning and commitment from those involved, responsiveness, support and reflection time. The programme elements are discussed below.

Planning, reflection, expectations and commitment

The results from the achievement data are consistent with high expectations that principals and teachers had before the programmes began with, that student learning would increase more than it usually would in the equivalent time. Principals and teachers across the schools made a strong commitment to improve student learning. This commitment appeared to come from the way the programmes were set up at the national level through the provision of teacher time for planning and briefing; time for preparation, teaching and reflection; and pre and follow-up sharing of teaching and learning outcomes and an expectation to report back on progress. There was a sense that schools as a whole made a commitment to make the programmes work and to embed positive influences into the programme teacher's own ongoing teaching and across the school.

Programme teachers planned their programmes and kept regular records of day-to-day modifications to plans. Ongoing reflection on formative assessment information informed iterative cycles of responsive planning. Information on student ideas and responses, especially formative assessment, were often shared with a student's regular teacher, with lead teachers in reading, in particular, playing an active role in fostering this communication, as well as helping teachers to refine their practice.

The daily planning and preparation was important to be able to be flexible, responsive and react to the learning that had gone before. Teacher and principal surveys conducted at the end of the programmes reported that students had made more or much more progress than usual. Maths teachers felt gains made during the programmes would be sustained, if at a lesser rate.

Support for teachers

Principals were expected to provide support for the programme teachers and they played a key role in organising satisfactory spaces, resources, scheduling and ensuring there was colleague support for planning and teaching.

In many schools the programmes were seen as a collective responsibility. The allocation of teacher time and the expectation to report back after the end of the programmes combined with commitment and enthusiasm to work to make the programmes successful for the students. Teachers and principals also mentioned the benefits for themselves and the schools in taking part in the programmes, such as adaptations to practice and use of data.

Within the reading and writing programme, another teacher (lead teacher, syndicate leader) frequently acted as a colleague in support or 'critical friend', especially for less experienced teachers. Reading and writing teachers found the most useful support came from the lead literacy teacher as a mentor, literacy advisors and Reading Recovery teachers. Mathematics teachers were provided with professional support from an external facilitator as part of the ALiM and SMT programmes. On the whole, maths teachers found the facilitators to be supportive, to be able to suggest new ideas and provide guidance. Teachers felt collegial support was also a contributor to the success of the programmes.

Strong principal leadership in terms of priority and organisation within the school was important to the success of the programmes. Most principals had high expectations for the programmes and this seemed to create a sense of urgency in teachers to raise student achievement. Most principals also initiated processes to extend any benefits from the programmes to other teachers in a school and many programme teachers were required to share their experiences, especially with the regular teachers of the programme students.

Culturally responsive teaching

How teachers taught was of major importance in this evaluation. Strong encouragement was given to teachers in planning days prior to the programmes to use teaching that was responsive to students' needs and in particular to practice culturally responsive pedagogy. Teachers were given background information and presentations on effective and responsive teaching. Pre-programme survey data showed that most teachers were able to describe aspects of their classroom teaching that were effective and responsive to students' cultural/ethnic characteristics and background. In general, they cited high expectations of students, knowledge of each student and a classroom where all students were included and valued. In post surveys, teachers cited changes they made to their general teaching for different students, such as tailoring the topics of lessons to student interest and catering for particular learning preferences.

Post-survey teacher data and teacher interviews showed that teachers, in general, used their usual teaching approaches but made specific adaptations for individual students based on the students' particular learning needs. Providing focused, individualised attention that was planned and adapted for students was likely to have been different from the daily classroom practice with a larger group of students.

Teachers themselves had mixed views about how they changed their teaching approach to better suit the Māori and Pasifika students in their groups. Teachers reported that they tended to teach all students in the group in a similar way to their usual classroom teaching because there are certain teaching approaches they know to be effective. On the other hand, teachers reported how they had changed specific approaches to suit Māori and Pasifika students, such as choice of reading books, letting each child choose writing topics that interested them and greater use of games and discussion about mathematical ideas. Teachers reported the importance of getting to know all programme students quickly and responding to what students were learning.

ALiM teachers found use of materials and equipment, a variety of approaches, discussion with peers, forming relationships and structured programmes were effective teaching approaches for these students. SMT teachers found working in small groups, modelling, using equipment and encouraging student talk to be effective strategies with their groups of students.

Family/whānau involvement

Most school plans for implementing the programmes emphasised the importance of family/whānau involvement. Almost all principals and teachers reported informing parents of the programme and many invited parents to visit schools to observe lessons and some ran group information sessions. Over the whole group, a large majority of teachers believed that parents and family/whānau were supportive of the programmes and in case study schools only a small minority of parents did not engage when invited to, such as help learning at home. It was not possible to measure the impact upon learning of parental support; however, almost all teachers interviewed in case study schools believed student achievement and attitudes were helped by family/whānau involvement and support.

Actual parental involvement varied across the schools. It was common for schools to advise the whānau/family the student was part of the programme, for activities to be sent home and to invite whānau/family to evenings or to view lessons, to varying degrees of success.

In the interviews and the surveys ALiM and SMT teachers identified support from home as a factor contributing to the success of the programmes.

Student experiences of the programmes

Overall, students enjoyed their time on the programmes and in some cases could identify activities that had helped their learning. In the post-programme surveys, nearly all students reported positive learning experiences in the programmes. Many, even the younger readers, could identify learning activities that helped their reading, writing or mathematics. Students reported enjoyment of the small group setting owing to fewer distractions. Students felt special to be part of the group and grew more confident. The student surveys showed positive changes in attitudes from the start to the end of the programmes as confidence, motivation, and engagement increased.

Mathematics students said they played more interactive mathematical games, used more materials and artefacts and had more discussions with peers. They thought they had gained confidence and motivation and had learned better than in their usual classroom; and they rated themselves as better at mathematics by the end of the programme. Many literacy students were able to identify particular strategies and learning skills that had been emphasised and to explain how they helped their reading or writing. The strategies and skills were not usually new; rather, they were given more emphasis in the group setting, such as systematically building vocabulary for use in writing. There was evidence from students that the changes the teachers made had helped their learning. Many reading students gave examples of learning improvements, like writing more interesting stories, being able to read higher-level books or understanding mathematical concepts and patterns.
ALiM and SMT teachers reported shifts in students' enjoyment, motivation and engagement over the duration of the programmes. Students reported increasing their enjoyment of the subjects and gaining confidence while on the programmes.

Sustainability

Principals who took part in data collection were convinced about the value of the programmes and many were looking for ways to spread the positive outcomes from the programmes school-wide. All principals' case study schools had either already begun to widen the impact across a part or all of the school or had plans to achieve a wider impact in the school in future. Principals and teachers saw advantages to the programme teacher being a regular member of staff and the classroom teacher of students in the programme. This was owing to existing relationships between students and teachers and the ability to capitalise on programme gains in the classroom more easily.

Many schools began to widen the reach of the programmes for sustainability by linking the programme teacher to other teachers. Programme teachers reported sharing their learning across the school, with many intending to formalise these processes. Teachers saw the knowledge and experience of working with the group as a valuable learning experience and many were taking approaches applied during the programmes to inform practice in their classrooms.

Staff meetings and teacher syndicates were being used as vehicles for influencing other teachers across the school in terms of changing classroom teaching to help students who were below learning expectations. Nearly all ALiM and SMT teachers shared learning and effective teaching practices with other teachers in their schools. Principals of ALiM and SMT teachers felt programme teacher sharing had positively influenced other teachers and students in their schools.

Elements of successful programmes

There was evidence that teachers were practising elements of culturally responsive pedagogy from teachers' self-report of their approaches and their plans for teaching and student feedback. We could not demonstrate causal links between responsive teaching and student achievement or rank the importance of particular teaching approaches. However, there were indicators that some teaching approaches and elements of the programme model were very important in improving student achievement, including the target groups of Māori and Pasifika.

These successful approaches were:

• systematic, detailed and responsive daily and weekly teacher planning
• support, commitment and leadership from the principal
• regular, uninterrupted small-group teaching of at least four lessons per week
• lessons that were more than 30 minutes
• focus on the specific learning skills and concepts required to learn to read and/or write and to learn mathematics
• focus on specific curriculum content at a suitable learning level
• effective and responsive teacher
• regular assessment, feedback to students, planning and record keeping
• teacher and principal enthusiasm and high expectations
• family/whānau participation in helping children's learning
• learning activities that capture student interest, make learning enjoyable and extend their learning
• regular reflection of their own teaching by programme teachers.

Of the above, the crucial factors appear to be detailed planning, lesson length of at least 30 minutes and frequent lessons, focus on particular curriculum content (but no pressure to cover a prescribed amount) detailed and regular feedback to students and teacher expectation and enthusiasm.

More broadly, teachers who were supported were more effective in implementing these successful approaches. Principals needed to be involved in teacher and student selection, support and resources for the teacher. They needed to organise active collegial planning of the programme and arrange the sharing of programme experiences with a student's regular teacher.

With the above factors in place, Māori and Pasifika students experienced success in the programmes and their learning was accelerated. The students were generally positive about the extra help they received and their achievement scores indicated that, on the whole, their learning had benefited. Changes in students' attitudes were consistent with the accelerated learning achieved by the majority.

In summary, the above factors interacted and together constituted effective and responsive teaching.

Concluding comment

This evaluation provides evidence that the additional learning programmes for groups of primary students in mathematics, reading and writing resulted in accelerated learning for the majority of the programme students not achieving National Standards at the start of the programmes.

School principals and teachers demonstrated a sense of accountability to succeed in raising student achievement. They saw benefits in a model that set out a clear goal and parameters for the programmes.

At the school level, principals and teachers worked within the parameters and resources of the programmes to use their own initiative and expertise to promote student learning and meet national level expectations.
Principals and teachers relished working within the high trust model and were empowered when they saw gains in student motivation, engagement and achievement. Programme teachers were keen to share what they had learned with colleagues, who responded positively to local evidence of the progress students could make. Programme students were positive about their experiences, reporting they had valued participating in the programmes and that their learning had benefited. Finally, school leaders saw the need to widen the positive effects of the programme of teaching and learning to other teachers and students across a school and many had already introduced processes to pursue this goal beyond the programme itself.

Overall, looking at the student achievement data and the teacher, principal and student surveys, the reading, writing and mathematics programmes were effective at raising student achievement for students not achieving National Standards. The surveys, case studies and interview also provide evidence for changes in students' attitudes and changes to teachers' practice.

The picture for the literacy programmes is less conclusive than for mathematics owing to the smaller sample of consistent student achievement data available for analysis. However, for both writing and reading, on average, students made progress on the measures provided. Principal, teacher and student surveys and case study feedback suggest these programmes were also effective in raising student achievement for students not achieving National Standards.