Executive Summary

The Purpose of the Project

The current project focuses on the effectiveness of Schooling Improvement initiatives for Pasifika. The purposes were to identify the practices that work to raise achievement and close the gaps for Pasifika students especially at the classroom, school and cluster levels; to find out how effective existing Schooling Improvement initiatives are in raising achievement for Pasifika students; and to provide information to help existing and new initiatives to improve their effectiveness for Pasifika students.

Two overarching research questions were asked:

1. What works in schools for Pasifika students and under what conditions?
2. What are the barriers to schools achieving positive learning outcomes for Pasifika students?

The Process

In this report we evaluate the initiatives using a three-step process. First we summarise the general achievement data across nine interventions that have high numbers of Pasifika students. This is followed by a close analysis of a Focus Cluster, in which we use detailed statistical procedures to examine features of students such as language status, gender and ethnicity to answer questions about the patterns of effects for Pasifika students. Essentially this section provides some insights into the question of whether interventions are meeting the needs of Pasifika students or if there are limited areas of effects.

This is followed by systematic case studies that provide quantitative and qualitative data on several general hypotheses at the level of school effects. The hypotheses were: that schools that are more connected with their communities will generally be more effective; that schools that have well embedded inquiry practices and have a heightened sense of collective efficacy will be more effective; that schools in which instruction has specific features of quality and is culturally responsive (developing distinctive approaches for Pasifika learners) will be more effective; and lastly, that there will be some attributes of students which are associated with greater gains and levels of achievement, probably relating to language status and familiarity with the New Zealand educational system. Also, that students' beliefs and values relating to teaching and learning will provide further evidence of the features of schools that are likely to be more effective. In this last section we add the voices of students, their parents, teachers and Principals to provide rich and integrated tests of these hypotheses.

In addition to the above, because we were able to survey students, teachers and leaders from clusters, we also have general descriptions of features of language status across schools, aspects of leadership patterns across schools and aspects of teachers' pedagogical content knowledge across schools.

Our Findings

1. Data systems across and between schools and clusters vary

The question of the general effectiveness of the nine Schooling Improvement initiatives could not be answered at a generalised level. The reasons, detailed in an accompanying paper 'A systems level approach to learning from aggregated achievement data: Implications for policy' (Lai, McNaughton & Amituanai-Toloa, 2009), are to do with the state of databases, the management of those databases and the uses of the databases. Three clusters had sufficiently robust data which were longitudinal and could meet criteria of accuracy and reliability. Interventions generally will need much better managed databases than currently exist and recommendations about guiding principles and systems which would enable these to develop are contained in the paper.

2. Schooling improvement can work for Pasifika, but progress is slow

The data from three clusters with varying types of databases for Years 4 - 8 in reading comprehension (one of whom was also a Focus Cluster) show that clusters vary in effectiveness. One cluster made expected gains over a year. Two of the clusters made accelerated gains (over and above expected gains) during individual school years with average effect sizes (d) of between 0.2 and 0.5. Over two years, one cluster had an effect size (d) of 0.5. Clusters had varying drops associated with summer (the 'Summer Learning Effect') which meant that in two out of three clusters, continued gains were slowly, cumulatively, enabling achievement levels to reach average bands. A rigorous educational (and equitable) criterion used to judge effectiveness shows that more gains are needed to reach a full match with a nationally expected distribution in achievement (McNaughton & Lai, 2009). One cluster is close to this match.

3. Similar gains occur for Pasifika groups, but there are gender differences

In the overview of clusters there was no evidence that different Pasifika groups were substantially different in their responses to the programmes, either in terms of rates of gain or levels. However, while Pasifika students make similar rates of gain to others, their achievement levels tend to be lower. There was also evidence that there were substantial gender differences in the levels achieved, although rates of gains can be similar (creating a progression which is like 'parallel tracks'). What this means is that the focus on Pasifika groups needs to have, even within this differentiation, a possible differentiation in instructional focus for boys.

A tentative conclusion from the first step in the results, then, is that the most effective Schooling Improvement projects can 'work' to make a real educational difference. However, the progress is slow and cumulative, and clearly from the descriptions of the projects requires substantial resourcing and long-term focus.

4. There are school by school differences

More detailed analyses of the data from one 'Focus Cluster' confirmed these general results. There were gender differences in the levels achieved although not in the rate of gains and while different Pasifika groups achieved at similar rates, Samoan students tended to score at higher levels (but not always). The more detailed analyses showed differences between classrooms (although all but a few classroom made accelerated gains during years), and at the school level (over two years the effect sizes (d) across schools varied from 0.30 to 0.77). From these analyses we found that there were high gain and low gain schools within the cluster and it will be important for additional research to further tease out the features of schools associated with these differences.

5. Different patterns emerge with gain and level analyses

Two sorts of statistical models were developed to further explore patterns. These were 'gap difference' models which explored patterns of achievement over time in terms of rates of gain, and 'level difference' models to examine patterns in overall mean levels of achievement. There was no evidence from the 'gap' models of differences in achievement due to language status (rates of gain were not different for students with different home languages or who identified their first language differently), country of birth, or gender. However, a student's starting level predicted the rate at which gains were made – higher gains were made by students who were in the lower stanines. But over time these differences disappeared.

What this means specifically for the Pasifika Schooling Improvement is that judgements about effectiveness need to be made over more than a year and it is very important to be able to examine how higher achieving students fare in programmes.

For the 'level difference' models; gender, time lived in New Zealand, home language, and school were associated with significantly different levels of achievement. Overall, the mean scores for the students that spoke mainly Pasifika languages and those that spoke two or more languages (Pasifika language as well as English) at home were significantly lower than that for the mainly English-speaking students. The mean scores for females were significantly higher than that for males. With respect to the length of time lived in New Zealand, the mean scores for those that had lived in New Zealand for more than five years and those that were born in New Zealand were significantly higher than those that had lived in New Zealand between one and five years. The mean levels of achievement differed significantly between schools, and part of this difference could be due to the different year levels (i.e., cohorts) that the schools catered for.

6. Both rate and level criteria need to be used to judge effectiveness

These two sets of models underline an implication for further evaluations of interventions. There is a need to have two related criteria for judging educational significance of interventions, especially in terms of equitable outcomes (McNaughton & Lai, 2009). The tests of effectiveness should be, firstly, whether clusters are achieving accelerated rates of achievement, and secondly whether they are shifting distributions of achievement to match national expectations. The former sets the test at being about making more than just a normal rate of progress because that means perhaps higher levels but parallel tracks of achievement. The latter sets the test as achievement for students in the schools being no different from the distribution of the achievement for students nationally (i.e., the same proportions of low, middle and high achieving students).

7. Case studies of schools add detailed information

The case studies added more qualitative evidence to these outcomes. In terms of quantitative data there were systematic observations of classroom instruction which included measures of the quality of instruction as well as cultural responsiveness judged across two levels (positive affect and incorporation of students' cultural and linguistic resources). The qualitative data include interviews with Principals, Literacy Leaders, parents and students. The results modify some of the conclusions developed at a cluster level to a school level. What they contribute also is both the indicators of success and by corollary what doesn't work for Pasifika learners.

8. Greater effectiveness is associated with a range of home-school connections

The case studies suggest greater effectiveness is associated with practices between schools and their communities that involve sharing knowledge and resources with a degree of reciprocity, with the specific outcome of increasing parent involvement, which may then impact on students' motivation and academic skills. Putting together the evidence across the various sources, three conclusions were suggested: (a) parents' understanding of information about their own individual child's learning and achievement, both strengths and weaknesses as well as progress across time, can increase parental impact on motivation and skills; but (b) parents need guidance and advice on both motivational and academic involvement; and (c) parents are keen to receive advice and they have ideas about practices both at home and at school that could contribute. The latter may or may not be effective but they are important ideas that can be the basis of reciprocity – an example is the role and forms of homework. The findings of substantial (but variable) Summer Learning Effects underscore the need to more deliberately develop and share practices between school and family settings.

9. Coherence within a school at all levels is important to effectiveness

Our hypothesis about developing inquiry practices that are evidence-based and outcomes-focused was well illustrated in the case studies. Each of the Case Study Schools was engaged in clusters of Schooling Improvement which focus on inquiry and it would be expected that these practices would be in place. But the schools varied in how deeply ingrained, extensive and coherent their practices were. The patterns suggest that greater coherence will be associated with greater effectiveness. Coherence matters: (a) between levels in the schools, across members of the school professional community, and between different instructional parts including teachers; (b) for new members of the system so that detailed induction as a member to share values and skills is important; and (c) so that all programmes – existing and new – are integrated into the inquiry practices and are 'tested' by the inquiry process. The coherence between teachers appears to be especially significant so that there is consistency in pedagogical approaches as well as in focus and goals.

10. Generally effective teaching practices are present which have been adapted to be responsive to Pasifika students

There was some ambiguity detected at a school level in how terms such as 'cultural responsiveness' and 'Pasifika pedagogy' are used, and there is a need to clarify more specifically what is meant by these terms. However, in general, the evidence across schools was that the schools, to varying degrees, taught using generically effective forms of instruction, but adapted them to be applicable to and responsive to different Pasifika learners. The specific measures from classroom instruction, when examined at a teacher level, were not related systematically to either rate of gain in classroom or achievement levels. However, when combined and averaged across schools, there was evidence that the teachers' measures of instructional quality and cultural responsiveness were associated with overall school achievement. The highest scoring schools had higher levels and moderate to high rates of gain. This suggests that coherence in instruction and cultural responsiveness in schools may be more important than individual teachers' expertise.

It is possible to identify elements of what the model is that the schools are moving towards. Clearly, schools are effective to the degree that they use known attributes, such as explicit instruction for both basic knowledge and strategies, high levels of elaborative talk and inquiry are promoted, there is a focus on the language needs including those for vocabulary and there are well-developed forms of feedback. Running across these is the need to be clear and explain goals and needs for learning. On the other hand, specific dimensions of cultural responsiveness are clearly part of more effective teaching. The twin dimensions of positive relations and incorporating students' resources were identified to varying degrees in classrooms. Importantly, these themes were echoed by the students. Pasifika pedagogies that are being developed in these schools, in the sense of being adapted to Pasifika learners, draw on background knowledge including topics and event knowledge, language patterns and activities, and the students and teachers are aware of this. But in addition, there is the dimension of a strong emotional relationship which, together with the instructional attributes, has elements of being both rigorous and challenging as well as being respectful and empathetic. The former includes high expectations and the latter a Pasifika sense for the students of education being service-oriented and, from the teacher, positive affect expressed with devices such as Pasifika-oriented humour.

11. Students are clear on what instruction works for them

The student voices were very similar to those from the Te Kotahitanga project (Bishop, Berryman, Tiakiwai, & Richardson, 2003) but the adaptations suggested above include a need for teachers to provide a strongly supportive base enabling the students to take risks and be critical and engaged. For example, students want teachers to break down the questions for simplicity including clearer explanations and challenges in their work. The evidence supports previous research showing Pasifika learners to be generally highly motivated to succeed and to learn across the schools. Students are more consistently positive and motivated at primary schools. This is true generally, and like the more general need Schooling Improvement will need to consider how to increase engagement and emotional connection at secondary levels (Paris & McNaughton, in press).

12. Parents want to know how they could support their children

The overall parent interviews strongly echoed the case study conclusion: (a) parents need guidance and advice on both motivational and academic involvement; and (b) parents are keen to receive advice and they have ideas about practices both at home and at school that could contribute

13. Being bilingual is not an impediment to academic achievement of Pasifika learners

Looking at language status from the point of view of achievement, there was no evidence from the Case Studies that having two or more languages is an impediment to high success either at primary or at secondary. The patterns of development may look different for those students with a Pasifika language or both a Pasifika and English language background in the earlier years, compared with English only students. But from the middle and upper primary and into the secondary years the sense is that bilingualism may (under important conditions not tested here, such as level of bilingualism) lead to similar outcomes (as having a strong English-only status), and in a wider sense confer other advantages.

14. The need for induction for newly arrived Pasifika students

There is perhaps an obvious suggestion in the data that more familiarity with the New Zealand education system is advantageous and we take this to mean that for newly arrived students there is a need to have very explicit induction and support to develop the knowledge and skills required for local schooling.

Three other documents were generated from this report: the first, a Policy Paper (Lai, McNaughton & Amituanai-Toloa, 2009) to assist the Ministry with further Schooling Improvement policy development for Pasifika; the second, a Summary Report which succinctly summarises the main findings of this study but without technicalities (Amituanai-Toloa, McNaughton, Lai, & Airini, 2009b); and the third, a Communication Template which provides guidance to schools about principles and practices to help support communication between parents and the school (Amituanai-Toloa, McNaughton, Lai, & Airini, 2009a).