Ua Aoina le Manogi o le Lolo: Pasifika Schooling Improvement Research - Final Report

8. Conclusions

The overarching research questions in this report were:

• What works in schools for Pasifika students and under what conditions?
• What are the barriers to schools achieving positive learning outcomes for Pasifika students?

The research questions specific to Schooling Improvement were:

• Are the nine existing Schooling Improvement initiatives with significant numbers of Pasifika students bringing about significant gains in achievement for Pasifika students, and if so, what are the gains from each initiative and each school within the initiatives?
• What, if any, are the differences between the gains seen in the Schooling Improvement initiatives for different student groups within Pasifika (ethnicity, gender, generation in New Zealand, language)?
• If there were any significant positive gains identified in response to questions 3 and 4 above, what appears to have contributed to those gains?

These questions have been answered in a variety of ways and in this conclusions section we summarise those answers with a brief commentary.

8.1 General Effectiveness of Schooling Improvement Projects for Pasifika students

This question cannot be answered at a generalised level. The reasons are detailed in the body of the report and in the accompanying paper ‘A systems level approach to learning from aggregated achievement data: Implications for policy’ (Lai, McNaughton & Amituanai-Toloa, 2009). A full answer to this question will need to draw on much better databases than currently exist and recommendations about guiding principles and systems which would enable these to develop are contained in the paper.

From the data from three clusters with varying types of databases for Years 4 - 8 in reading comprehension (one of whom was also a Focus Cluster) it appears that the following conclusions are possible. The first is that clusters vary in effectiveness for their Pasifika students, which is not a surprising outcome given that programmes in the United States can be shown to have differential effects (Borman, 2005). Two of the clusters made accelerated gains during individual school years with average effect sizes (d) for the clusters of between 0.2 and 0.5. Over two years Cluster A had an effect size (d) of 0.5.

The two clusters that made greater than expected progress within years had varying drops associated with summer (the ‘Summer Learning Effect’). This meant that two out of three clusters showed evidence that continued gains were slowly and cumulatively enabling average achievement levels to reach average bands. However, the levels were still low and the data show that across clusters more gains are needed to reach a full match with a nationally expected distribution in achievement, which is the most rigorous criterion we can apply (McNaughton & Lai, 2009). Overall, then, the most effective Schooling Improvement projects can ‘work’ to make a real educational difference. However, the progress is slow and cumulative, and clearly from the descriptions of the projects requires substantial resourcing and long term focus. Again, these are not surprising conclusions given the international picture (Borman, 2005). But it is important at least to be able to make a qualified positive answer to the question.

The second conclusion is that in the overview of clusters there was no evidence that different Pasifika groups were substantially different in their response to the programmes in Schooling Improvement either in terms of rate of gain or levels.

Thirdly, there is evidence that there are substantial gender differences in the levels achieved, although rates of gains can be similar (creating a progression which is like ‘parallel tracks’). The gender differences are well known nationally also (Crooks & Flockton, 2005b). But what this means is that the focus on Pasifika groups needs to have, even within this differentiation, a possible differentiation in instructional focus for boys.

These general results were supported by the data where available from the clusters with less evidence. But we also examined these questions of overall effectiveness and differential effectiveness in the Focus Cluster (again with a focus on reading comprehension in Years 4 - 9). We used more detailed statistical procedures to provide answers.

In these analyses again there were gender differences in the levels achieved although not in the rate of gains, and while different Pasifika groups achieved at similar rates, Samoan students tended to score at higher levels (but not always). The more detailed analyses showed differences between classrooms (although all but a few classroom made accelerated gains during years), and at the school level (over two years the effect sizes across schools varied from d = 0.30 to 0.77). From these analyses we found there were high gain and low gain schools within the cluster and it will be important for further research to tease out the features of schools associated with these differences (Lai, McNaughton, Amituanai-Toloa et al., 2009).

Two sorts of models were developed to further explore patterns. One set were ‘gap difference’ models which explored patterns of achievement over time in terms of rates of gain over different time points. The second set of models were ‘level difference’ models and these collapsed the averages over time to examine patterns in overall levels of achievement. The overall findings from the former were that across the two ‘gap difference’ models, no evidence of a language effect, country of birth effect, or gender effect was found. A student’s starting level predicted the rate at which gains were made – higher gains were made by students who were in the lower stanines. But over time these differences disappeared. The initial effect on lower achieving students is not uncommon in intervention studies, as is the case in the national LPDP intervention (Parr et al., 2007).

What this means specifically for Pasifika Schooling Improvement is that judgements about effectiveness need to be made over more than a year and it is very important to be able to examine how higher achieving students fare in programmes.

There were complicated school effects in the ‘gap difference’ models. But these are hard to interpret because of the presence of different cohort groups associated with the full primary, contributing primary, intermediate and middle schools in the cluster.

For the ‘level difference’ models where we examined the achievement levels aggregated across four tests, gender, time lived in New Zealand, home language, and school were associated with significantly different levels of achievement. Overall, the mean scores for the students that spoke mainly Pasifika languages and those that spoke two or more languages (Pasifika language as well as English) were significantly lower than that for the mainly English-speaking students. The mean scores for females were significantly higher than that for males. With respect to the length of time lived in New Zealand, the mean scores for those that had lived in New Zealand for more than five years and those that were born in New Zealand were significantly higher than those that had lived in New Zealand between one and five years. The mean levels of achievement differed significantly between schools, and part of this difference could be due to the different year levels (i.e., cohorts) that the schools catered for.

These two sets of models underline an implication for further evaluations of interventions. It is the need to have two related criteria for judging the educational significance of interventions, especially in terms of equitable outcomes (McNaughton & Lai, 2009). The tests of effectiveness should firstly be whether clusters are achieving accelerated rates of achievement and secondly whether they are shifting distributions of achievement to match national expectations. The former sets the test at being about making more than just a normal rate of progress because that would perhaps mean higher levels but parallel tracks of achievement. The latter sets the test as achievement for students in the schools being no different from the distribution of the achievement for students nationally (i.e., the same proportions of low, middle and high achieving students).

The case studies add more qualitative evidence to these outcomes. They also modify some of the conclusions at a school level. What they further contribute is not only the indicators of success but also by corollary what doesn’t work for Pasifika learners.

8.2 Connectedness

The relationship between a school and its families is important (Bronfenbrenner, 1979). The case studies suggest greater effectiveness deriving from practices that involve sharing knowledge and resources with a degree of reciprocity, with the specific outcome of increasing parent involvement which may then improve students’ motivation and academic skills. It is likely that parents’ involvement enhances achievement through both skill and motivational development (Pomerantz et al., 2007). Putting together the evidence across the various sources, three conclusions were suggested: (a) parents’ understanding of information about their own individual child’s learning and achievement, including both strengths and weaknesses as well as progress across time, can increase parental impact on motivation and skills; but (b) parents need guidance and advice on both motivational and academic involvement; and (c) parents are keen to receive advice and have ideas about practices both at home and at school that could contribute. These may or may not be effective but they are important ideas that can be the basis of reciprocity - an example is the role and forms of homework. The findings of substantial (but variable) Summer Learning Effects underscores the need to more deliberately develop and share practices between school and family settings.

8.3 Inquiry and Collective Efficacy

The hypothesis about developing inquiry practices that are evidence-based and outcomes-focused was well illustrated in the case studies. The Lai et al. studies (Lai, McNaughton, & Amituanai-Toloa 2009; Lai, McNaughton, Amituanai-Toloa et al., 2009; Lai, McNaughton, Timperley et al., 2009) suggest the importance of collaborative inquiry, i.e., the role of professional learning communities in inquiring into their own data. Building a sense of collective efficacy in schools is important also because it is a predictor of student achievement (Bandura, 1995). Each of the Case Study Schools was engaged in clusters of Schooling Improvement which focus on inquiry and it would be expected that these practices would be in place. But the schools varied in how deeply ingrained, extensive and coherent their practices are. The patterns suggest that greater coherence will be associated with greater effectiveness. Coherence matters: (a) between levels in the schools, across members of the school professional community, and between different instructional parts including teachers; (b) for new members of the system so that detailed induction as a member sharing values and skills is important; and (c) so that all programmes – existing and new – are integrated into the inquiry practices and are ‘tested’ by the inquiry process. The coherence between teachers appears to be especially significant so that there is consistency in pedagogical approaches as well as in focus and goals.

8.4 Pedagogy and Cultural Responsiveness

There was some ambiguity detected in the data in how these terms are used, and there is a need to clarify more specifically what is meant by these terms. However, in general, the evidence across schools was that the schools to varying degrees taught using generically effective forms of instruction but adapted them to be applicable to and responsive to different Pasifika learners. At a general level, cultural responsiveness is a dimension of generically effective instruction.

The specific measures from classroom instruction when examined at a teacher level were not related systematically to either rate of gain in their classroom or achievement levels. However, when combined and averaged across schools, there was evidence that the teachers’ measures of instructional quality and cultural responsiveness were associated with overall school achievement. The highest scoring schools had higher levels on these measures and moderate to high rates of gain. This suggests that coherence in instruction and cultural responsiveness in schools may be more important than individual teachers’ expertise.

It is possible to identify elements of what the model is that the schools are moving towards. Clearly, schools are effective to the degree that they use known attributes such as explicit instruction for both basic knowledge and strategies, high levels of elaborative talk and inquiry are promoted, there is a focus on the language needs including those for vocabulary and there are well-developed forms of feedback. Running across these is the need to be clear and explain goals and needs for learning. On the other hand, specific dimensions of cultural responsiveness are clearly part of more effective teaching. The twin dimensions of positive relations and incorporating students’ resources were identified to varying degrees in classrooms. Importantly, these themes were echoed by the students. Pasifika pedagogies that are being developed in these schools, in the sense of being adapted to Pasifika learners, draw on background knowledge including topics and event knowledge, language patterns and activities, and the students and teachers are aware of this. But in addition, there is the dimension of a strong emotional relationship which, together with the instructional attributes, has elements of being both rigorous and challenging as well as being respectful and empathetic. The former includes the high expectations and the latter a Pasifika sense for the students of education being service-oriented and, from the teacher, positive affect expressed with devices such as Pasifika-oriented humour.

8.5 Leadership and Pedagogical Content Knowledge

The analysis of the leadership survey revealed no consistent patterns, either across clusters or across schools. Ratings were moderate to high across all sections. We do not report further details in any of the results sections but have summarised the results in Section 4.

There was no significant correlation between total PCK scores and any of the achievement measures (level or rate of gain). It may be that, like the classroom observations, the more significant relationship might be with school level of gains, reflecting the dimension of coherence across teachers. It may also be that the ‘inert’ measures of PCK (pencil and paper tests) are not useful. There was no significant correlation between total PCK scores and any of the demographic indices. There was no consistent pattern of relationship with subsections either, and so we assume this measure of PCK does not accurately distinguish between different forms of effective knowledge. It may be that we will need to design measures which tap ‘enactive’ knowledge (actual reflections in situ) rather than this ‘inert’ form of knowledge to find relationships.

8.6 Pasifika Learners

The student voices were very similar to those from the Te Kotahitanga project (Bishop et al., 2003) but the adaptations suggested above include a need for teachers to provide a strongly supportive base enabling the students to take risks and be critical and engaged. The evidence supports previous research showing Pasifika learners to be generally highly motivated to succeed and to be willing to learn across the schools. Students are more consistently positive and motivated at primary schools (but this is true generally, and there is a more general need to consider how to increase engagement and emotional connection at secondary levels; Paris & McNaughton, in press). If we only look at language status from the point of view of achievement, and putting the conclusion negatively, there is no evidence that having two or more languages is an impediment to high success either at primary or at secondary. The patterns of development may look different for those students with a Pasifika language or both a Pasifika and English language background in the earlier years, compared with English only students. But from the middle and upper primary and into the secondary years the sense is that bilingualism may (under important conditions not tested here, such as level of bilingualism) lead to similar outcomes as having a strong English-only status, and in the wider sense indicated in the Introduction confer other advantages. There is perhaps an obvious suggestion in the data that more familiarity with the New Zealand education system is advantageous and we take this to mean that for newly arrived students there is a need to have very explicit induction and support to develop the knowledge and skills required for schooling.