Ua Aoina le Manogi o le Lolo: Pasifika Schooling Improvement Research - Final Report

6. Summary of Classroom Instruction

The question addressed by the classroom observations of lessons was, “What are the practices that work (and do not work)?” In this area the focus was on aspects of instruction. An observational tool was designed to sample aspects of instruction, drawing on the extensive research literature on effective classroom instruction including previous research interventions both internationally and locally. Results for these observations were presented for each case study school and were combined with other sources of data including students’ views and school leaders’ views. The case data presentations provide insights at the level of the school. Here we summarise the data collected across the schools.

Because of the design ultimately adopted, we have data from two schools in addition to the Case Study Schools. This summary puts these data together with the Case Study Schools to draw limited conclusions with the extra data. The emphasis is on specific aspects of classroom instruction. These are tentative conclusions given the numbers of teachers and schools represented and extrapolating from them must be treated with great care. However, we indicate the extent to which they are consistent with other research evidence and theory-based predictions.

6.1 Results

A total of 34 individual lessons were observed from 12 teachers in the 6 Focus and Case Study Schools. The schools were chosen on two criteria. The Schooling Improvement schools (both those in the Focus Clusters and those not) were identified as highly effective for the cluster and willing to participate. The non Schooling Improvement schools were identified through professional judgement (using Ministry of Education and network sources) as highly effective with Pasifika students. The Principals were asked to recommend teachers who were willing to participate on the criteria of one highly effective for the school and one (relatively) less effective. Each school interpreted this request in their own way. One high school chose a teacher of a high stream class and a teacher of a low stream class. A primary school decided to choose two equally effective teachers. So the teachers need to be seen as functioning within well performing schools and identified in the context of these schools.

6.1.1 General patterns

The ratings (1-3) have been converted to percentage scores due to differences within and across lessons of completed scores on some instructional dimensions. In seven cases, time ran out to complete a full cycle in either the first or the second cycle of time sampling. Thus, a percentage score of 100% means an average rating of three, 66.7% means an average rating of 2 and 33.3% means an average rating of 1. In the following tables the Primary schools are schools A, C and E. The secondary schools are B, D and F.

Table 83 presents the overall summary for teachers and schools across the three components. The data show several patterns. The general impression from the table is that scores are high in the three components, above an average rating of 2. The teachers’ classrooms and instruction in these schools tend to be rated highly.

A second pattern is that teachers tend to rate similarly across components. The correlations between each of the components ranged between r = 0.66 to r = 0.95 (n = 12 teachers). This suggests that teachers who were rated relatively highly in terms of systematic observations of their instruction were also relatively high in ratings of the way their classrooms were organised and in a global rating of their cultural responsiveness; and the converse is true. However, there were large variations between teachers on each component. The total scores varied from 58.3% to 98.6% (SD = 12.8), meaning there were teachers who scored an average of less than 2 (Teacher A1) and teachers who scored close to an average rating of 3 (e.g., Teacher C2).

Another feature of the table is that the schools varied considerably in the total scores, from one in which the two teachers rated around 2 (School D; M = 66.2%) and one in which the teachers were rated around 3 (School C; M = 94.8%)

6.1.2 Components

In terms of each component, the highest score was for the Features (including ambient, resources and management) and the lowest was for the Instructional dimensions, where the associated lowest standard deviation suggests all the teachers tended to be lower.

Table 84 presents the breakdown for the instructional dimensions based on time sampling. The scores for instructional dimensions were relatively similar across the 12 teachers (with means ranging from 67.5% to 84.4%). Interestingly, the dimension of Teacher Feedback had the lowest score with around an average rating of 2 (67.5%) and the largest standard deviation (SD = 22.7). Teachers varied more markedly in the focus on providing high quality feedback than in other dimensions. Three teachers had a mean rating of 1 (or 33.3%) on this dimension. The highest scores were for a focus on core knowledge which was above an average rating of 2.5 (M = 84.4%) and it had the lowest standard deviation (SD = 15.6).

Primary schools tended to score somewhat higher in their total scores (M = 83.1%) than secondary schools (M = 73.0%). The instructional dimensions contributed to this and on four out of the five dimensions the primary schools scored more highly (the exception was Feedback).

Cultural responsiveness was judged from two elements. One was incorporation of students’ resources (such as their background knowledge) and the other was the positivity of their relationships, including the degree to which they were respectful and conveyed high expectations. The former tended to be judged as lower more often than the latter. In each instance of a difference in rating, the specific use of students’ backgrounds was lower (M = 2.2) than relationships (M = 2.5).

6.1.3 Relationships with effectiveness

Do these patterns relate to effectiveness, measured in terms of achievement? It must be remembered that there are multiple sources of influence on student achievement. Hattie (2009) calculates up to 30% of the variance in achievement is attributable to teachers. In his analysis, 70% is attributable to five other sources including student characteristics, homes, peers, schools and Principals. This means that we would not expect a simple relationship between a light sampling of instruction and student outcomes.

1. Teacher designation
   The simplest analysis is between the observational data and the designation of teacher being high or less high in terms of Principal nominated effectiveness. Note that there are variations in how this was interpreted, and that the teachers were agreeable to being nominated. There was a slight difference between the two groups of teachers (‘Higher’ teacher M = 79.7; ‘Lower’ teacher M = 76.3).
2. Achievement outcomes in classes
   The classroom observation data can be examined in relationship to the student achievement of the classrooms of those teachers. These data are shown in Table 85. Note that for Schools A, B, C and D the achievement data are comprised only of Pasifika students. For Schools E and F, the achievement data are comprised of all students. The latter schools were non-Case Study Schools and their data were not sufficiently differentiated.
   
   We have taken the position that judgements of achievement should include consideration of both rate and level. This is because, at higher levels, less gain may be possible due to a number of constraints. These include ceiling effects and the greater difficulties in terms of immediate gains of teaching to extend readers and writers, rather than bringing substantially below average students closer to an average (which itself also is confounded with regression to the mean effects). Also, a high level may indicate sustaining achievement patterns over a longer term and lower rates of gain may be indicative of this sense of maintaining steady progress.
   
   Putting these two indices together, there is little consistency in the patterns in Table 85 to suggest that differences between individual teachers on the measure of classroom instruction are systematically associated with achievement differences in classrooms. While there was little pattern around individual variations, there were two different relationships when teachers were grouped according to overall patterns of achievement gains or levels. The mean score for teachers whose classes had close to or greater than expected gains in 2008 (n = 8) was 74.1%. On the other hand, those classrooms in which consistently lower than expected gains occurred (n = 4) had a mean score of 85.8%. Both of the latter were a mixture of secondary and primary classrooms. In contrast, when teachers were categorised according to the level of achievement for the pair of teachers in a school, those who had higher levels had a mean score of 82.8% and those who had lower levels had a mean score of 73.2%.
3. Achievement outcomes in schools
   Using the same indices, the achievement data for schools can also be examined for relationships with the teachers’ instructional measures. There is a weak (but perhaps better termed ‘emerging’) pattern when these relationships are examined. Schools with both high levels and consistent rates of gains had teachers who tended to score higher on the measures of their classroom instruction. This is illustrated particularly by the primary schools ‘C’ and ‘E’; but is suggested in the data for the secondary school ‘B’. The corollary was that lower instructional scores were associated with lower levels although rates of gains might be high in one year (e.g., secondary school ‘D’). The more detailed case studies are consistent with this characterisation of achievement patterns in these schools and relationships with instruction.

6.2 Conclusions

6.2.1 Overall lesson quality: coherence between components

This overall association between the teachers’ instruction and gains is not surprising given the literature used as a basis for the tool. Previous interventions associated with gains for Pasifika students in primary schools (Lai, McNaughton, Amituanai-Toloa et al., 2009) and Māori students in secondary schools (Bishop et al., 2007) have targeted the overall quality of teaching in one form or another.

The patterns suggest a particular sense of consistency in teaching is important. A high scoring teacher in one instructional area was generally effective in each of the areas of concern, although, as noted earlier there was substantial between-teacher variation on total scores and on sub scale scores.

An important (tentative) conclusion is that while there is evidence presented in the case studies of teachers with high scores in one or other of the three areas, the evidence suggests that being high in each area is likely to be important. That is, teachers who are well organised, who have high quality instructional practices and who have high levels of cultural responsiveness generally tend to be more effective in terms of consistent achievement outcomes. Each of these may be necessary, although the research is not able to determine such an outcome. But it appears that any one by themselves may not guarantee effective teaching.

6.2.2 Instructional coherence over time and between teachers

The two highest performing primary schools in terms of both overall levels and rates of gain (School ‘C’ and School ‘E’) had teachers who, over three lessons, scored the highest on the instructional tool and were very consistent and similar in their instructional approaches. While this judgement is very dependent on how the schools selected teachers, the consistency over time and between teachers (teaching at different levels) is striking. Both schools were primary schools but they differed in terms of whether they were in Schooling Improvement and their special nature. Recent large scale studies which have used classroom observations (Croninger & Valli, 2009) and teacher log books (Rowan & Correnti, 2009) report that the overwhelming variation in teachers’ instruction is within the same teacher over different lessons, and this variation within teachers is even greater than between teachers. Indeed, the authors of these studies recommend observing at least 6 - 8 lessons per teacher (Croninger & Valli, 2009) or collecting at least 20 logs over a year (Rowan & Correnti, 2009) to gain enough samples to differentiate well between teachers. Having said that, there is also substantial variation between teachers – even among the teachers at the same year level in the same school – which covers a wide range of teaching practices not only in reading comprehension but also writing and word analysis. But in very successful Schooling Improvement models (though success can be quite restricted here, but defined in terms of predictable outcomes on specific aspects of literacy) the wide variation between teachers and within the same teacher over time is reduced.

These relationships were not apparent with the 6 teachers at the three secondary schools. None of the secondary schools had greater than expected gains for Year 9 students. The most consistent secondary school (as judged by the similarity of scores between teachers) was also the school with the highest average for secondary schools, but this school had the lowest gains on asTTle reading. However, one secondary teacher made greater than expected gains. This teacher had a relatively high overall score among the secondary teachers (70.8%).

The tentative conclusion is that well-designed instructional approaches consistently used by the same teachers and consistent between groups of teachers is associated with greater effectiveness. The converse is that inconsistency in approaches over time and between teachers may be a barrier to greater effectiveness. This may apply primarily to primary schools.