Executive Summary

Part A of the report is concerned with measuring progress. Chapter 2 concludes that students in the 13 research schools made large gains in writing and more modest gains in reading. In writing, there was a very large average effect size gain of 1.28 while, in reading, the effect sizes were more moderate with an average effect size of 0.46 in the first year of the project and 0.49 in the second year. While the average effect sizes are a little lower than those reported for the national cohort, the relativity between gain in reading and writing is similar.

With respect to differential achievement by ethnic group, in reading, there was a difference in performance of the major ethnic groups at both points in time. However, all groups made significant progress in terms of gains in total score and stanine level from Time 1 to Time 2 and there was no difference in extent of gain. In writing, the median score at Time 1 was barely different across ethnic groups. At Time 2 there was a significant difference between groups with both NZ European and Other significantly higher than Māori  or Pasifika. However, this may be a function of a very small sample of schools as in a school that made excellent progress, all groups progressed equally.

There were no significant differences by gender in reading achievement scores in this sample or differences in rate of progress. In writing, however, females scored significantly better at both points in time but both girls and boys made significant progress and there was no difference in the rate of progress.

A discussion of problematic issues associated with the use of gain scores to measure progress is presented in Chapter 3. It concludes that, despite reservations in the somewhat dated literature, gain scores can be as reliable and valid as other measures. However, significant drawbacks associated with using STAR as an indicator of progress are identified. Sizeable ceiling effects are shown to occur. These appear at year 5 and are particularly marked at years 6 to 8. The reliability of difference scores for STAR appears low.

Part B of the report deals with teacher knowledge, beliefs and practices. The level and extent of growth in teacher knowledge and its relationship to student progress are examined in Chapters 4 and 6. The first of these, Chapter 4, presents a way of exploring teacher pedagogical content knowledge (PCK), that is, their knowledge of their subject (reading or writing) from the point of view of teaching it. While there are issues around measuring this knowledge (and these are discussed in the chapter), the results show growth in knowledge of writing, particularly in teachers’ ability to give feedback to students at higher levels of text. Most significantly, results demonstrate that the level of a teachers’ knowledge of writing is significantly and strongly related to the extent of their students’ progress in writing.

Chapter 5 examines the conditions under which teachers promote student self-regulation of their learning. It is argued that self-regulation depends on students being clear about their learning goals, knowing what success looks like and receiving feedback that assists them to reach their goals. Twenty writing lessons for 15 different teachers were observed and, at the end of each lesson, students were interviewed about their understanding of the above features. The observations took place at three phases of the professional development. The first observations were undertaken prior to the professional development, the second before the end of the first year, with the third at the end of the second year.

Teachers showed substantive change in the extent to which they shared learning aims, success criteria and directed their feedback to the learning aims. Students’ responses were very different in the lessons of those teachers who were explicit in these lesson features and those who were not. When teachers were explicit, interviewed students referred to the deeper features of writing when asked what they were learning about writing, what success looked like and what their teachers told them to work on in their writing. When those teachers were less explicit, the students typically referred to learning aims and success criteria in terms of the surface features of spelling, punctuation, neatness and length.

Chapter 6 presents data with respect to teachers’ knowledge of the use of evidence in decision-making. This includes knowledge of the principles of evidence-informed decision-making and knowledge of how to interpret data. Significant progress was demonstrated by teachers in terms of accurately referencing data when drawing inferences about student needs from reading or writing assessment data. However, the level of this knowledge did not relate directly to student progress as it is argued that knowledge of the principles of using evidence in decision-making need not mean placing value on its use. Nor does the ability to interpret data mean that a teacher has the pedagogical content knowledge needed to apply that interpretation.

The last chapter in this part of the report, Chapter 7, presents a series of findings from teacher self-perceptions of both their role and their learning. Teachers’ beliefs about how much influence they had on student achievement changed significantly over the course of the project. They attributed to themselves increased influence on student achievement while reducing the amount they thought was attributable to the home and the child him/herself. Teachers also reported increased satisfaction with the level of achievement of their students. Similarly, their self-reported confidence level with respect to a number of dimensions of pedagogical content knowledge increased significantly. Finally, the results of a questionnaire designed to enquire as to whether teachers thought their skills were improving and as to whether they felt supported and not overwhelmed yielded positive reports.  Teachers reported they were learning new material; they anticipated learning more and the time was well spent. It is of some concern, however, that the patterns of ratings seem not to be related to improvements in student learning and this aspect is something that needs further investigation.    

Part C, the third section of this report, comprises four chapters that focus on conditions that have been identified that are more or less effective in promoting teacher learning. Chapter 8 outlines a single school case study illustrating how one facilitator combined the challenges of meeting the four project outcomes with a needs analysis approach to promote teacher learning.  Initial data were collected through the needs analysis tools used in all research schools, including observations, interviews and questionnaires. A follow-up visit from a member of the research team four months after the school began its involvement with the contract involved further interviews and observations.

The initial classroom observations showed teachers to have carefully prepared writing lessons with instruction focused primarily on motivating the students to write. Instruction about writing during the lesson related primarily to surface features. Student interviews established that all knew what it was they were supposed to be doing, but did not understand what it was they were learning about writing, or the features of effective writing beyond surface features of spelling, length, neatness and punctuation. Their understanding of feedback received also related to surface features.

When providing feedback to the teachers, the facilitator summarised the teachers’ practices, elicited the beliefs on which those practices were based, and related them to the students’ responses. When the teachers realised that their students did not understand the learning aims of the lesson or success criteria for their writing, they decided that they needed to be more explicit in their instructional approach. These student interviews provided the catalyst for change. Other information from the scenarios on the teachers’ pedagogical content knowledge and student asTTle scores helped the teachers to further define their learning needs.

During the next four months, the teachers engaged with relevant readings, classroom observations and staff meetings focused on building the knowledge and skills to make their teaching of writing more explicit. At the end of this time, further classroom observations and student interviews showed they had been successful and student achievement improved substantially during these four months.

Chapter 9 provides reflections on the national facilitators’ learning journey. By the end of the two years four themes were evident in their open-ended discussions of the most significant learning for them. These themes included, in descending order of frequency, the importance of addressing leadership and change management issues, the need to have structured conversations with teachers and leaders, key issues around the development of their content knowledge, and how to exit from schools at the end of the contract in ways that support ongoing learning.

Research data collected over the two years on facilitation showed that this journey had not been easy. At the end of the initial needs analysis phase, it became evident that some were using the data primarily for their own ongoing planning to shape the professional development activities, rather than using the information to co-construct a needs analysis with staff so that they understood their own learning needs and help to construct what needed to happen to meet them.

A few months later, facilitators were asked to write how they would respond to a hypothetical typical scenario in a school. The majority of the responses indicated the impact of the emphasis of the project on facilitating deeper professional learning by examining the beliefs on which practice is based, making decisions on the basis of the evidence in relation to student needs or the impact of practice on students, and promoting student learning through improved knowledge on which to base more effective teaching. However, when facilitators were asked to articulate exactly what they would say in these situations in the follow-up discussions, it became apparent that most restricted their input to asking questions. They were careful not to give their own opinions directly. Although the questions, as facilitators formulated them, were probing of teacher and literacy leader beliefs, at the same time they were highly leading in the sense that the facilitator’s reasons for asking them and suggestions for how to improve decision-making were embedded within the questions. The consequences of this style of facilitation in terms of under-utilizing their expertise was discussed with them with some showing a more co-constructed approach to their ongoing work.

Chapter 10 examines how teacher learning can be promoted within professional learning communities by examining the evolution of teacher meetings over the two years. All participant groups gave significantly higher ratings to the usefulness of meetings to assist teachers to teach those students they found challenging. Unfortunately, with the attrition of schools over the research period, it is difficult to make any general statements about change from the meeting observations and interviews. From the available evidence, it appeared that one school made substantive changes in their use of evidence of student outcomes to reflect on the effectiveness of their teaching and to plan future units. However, little change appeared in the observed meetings of the other two schools.

The final chapter in this section, Chapter 11, identifies ways to promote teacher learning through structured learning episodes. Field notes or taped transcriptions were taken of 18 different structured learning episodes involving twelve different teachers. Nine episodes involved observation and feedback, seven of meetings and two of a teacher observing others teach. The teachers’ reactions to these episodes were variable. No particular activity was more successful than any other, rather it was what happened within that activity and how well it related to the particular teachers’ learning needs that was more important.

Three interrelated principles were identified that were associated with teachers responding positively to the activities. The first was the principle of negotiation and co-construction as ways to promote engagement and learning. Whether it is the purpose of a particular activity, the form of the activity, the learning the activity was supposed to promote, or the meaning of particular data, each aspect needed to be negotiated and co-constructed with participants. Ongoing checking with teachers was an essential part of the process. Associated with this principle is the second, that of engaging teachers’ existing theories about students and how to teach them effectively. This principle is important in the activation of prior learning so that new learning can build upon prior learning. In some situations, change may depend on challenging existing theories and creating dissonance with particular positions. In the absence of theory examination, new practices are typically overlaid on previous practice, with superficial, rather than substantive, change evident. A third related principle was that of promoting self-regulation. When teachers have participated in setting personal goals, understand how to monitor their progress towards them and have the support to make appropriate changes, they are more likely to engage at a deeper level and sustain change.