Research into the implementation of the Secondary Literacy Project (SLP) in schools Publications
This is the second of two reports on the SLP and it provides results of a probe study to investigate what an optimal implementation model for professional learning in literacy might look like, with the aim of informing the ongoing delivery of professional development in secondary literacy.
Author(s): Woolf Fisher Research Centre, University of Auckland.
Date Published: April 2013
The Secondary Literacy Project (SLP) is a Ministry of Education funded professional development initiative in cross-curricular secondary school literacy (reading and writing). Each participating school received support, including professional development support and funding, over a two-year period. Thirty schools participated in 2009–2010 (Cohort 1), and another thirty in 2010–2011 (Cohort 2). The Woolf Fisher Research Centre is responsible for the national coordination of SLP. External professional development support is delivered via regional School Support Services.
The overarching aim of SLP is to increase the achievement of underachieving Year 9 and 10 students in reading and writing, to develop the kinds of sophisticated, subject-specific literacy skills and knowledge they need in order to succeed at school and beyond. There was a specific targeting of underachieving Māori and Pasifika students. For Māori students SLP was designed to give effect to Ka Hikitia – Managing for Success. For Pasifika students SLP was designed to give effect to Pasifika Education Plan, to improve outcomes for Pasifika peoples in New Zealand and increase Pasifika presence, engagement and achievement.
The expected outcomes of the project, over three years, were to raise student achievement in literacy in Years 9 and 10, particularly for underachieving Māori and Pasifika students; increase leaders’ and teachers’ knowledge and skills for evidence-based practice; enhance leaders’ and teachers’ knowledge of effective practice; and develop effective professional learning communities that promote ongoing inquiry into the effectiveness of literacy teaching and learning, professional learning, collaborative problem solving, and reflective practice. The measure of literacy achievement was e-asTTle Reading1 and all SLP schools were required to assess all Year 9 and 10 students using e-asTTle Reading at the beginning and end of each year. Because of difficulties with the tool in 2009, judgements about effectiveness of SLP overall and relationships with implementation features are very tentative.
Purpose of the Research
The overarching aim of SLP is to increase the achievement of underachieving Year 9 and 10 students in reading and writing, specifically targeting underachieving Māori and Pasifika students. The expected outcomes of the project, over three years, were to:
- Raise student achievement in literacy
- Increase leaders’ and teachers’ knowledge and skills for evidence-based practice
- Enhance leaders’ and teachers’ knowledge of effective practice
- Develop effective professional learning communities that promote ongoing inquiry into the effectiveness of literacy teaching and learning, professional learning, collaborative problem solving, and reflective practice.
Specific purposes of the Research into the Implementation were to:
- Test the fidelity of the implementation of the SLP model and the significance of variations in implementation of the model in terms of the intent of SLP
- Develop theoretical understanding of the processes of implementation across contexts
- Enable refinement and modification of the model for SLP from evidence of implementation.
The associated research questions were:
- What is the (theory of the) generic model as intended?
- What are the variations to the model as implemented at facilitator and school levels?
- What explains the variations at different levels? To answer this question possible sources of influence to be investigated included: (a) beliefs, knowledge and goals relating to the model including the purposes and foci for underachieving students, particularly in schools with low achievement levels for Māori and Pasifika students; (b) attributes of inquiry at each level including problem solving and needs-analyses in schools; and (c) properties of schools including leadership and management.
- What are the relationships between the SLP model as implemented in the schools and achievement results across schools for underachieving students, particularly in schools with low achievement levels for Māori and Pasifika students?
SLP Project Design
SLP was intended as a school-wide literacy project. It aimed to improve student literacy achievement across the curriculum for underachieving Year 9 and 10 students, specifically targeting underachieving Māori and Pasifika students. It was intended that this would be done through the development of strong literacy leadership, more effective literacy teaching across the curriculum and more effective school leadership and organisational structures.
These were built through:
- A process of inquiry that was implemented at each level of the implementation model
- An associated practice and knowledge-building process to develop the knowledge base to inform teaching practices and inquiry
- Organisational and leadership structure conducive to enhancing professional learning communities
- Mechanisms to promote coherence
- Cascading implementation.
In both the SLP project overall and in this research component, we have taken a specific stance in relation to the focus on underachieving Māori and Pasifika students. Our stance has been that the schools selected for SLP have Māori and Pasifika students whose achievement levels and distribution are not well matched to nationally expected levels and distributions. This means the Māori and Pasifika students at these schools can be considered, as a group, as ‘underachieving’ (although we would prefer terminology that focused on the school, eg, the students are ‘underserved’). We have not focused on Māori and Pasifika students who are underachieving by comparison with peers at their schools, but rather nationally. Schools used their own data to identify how their Māori and Pasifika students were achieving relative to national norms and distributions. They also used their data to identify where their own Māori and Pasifika students were in relation to other students in their school, and again in turn relative to national patterns.
The project took place over three years. The first cohort of thirty schools participated in 2009–2010, and the second cohort of thirty schools participated in 2010–2011. The first year of the project (2009) was devoted to testing and refining the implementation of the model at the school level. The second year of the project (2010) aimed to refine a theory of an Optimal Model for implementation which was then used to inform the professional development of both cohorts of schools.
The development and testing of what we have called an Optimal Model was an iterative process using multiple sources of evidence. Having collected initial data and from our review of the extant literature, we proposed and refined several dimensions of what the literature suggested was a generic model for examining the implementation of SLP. Values and features of these dimensions were examined against the evidence of achievement outcomes. The Optimal Model was developed using the dimensions from the generic model and testing these to produce the best values given the evidence.
Data Collection and Analysis
The evidence base came from four sources. Questionnaires were issued to Literacy Leaders (LLs) and Literacy Facilitators (LFs). Four Case Study Schools were selected, and interviews and in-depth analyses of their data provided a second source of evidence. The mid-year progress reports prepared by LFs were also analysed as an evidence source. Student achievement data (e-asTTle Reading) were also collected and analysed. The results of multiple analyses are summarised in relationship to the four research questions.
Summary of Findings
The Generic Model: Initial Implementation (2009) and development of an ‘Optimal Model’
In essence, the implementation model was a cascading model of influence in which the National Coordination Team worked with the Ministry of Education personnel to design and implement SLP. The National Coordinator worked directly with regionally-based School Support Services LFs who in turn worked with designated LLs in individual schools. They worked with Focus Group teachers who activated new knowledge and practices in their classrooms. The original Focus Group concept was defined as a small group of (approximately 12) teachers to whom the LL delivered more intensive professional development and for whom funding for release was provided. A deliberate ‘ripple’ effect within a whole-school approach was designed by which Focus Groups and LLs would take new knowledge and practices to the rest of the staff. In practice, what occurred at each layer was not discrete. There was knowledge transfer both ways at each ‘interface’ as would be expected in discussion and professional development settings. For example, regionally-based LFs sometimes worked directly with teachers as a means of supporting LLs. Also it was not as hierarchical as it is presented for the reasons already noted, i.e., there was collaboration and exchange of knowledge and practices between parties involved at each layer.
The model had a focus on inquiry at all levels. It aimed to develop the expertise of LLs and Focus Groups, including increased pedagogical content knowledge in content area literacy. It promoted supportive school organisation and leadership and implementation programme coherence amongst levels and within schools, departments and teaching teams.
From the existing literature, several dimensions of a generic model were identified for examining the implementation of SLP. More details are provided in the body of the report.
Professional Learning Community Design
This dimension refers to the design of the Focus Groups of teachers, and particularly its relationships with students, with the LL, with the Senior Management Team and the ongoing relationships between each of the above and the wider staff. This organisation could vary in a variety of ways. Through the four sources of evidence this dimension was refined to those with a Student Focus, and those with another focus.
Student Focus with leadership and planned extension
In one type the Focus Group comprises those teachers who teach a common class or classes (Focus Class) of students. Here the evidence base and the focus of inquiry was specific to that common group of students (within one class) and their learning across the content areas represented by the teachers. Some of the schools which adopted a Focus Class approach were small schools in which all teachers of Year 9 and 10 were grouped around the classes that they taught. That is to say that all teachers were Focus Group teachers and all Year 9 and 10 students were in Focus Classes. Others were larger schools in which a small number of classes (typically 3 or 4) were selected as Focus Classes. It features a dedicated and highly active LL leading the inquiry with the Focus Group teachers, plus strong support by a member of the Senior Management Team; often a Deputy Principal with special responsibilities. It also features planned extension to the wider staff through staff meetings and structured professional development.
A range of other types are possible and were revealed through the questionnaires. The most frequent other types reported by schools were teachers being selected to represent content areas (Content Focus), no specific Focus Group but rather all teachers being involved (Staff Focus) or selection of potential leaders (Leadership Focus). In 2009, some large and some small schools adopted a Staff Focus model. In 2010, all schools with 12 or fewer teachers of Year 9 and 10 adopted a Student Focus model.
From the 2009 cohort (n = 23 schools), there was evidence supporting the effectiveness of the Student Focus design which was adopted by 10 schools (while 13 schools adopted various other foci). The evidence was both quantitative and qualitative. Given the questions about the achievement data, analyses of relationships with achievement are very tentative (and this applies to each of the dimensions below). Comparisons were made between schools with Student Focus compared with Other Focus, specifically on e-asTTle Reading Scores (e-aRs) and gains for Māori and Pasifika students (seeTable 13). These showed some marked differences in scores (range from d = 0.41 to d = 1.54) at the schools with Student Focus compared with Other Focus, and large differences in gains at Year 10 (d = 0.84), but not at Year 9 (d = 0.13). Although overall the project was associated with greater gains in Year 9 than in Year 10, this shows within the generally weaker gains in Year 10 there was an advantage for students in Focus Class schools.
This dimension was composed of several parameters. The implementation could be evaluated in terms of intensity and breadth (eg, frequency and length of Focus Group sessions), integration into the schools’ programmes with high coherence between SLP and other programmes, and level of supportive leadership. There was inconsistent evidence from questionnaire data for the range of implementations on this dimension being related to levels of achievement or gains. The likely explanation for this was that the majority of schools implemented a relatively intense model. For example, in 20 schools, five or more sessions were held over 2009 with Focus Group teachers, which typically ran for two hours or more. Similarly, the majority of schools had four or more sessions with the whole staff which typically lasted between thirty minutes and two hours. The qualitative data from the Case Study Schools more strongly reinforced the claim that this was an important dimension in the level of implementation.
The third dimension was concerned with inquiry about teaching. Specifically this tapped the level and potential informativeness of the inquiry as part of teacher professional learning in the schools. Again, there was inconsistent evidence for this dimension being related to achievement, and in some instances there were quite strong negative correlations between amounts of reported inquiry and achievement. This may be because of variation in levels of inquiry which was in response to particularly strong learning (and teaching) needs; that is, schools with very low achievement levels may report more intense inquiry.
Content: Student Focus
The 2009 schools differed on whether they had adopted the SLP guidelines and had a specific focus on Māori and Pasifika students who were underachieving when considered in terms of national expectations. By default, given that the 2009 analysis conducted by the SLP coordination team was restricted to students scoring 3A and below on e-asTTle Reading (because of problems with the tool), the students in the statistical analysis were relatively low achieving. Nevertheless, there were some differences between schools. Almost all school targeted ‘all’ students, but the LLs at only eight schools identified Māori and Pasifika students as the specific focus. However, when tested in a number of ways, this difference was not statistically related to achievement outcomes.
The Optimal Model derived from these data is one that has a core design focused on common students (Focus Class), where the Focus Class teachers have considerable guidance and professional input from the LL, and at least one member of the Senior Management Team is an active and dedicated supporter. The professional development has high intensity and breadth in the sense of having frequent sessions and planned extension to the whole staff. Inquiry processes which are evidence-based are focused on the common students, within the context of an overarching concern for underachievement and Māori and Pasifika students’ achievement. SLP would have high coherence with the core programme in the school and other intervention programmes. Further details are contained in the body of the report.
However, the data from 2009 indicated some barriers to achieving a closer fit to this model in 2010. The first was a need to have an appropriately recognised and resourced position as LL. Half of the LLs identified having not enough time and competing priorities to implement their role. In the report an analysis of funding and resources is provided which, when compared with other programmes, would suggest that in SLP the LLs had a less than optimal focus on SLP. A limited estimate of the overall funding per school per year, including research and evaluation, suggested that SLP was resourced at approximately $50-60,000 per year per school (from both Ministry and school funding; see Table 1 in the full report). The second barrier was the expertise of the LL as the leader of inquiry. Although LLs tended to rate their knowledge of effective practices in adolescent literacy highly, they were less positive about using evidence to identify and prioritise needs (13 LLs rated themselves as confident and 9 rated themselves as not very confident).
Together, the Optimal Model and the identified constraints from the 2009 evidence were fed back to Cohort 1 and Cohort 2 schools at the beginning of 2010. Specific emphases (and guidelines) were identified for schools to:
- Increase levels of intensity of implementation
- Increase degree of coherence in specific areas, and specifically position SLP as central with which other interventions should be deliberately integrated
- Increase leadership support
- Focus inquiry on specific evidence to do with learning and achievement, and specifically for the underachievement of Māori and Pasifika students
- Build the expertise of the LLs
- Employ Focus Class organisation more widely.
Implementation in Cohort 1 and Cohort 2 schools in 2010 could be compared directly against these six evidence-based directions. In the next section, evidence for how the SLP shifted in its implementation further towards the Optimal Model is presented.
Shifts in Implementation from 2009 to 2010: Explaining Variation
A limited comparison between elements of the Optimal Model of implementation can be made between 2009 and 2010. The comparison is limited because some items on the Literacy Leader questionnaires changed across years, and because only Cohort 1 schools were represented in the 2009 data, while both Cohort 1 and Cohort 2 schools provided the 2010 data.
The limited comparisons suggest that the levels of implementation increased in four specific areas. Leadership support within the schools changed, as there was greater contact between LLs with the Principal or Senior Management Team in 2010 and there was a 50% increase in Inquiry (as indexed by observations and feedback sessions with teachers in general as well as with Focus Group teachers). There was evidence of an increased focus on Māori and Pasifika students and underachievement from 2009 to 2010 and the Focus Class organisational structure was more widely adopted by schools (with an increase of over 50%). Overall, there was some indication that LLs felt they knew more or were more confident about their knowledge in 2010. However, the implementation appears not to have changed in the area of degree of Coherence with other professional development programmes in the schools, nor in Intensity.
The evidence for 2010 from both the LFs and the LLs was that there was a reasonable or close to medium level of implementation in these areas. However, whole-staff and departmental foci were the lowest rated of all the dimensions with wide variation by both LFs and LLs (though LFs were not questioned about the departmental focus).
Implementation 2010 and Achievement
Of the dimensions of the Optimal Model in which schools varied in 2009, the only one associated with achievement was the Focus Class organisation. The same was true in 2010 when more systematic analyses were completed. The analyses were now based on comparisons between those Māori and Pasifika students in a Focus Class organisation versus those not. Not all students in Years 9 and 10 were in Focus Classes. The students in a Focus Class tended to be the lower achieving students. At Year 9, Māori students in Focus Classes made significantly greater gains than Māori students not in Focus Classes in the same school. This amounted to a 44% increase over expected gain (31 e-aRs) by those in Focus Classes, compared with a less than expected gain (20 e-aRs) by those not in Focus Classes. The effect size was small to moderate (d = 0.21), but students were within expected curriculum levels in Term 4, 2010. A similar pattern occurred at Year 10 where Māori students in Focus Classes made significantly greater gains (28 e-aRs) which were close to four times more than students not in Focus Classes (8 e-aRs) and were within expected curriculum levels. The effect size was moderate (d = 0.37). This is a comparison between groups within an intervention. Given all Māori students received the intervention in some form, the effect size reflects a very significant educational finding. The marked pattern for gains did not hold at either Year 9 or Year 10 for Pasifika students, although at Year 10 the difference in gain meant that Pasifika students were within 1 sub level of their expected curriculum level.
Apart from Focus Class variation between schools, none of the other dimensions of the Optimal Model as judged by the LLs and LFs were systematically associated with achievement patterns. There are several reasons for this nil finding. There are psychometric possibilities (the variation between schools was not enough); there are measurement and conceptual possibilities (these are not the right dimensions); and there are possibilities to do with how the dimensions interact with characteristics of schools which mean simple linear relationships are not likely (eg, Intensity may vary positively with the degree of need for professional development in a school).
- An appropriately recognised and resourced position is established as the leader of inquiry and professional development within the school, with status and time to implement the role.
The trends across 2009 and 2010 suggest that SLP was implemented in part with increasing fidelity towards what we describe as an Optimal Model. The level of Inquiry increased, the perceived focus on underachievement and Māori and Pasifika students increased and the use of Focus Class structure increased. However, some areas remained at 2009 levels, specifically the Intensity of implementation (eg, number of whole-staff sessions) and the Coherence (eg, degree of integration with other programmes including implementation of the New Zealand Curriculum in the school). These limits appear to be related to constraints identified at the school level, especially the need to ensure through training, school organisation and funding levels that an appropriately recognised and resourced position is established as the leader of inquiry and professional development within the school, with status and time to implement the role. A limited estimate of the full costs (Ministry, School, National Coordination and Research) per school per year suggests approximately $50-60,000 per school (see Table 1). This may not provide enough funding to enable effective levels of leadership in a ‘ripple’ process within the school involving teacher release and whole staff and departmental meetings, where systematic professional learning communities are established and maintained. Schools reported less focus on SLP than other, apparently higher funded, programmes at a school level. Currently, no research evidence exists in New Zealand that provides cost-benefit comparisons between interventions. Some very successful school change programmes report one or more full time staff placed in, or released in, a school to implement change at a school level (McNaughton, 2011).
- A Focus Class organisation needs to be present in intervention designs for secondary schools.
A major finding is that a Focus Class organisation is strongly indicated in intervention models in secondary schools. SLP provides the first systematic evidence for New Zealand contexts of the significance of a student focus in implementations. Other projects have adopted forms of this organisation, including Te Kotahitanga and projects by Starpath. The evidence supports making this an evidence-based design feature. However, there are caveats. The evidence from SLP also shows that without sufficient resourcing including a dedicated role within a school, the wider effects on staff and programme coherence may be threatened.
- Further research into and testing of an Optimal Model for intervention in schools needs to be undertaken.
No other dimensions of the Optimal Model appear to be related to student outcomes. There are several possible reasons for this nil finding. The implication is that more research is needed on how these features of an Optimal Model impact at the level of school and classroom implementation on teaching and learning, with specific attention to parametric analyses (eg, how ‘much’ of these dimensions is needed to achieve what effects?) as well as qualitative analyses (eg, what are the qualities of effective data discussions focused on the literacy needs that make a difference to the underachievement of Māori and Pasifika students?). Ongoing research suggests that effective data discussions need to be very specific and, to be effective, draw on extensive pedagogical content knowledge (Lai, Timperley, & McNaughton, 2010).
- A research and development model built around a leader of inquiry is suggested.
Going further beyond the direct evidence, there are suggestions from this research and evaluation that a model other than the cascading implementation model might be needed. A new model would focus the functions of inquiry and implementation more directly within the role and function of the leader of inquiry and professional development in the school. In this model, a research and development team would provide direct professional development support and research and evaluation support to build the expertise for that leader to design and lead the systematic inquiry into students’ needs and the fine-tuning of instruction across content areas. The role would need extensive graduate level
training and would need to be a specifically designated and funded position rather than an ‘add on’ or modification of an existing project such as a HOD. Rather than a cascading model, a research and development partnership model may be more powerful (McNaughton, 2011). This would involve an overlap involving professional learning communities with the Lead Teacher at the union (the common intersection).
- Impacting reading and writing at Years 9 and 10 may not be sufficient to enable higher pass rates consistent with Ka Hikitia targets and the expectations of the Pasifika Education Plan.
Analysis of the effects of the implementation on student achievement will be completed through the National Coordination reporting process. The data from 2011 will be crucial for this because these achievement data will provide the first full data set of longitudinal changes in literacy across two years (for the Cohort 2 schools). Analyses will not be restricted to Years 9 and 10. The National Coordination Team is also analysing trends in pass rates at NCEA levels. This is an appropriate requirement from the SLP design. A very significant research and policy question remains: Is the focus on underachievement in Years 9 and 10 necessary and/or sufficient to impact markedly on measures of engaging in the New Zealand Curriculum and, notably, nationally expected levels of success at NCEA Level 2 and University Entrance for Māori and Pasifika students? If the answer to that question is largely negative, then new implementations of professional development for secondary literacy may need to shift their focus. The concern here is that impacting reading and writing at Years 9 and 10 may not be sufficient to enable higher pass rates consistent with Ka Hikitia targets and the expectations of the Pasifika Education Plan.
- e-asTTle Te Kete Ipurangi (TKI) website.
Education Data Requests
If you have any questions about education data please contact us:
Email: Requests EDK
Phone: +64 4 463 8065