Monitoring and evaluating curriculum implementation: Final evaluation report on the implementation of the NZ Curriculum 2008-09 Publications
Publication Details
This report presents findings from a national evaluation of the implementation of The New Zealand Curriculum. The project sought to establish a national picture of implementation progress in English-medium schools in the first two years following the curriculum's launch in November 2007.
Author(s): Dr Claire Sinnema, The University of Auckland. Report prepared for the Ministry of Education.
Date Published: March 2011
Executive Summary
In November 2007, a revised national curriculum was launched in New Zealand. The New Zealand Curriculum (NZC) (Ministry of Education, 2007) replaced the previous series of curriculum statements developed during the 1990s which were the focus of a curriculum stock-take between 2000 and 2002. It was developed through a lengthy and inclusive development process that involved participants from a wide range of stakeholder groups. This report summarises a national evaluation of the implementation of that curriculum undertaken by researchers at the Faculty of Education at the University of Auckland for the Ministry of Education. The project, called MECI (Monitoring and Evaluating Curriculum Implementation) sought to establish a national picture of implementation progress in English-medium state and state-integrated schools in the first two years following the curriculum's launch.
Overview of methodology
The main research questions focused on gaining a national picture of implementation progress:
Research Question 1: What progress was made in the first two years of implementation of The New Zealand Curriculum?
Research Question 2: What factors explain the degree of progress in implementing The New Zealand Curriculum?
To address these questions a mixed-methods approach was taken, in which data were gathered through four surveys of educators in random stratified samples of schools (see Table 1), complemented by a series of 26 focus groups involving 247 participants from across a range of school types1 and roles2.
Surveys | Number | |
---|---|---|
Participating Schools | Respondents | |
Web 2008 (Aug) | 230 | 579 |
Paper 2008 (Oct/Nov) | 221 | 2578 |
Web 2009 (Nov) | 345 | 604 |
Paper 2009 (Oct/Nov) | 176 | 1800 |
In August 2008 there were 579 respondents to a web survey from 230 (19%) of the 1210 schools invited to participate. The 579 respondents represented 13% of the teachers in the participating schools. In October 2008 an additional sample of schools were invited to respond to a paper survey in an effort to increase the response rate. There were 2578 responses during October and November to the 2008 paper survey from teachers and principals in 221 (37%) of the 593 schools invited to participate. The 2578 respondents represented 41% of the teachers in the participating schools. In October and November 2009, the second administration of both paper and web surveys took place. There were 604 responses from educators in 345 (29%) of the 1191 schools invited to take part in the 2009 web survey. The 604 responses represented 8% of the teachers in the participating schools. The 2009 paper survey was identical to the first paper survey, with the addition of two support encounter items and one support quality item. It was sent to the principals of the same 593 schools who were sent the paper survey in 2008, with a request for them to again extend the invitation to all teachers in their school. Responses were received from 1800 educators in 176 schools. The 1800 responses represented 36% of teachers in the participating schools.
Framework for the evaluation
Support: the kinds of support educators encountered (including people within and beyond their schools, publications and web) and how valuable they perceived those supports to be.
Receptivity: the extent to which educators value the curriculum, their confidence in implementing it in their own context, and the degree to which they perceive implementation to be feasible.
Understanding: how educators understand a range of key elements of the new curriculum (including Teaching as Inquiry, values, principles, key competencies, the vision, and the learning areas) and their views about the extent of shift required or offered under the new curriculum.
Practice: the extent to which practices that reflect the intentions of the new curriculum are becoming evident in both leaders' and teachers' practices.
Summary of findings
Receptivity
To what extent do schools and teachers feel confident about, or challenged by, the implementation of The New Zealand Curriculum?
The New Zealand Curriculum is well regarded by educators across the system. They generally view it positively and consider it to be a high quality document that is an improvement on the previous curriculum. The mean rating on the 0–5 scale for regard (incorporating items about how flexible, practical and improved the curriculum is) was 3.3 in 2008 and 3.4 in 2009. In 2009, more than three quarters (77%) of respondents rated at the positive end of a 6-point continuum asking if they view the curriculum to be worse or better than the previous one. Comments from educators in a range of contexts also indicated that there is widespread approval of the direction set out in The New Zealand Curriculum, and in most cases enthusiasm and eagerness to implement it well. Educators particularly value the curriculum for its flexibility, its relevance to 21st century learners, and its potential to support high quality teaching and learning for students.
However, the ratings for confidence (which incorporated respondents' views of how complicated they view The New Zealand Curriculum to be, how reasonable they consider the workload, and how easy and confidently they view implementation) were not as high as the ratings for regard. The mean rating on the 0–5 scale for confidence was 2.7 in 2008 and 2.8 in 2009.
Low levels of confidence were also indicated in the difficulty ratings for key New Zealand Curriculum practices included in the surveys. Many key aspects of the curriculum continue to be considered difficult to implement. Across 23 practice items, an average of 48% of respondents indicated those practices to be difficult or very difficult, rather than easy or very easy. Additionally, the mean ratings for difficulty did not shift significantly between 2008 and 2009.
There were also marked differences between primary and secondary respondents in relation to regard and confidence. While the overall pattern of higher regard than confidence ratings stands for both groups, secondary respondents rated both of these lower than primary respondents.
In summary, implementing key practices related to The New Zealand Curriculum continues to be difficult. The New Zealand Curriculum is cherished but is challenging.
Understanding and practice
What progress is being made in schools and by leaders in implementing school-wide curriculum design?
There has been widespread engagement with the curriculum since its launch in 2007 and progress in reviewing curriculum elements necessary for designing a local school curriculum. The majority of principals reported having reviewed all elements, including values (88%), principles (80%), key competencies (89%), pedagogy (71%) and the individual learning areas (77%). There remain however, between 10% and 20% of schools who have not yet reviewed those elements.
There is evidence of a strong understanding of the school-based curriculum design implications of The New Zealand Curriculum. Respondents typically both recognise and value the opportunity to design a curriculum that meets the needs of their own students and addresses both national and local community aspirations for learning.
What progress is being made by schools and leaders in implementing the purposes and key understandings of The New Zealand Curriculum?
Despite progress in educators' familiarity with The New Zealand Curriculum (99% have encountered the document at least once, and 68% six or more times), there have not been notable shifts in teaching and learning practices. Many have begun thinking about and considering how practices could shift to more strongly reflect The New Zealand Curriculum, but fewer have actually applied those practices. Responses indicated that key curriculum aspects, including competencies, values, Teaching as Inquiry, partnerships and student agency are generally not yet strongly evident or consistently embedded in practice. About a third of respondents view the curriculum overall to be more the same, than different to the previous curriculum (30%). About one-third of respondents also tend to view The New Zealand Curriculum as requiring few shifts (35%).
Comparisons were made between 2008 and 2009 responses to questions about how evident key curriculum aspects are in educators' practices. While there were pockets of significant progress in particular schools, the general pattern was of only slight shift, surface-level change, or for just discrete aspects of The New Zealand Curriculum to have been addressed. The mean ratings for each of these aspects did increase slightly between 2008 and 2009, but Cohen's d effect size calculations3 signal that the magnitude of the shift was very small, as shown in .
Notes:
| |||||||
2008 | 2009 | Effect Size2 | |||||
---|---|---|---|---|---|---|---|
Mean | No | SD1 | Mean | No | SD1 | ||
Key Competenties: | |||||||
- Intra/Inter-Personal3 | 2.11 | 2487 | 0.618 | 2.20 | 1752 | 0.564 | .16 |
- Disciplinary | 2.11 | 2519 | 0.533 | 2.17 | 1761 | 0.505 | .11 |
- Pedagogical | 1.52 | 2481 | 0.669 | 1.60 | 1750 | 0.643 | .13 |
Values Factor | 1.83 | 2509 | 0.621 | 1.86 | 1756 | 0.610 | .05 |
Teaching as Inquiry | 1.98 | 2507 | 0.542 | 2.01 | 1757 | 0.519 | .07 |
Student agency | 1.40 | 2480 | 0.719 | 1.44 | 1751 | 0.709 | .05 |
Parent Involvement | 1.26 | 2441 | 0.749 | 1.29 | 1730 | 0.705 | .05 |
In addition, it is apparent that the understanding and interpretations of key curriculum aspects continue to be an implementation issue. While the curriculum promotes school-based curriculum design, there are many requisite aspects which are misunderstood, or understood in less depth or with less complexity than the curriculum requires. Understanding issues, in which practitioner interpretations differ from curriculum intentions, include for example:
- Confusion between Teaching as Inquiry (when teachers inquire into: what is most important; what strategies or approaches are most likely to work; and the impact of teaching on students) and Inquiry learning (one approach teachers might use, but don't have to, in which students learn about learning, investigation and research as they explore topics of interest).
- A superficial view of values in the curriculum (focused on inculcating the national curriculum and school values) rather than the deeper three way values education process signalled in The New Zealand Curriculum (education about values, and in valuing skills alongside encouraging affiliation to key public values).
- The tendency to emphasise familiar aspects when faced with unfamiliar curriculum ideas as seen in the data about partnerships for teaching and learning. There is a tendency to emphasise the educators' role in informing more so than consulting parents/communities, and even less emphasis on collaborating on teaching and learning matters.
- The tendency to emphasise the more familiar idea of skills (from the previous curriculum) when explaining understandings about key competencies, despite key competencies requiring much more nuanced and complex interpretations about how knowledge, attitudes, skills and values are integrated.
Support
To what extent are the materials, resources and programmes supporting schools and teachers to make the changes needed?
Quantity
In both 2008 and 2009, there was a trend for far greater engagement with internal supports (both colleagues and resources) than with external supports (advisors, consultants, facilitators or colleagues from other schools). Supports in a sustained way (encountering the support six or more times since 2008), was 68% for The New Zealand Curriculum document, 62% for colleagues within their school, 29% for other Ministry of Education publications, and 22% for The New Zealand Curriculum website. For external supports (those not consistently available within a school) the percentage who had encountered them in a sustained way were 14% for colleagues at other schools, 14% for advisors, 7% for facilitators and 4% for private consultants.
A large proportion of teachers, principals and leadership team members reported not engaging with Te Marautanga o Aotearoa at all during 2009. Teachers in particular had engaged less with Te Marautanga o Aotearoa than others. Just over one quarter of principals (27%) and one third of leadership team members (33%) had referred to Te Marautanga more than three times, but only 7% of teachers reported the same level of engagement.
Slightly more than half (51%) of the teacher respondents reported engaging with a Best Evidence Synthesis at least once to support their implementation of The New Zealand Curriculum. Slightly more than half of the principal respondents (51%) and other leadership team members (58%) had used a Best Evidence Synthesis more than three times.
Educators' responses about the quantity of support provision (rated on a miserly–generous continuum) showed that in 2009 more educators (56%) continue to view the quantity of support provision on the miserly, rather than generous end of the continuum. Views about insufficient provision of support were also emphasised in focus group and web survey comments.
Quality
Respondents were asked to rate support quality on 6-point Semantic Differential scales which had a negative anchor at one end, and a positive anchor at the other. Ratings were collapsed to indicate the percentage of respondents who viewed support quality at the negative end of the scale, or the positive end (productive, relevant, stimulating and challenging) of the scale. In 2009 the aspect of support quality with the greatest proportion of respondents at the positive end of the continuum was 'relevance' (70%) indicating that support tends to be considered relevant to the task of curriculum implementation. The aspects with the greatest proportion of respondents at the negative end were on the items with 'tedious' (43%) and 'waste of time' (39%) as the negative anchors. These findings indicate that approximately one third of respondents in 2009 (and close to half in the case of tedious-stimulating) continue to view their experience of support as more low than high quality.
What alternative or further supports do schools and teachers feel they need to effectively implement The New Zealand Curriculum?
The most significant barrier to implementation reported by respondents in 2009 was lack of time for planning and implementation. More than two thirds of respondents (67%) indicated this to be a moderate, difficult or extremely serious barrier. Other notable barriers to implementation (rated as moderate, difficult or extremely serious) included lack of expertise availability (43%), lack of face-to-face support (44%) and lack of professional learning community support (37%).
Explaining the implementation progress
What explanations are there for where more or less significant change is occurring?
To inform the question about what explains the degree of progress in curriculum implementation, a series of stepwise linear regressions were carried out, examining the relationship between the support, receptivity and practice variables4. When the findings from the series of linear regressions (detailed more fully from p. 78) are considered together, a picture emerges of important influences on key aspects of implementation (see Figure 1).
Figure 1: Regression summary: Predictors in curriculum implementation
When practice was taken as the dependent variable, confidence (b = 0.199) accounted for 16%, and internal support quantity (b = 0.199) for a further 6% of the combined 22% variance in practice. This shows an important relationship between educators' views of the complexity, workload and difficulty involved in implementation and their response in terms of classroom practice. When confidence was taken as the dependent variable, regard (b = 0.582) accounted for 46% of the variance in confidence, and support quality (b = 0.105) accounted for the additional percentage in the total 47% of the variance predicting confidence. The factors for support quantity, both internal and external, were not shown in the model as predictors of confidence. When regard was taken as the dependent variable, support quality (b = 0.404) accounted for 24% of the variance in regard, and internal support encounters (b = 0.203) accounted for the additional 2% in the total 26% of the variance predicting regard.
This suggests that support (both quantity and quality), unsurprising, does not have a direct and certain relationship with shifts in practices that reflect The New Zealand Curriculum. Rather, high quality support functions to improve the regard that educators have for the curriculum, in ways that increase their confidence, which in turn contributes to their ability to give effect to the curriculum in their practice. It is important to note, that variables in the relationship between support quality and practice were beyond the scope of this research.
Confidence was shown to be a key variable in explaining limited progress, and variable implementation of the curriculum. Educators were no more confident about giving effect to the curriculum in 2009 than they were the year before, despite there being a programme of support for implementation in place. The reasons for the lack of confidence were diverse, relating to teachers' own curriculum, assessment and content knowledge and also to contextual barriers.
What about understanding?
Interpretations and understandings about curriculum also explain the limited degree of progress overall. The role of teachers' theories, understandings, knowledge and beliefs as key influences on their practice is widely recognised. Understanding the curriculum as more similar to than different from the previous curriculum, for example, is not conducive to change in practice. New information, such as that embedded in a new curriculum, does not simply supplant existing knowledge and practice (Spillane, Reiser, & Reimer, 2002). Well-intentioned practitioners are likely to be influenced by expectations embedded in their existing schema, and to over-assimilate reform ideas as similar to their existing ideas. The limited shifts in curriculum related practices found here signals a need for more opportunities for theory engagement in relation to the curriculum as a whole and its elements. Educators' existing beliefs need to be cued (in relation to the new understandings) and examined in relation to the new learning. Giving effect to real change in response to the curriculum requires confidence, and confidence requires (amongst other things) deep understandings about the distinctions between the old and the new.
Footnotes
- Including Primary, Intermediate, Full-primary, Contributing, Secondary, Composite and Special
- Including Principals; Deputy, Assistant and Associate Principals; and Classroom teachers
- Standardised mean effect sizes (such as Cohen's d) indicate the mean difference between two variables expressed in standard deviation units. A score of 0 represents no change. An effect-size of d=1.0 indicates an increase of one standard deviation. While interpretations of effect sizes are dependent on the measurement context, Cohen (1988) offered the following guide: .8 = large, .5 = moderate and .2 = small.
- Regression is used to test the effects of independent (predictor) variables on a single dependent (criterion) variable. For the purposes of these regressions the multiple practice variables (as reported in the group comparisons section) were substituted for a single practice scale (α = 0.92) that incorporated all 23 practice items, and was the dependent variable in the first regression.
Navigation
Where to find out more
Contact Us
Education Data Requests
If you have any questions about education data please contact us:
Email: Requests Data and Insights
Phone: +64 4 463 8065