National Standards: School Sample Monitoring & Evaluation Project 2010-2014

Publication Details

The National Standards School Sample Monitoring and Evaluation Project describes and evaluates the implementation of National Standards in New Zealand schools from 2010 to 2014. This is the final report from the project, which has been operating since the standards were first introduced. This report summarises findings from all five years of the project, and describes results collected in 2014.

Author(s): Jenny Ward and Gill Thomas NZ Maths Technology Ltd. Report prepared for the Ministry of Education.

Date Published: July 2016

Please consider the environment before printing the contents of this report.

This report is available as a download (please refer to the 'Downloads' inset box).  For links to related publications/ information that may be of interest please refer to the 'Where to Find Out More' inset box.

Executive Summary

Over the duration of the project, information has been collected from a sample of approximately 100 schools selected to be representative of the population of New Zealand schools with respect to three variables: school type, school decile, and region. School size was also accounted for during the sample selection process. Response rates have been consistently high so findings provide information about the implementation of National Standards in schools across New Zealand.

The project utilises a variety of data sources. In 2014, four types of data were collected at two time points. Copies of schools' 2014 student achievement targets and 2013 analysis of variance reports were collected in the middle of the year. At the end of the year, overall teacher judgments (OTJs) in reading, writing, and mathematics were collected for all students, copies of end-of-year reports to parents, families and whānau were obtained, and an online survey of principals was conducted. In previous years the project has also included online surveys of teachers and Boards of Trustees Chairpersons (2010, 2011, and 2012), teachers' responses to assessment scenarios designed to collect information about teachers' judgments in relation to the standards (2011 and 2012), and judgments from the PaCT mathematics national trial (2013).

Overall teacher judgments

The OTJ, as a judgment of each student's achievement in relation to the National Standards, is central to the implementation of the standards initiative overall. The information OTJs provide is reported to parents, families and whānau and to Boards of Trustees. It is also used by schools to tailor teaching programmes and target students for intervention. For these programmes and interventions to successfully raise student achievement, OTJs need to be dependable.

Generally, it was more common for schools to moderate writing OTJs than it was for them to carry out moderation in reading or mathematics. On average 84% of schools used formal processes to moderate writing OTJs from 2010 to 2014, while an average of 62% of schools used formal processes to moderate reading and mathematics OTJs in this period. There were small overall increases in the proportions of schools using formal processes to moderate OTJs during the first five years of implementation. In reading this proportion rose 5% from 2010 to 2014, and similar sized increases were seen in writing (3% increase) and mathematics (8% increase). Principals reported that a variety of reference points for student achievement were used in moderation discussions at their school. In 2014, all principals noted that the National Standards themselves were used for this purpose, while the majority of principals (88%) noted that learning progressions such as the literacy learning progressions or the Number Framework were used. In both 2013 and 2014, just over half of the principals (51%) indicated that school-developed resources such as descriptions of performance or annotated work samples were used. The extent to which these resources accurately describe student capability in relation to the National Standards is unknown and if there is variation in the criteria developed by schools this may be an element which contributes to any inconsistencies in judgments between schools.

Over its duration, this project has collected a variety of evidence regarding the dependability of OTJs. The strongest evidence is found in the substantial variation by school type of the OTJs of students in Years 7 and 8 reported in relation to the 2012, 2013, and 2014 data collection. For example, 79% of Year 8 students at full primary schools were rated 'at' or 'above' the mathematics standards in 2014 compared with 67% of Year 8 students at intermediate schools. The low levels of agreement between mathematics OTJs made using schools' regular processes, and mathematics ratings generated from teachers' judgments in the PaCT mathematics national trial in 2013 also provided strong evidence that OTJs lack dependability. Overall, 40% of school OTJs were in agreement with PaCT mathematics national trial ratings. Substantial variation has also been observed in the OTJs of individual students from 2011 to 2014, and considerable variability was observed in the accuracy of teachers' ratings in relation to the writing and mathematics standards in the assessment scenarios in the 2011 and 2012 data collection. In 2013 and 2014, principals' perceptions provided another source of evidence that casts doubt on the dependability of OTJs.

Considered together, this body of evidence strongly suggests that OTJs lack dependability, which is problematic as OTJs are a central element of the National Standards system. It should be noted that there is no suggestion that all OTJs are inaccurate, but evidence indicates that a reasonable proportion may be. While the trends described above support the view that OTJs lack dependability, it is unsurprising that these consistency issues are present, given the recentness of the initiative and the ongoing development of tools to support teachers to make judgments in relation to the National Standards.

Reporting to parents, families, and whānau

Clear reporting to parents, families and whānau is a key element of the National Standards initiative. The intention is that families are well informed about their children's learning and, therefore, better able to support this in the home.

Findings indicate that increasing proportions of schools reported National Standards information to parents, families, and whānau from 2010 to 2014. The proportion of end-of-year reports that referred directly to the National Standards increased over time (from 79% in 2010 to 97% in 2014), although overall the proportion of reports that sufficiently described student achievement in relation to the National Standards did not increase substantially in this period (60% in 2010 and 65% in 2013).  Small proportions of reports (up to 18%) included information about students' progress in relation to the National Standards in reading, writing, or mathematics from 2010 to 2014. Results suggest the clarity of National Standards reports has been reasonably consistent, although concerningly low, over the first five years of implementation, with 40 to 50% of reports rated as clear from 2010 to 2014.

Student achievement targets

OTJs are reported to Boards of Trustees and used to inform annual student achievement targets, which guide decisions about the teaching support individual students receive. Evidence from the project indicates that overall, increasing proportions of schools included targets in their charters that addressed student achievement in relation to the National Standards (75% in 2011 and 90% in 2014). There was, however, a small decline in this proportion from 2013 to 2014 (from 95% to 90%). National Standards achievement targets were also increasingly differentiated to accelerate the progress of specific groups of students as implementation progressed. For example, 65% of National Standards reading targets were differentiated in 2012 and this increased to 88% in 2014.  In general, year level was the most common focus for differentiated National Standards targets. Between 43% and 60% of National Standards reading, writing, and mathematics targets were differentiated by year level from 2012 to 2014. A focus on Māori students was also reasonably common, with 26% to 50% of targets differentiated on this basis. Smaller proportions of National Standards targets focused on students by gender (12% to 35%), Pasifika students (6% to 23%), and students with special education needs (1% to 10%).

Results indicate that the majority of schools used baseline data to inform their National Standards targets (over 90% from 2010 to 2014), and focused targets on students who were 'below' or 'well below' the standards (over 80% from 2010 to 2014).  There was, however, a notable decrease in the proportion of schools that considered all year levels when setting targets (from 83% in 2012 to 70% in 2014).

Schools' use of National Standards data

It is intended that schools will use National Standards data to inform the provision of targeted teaching interventions to students, with the ultimate aim of improving student achievement.

Results indicate that, overall, the proportion of schools that collated National Standards achievement data increased from 2011 to 2014 (in mathematics this was from 76% to 92%). The majority of schools also appear to be systematically tracking the progress of individual students in relation to the National Standards (72% of schools tracked mathematics achievement from the end of 2013 to the end of 2014).

Evidence suggests that the majority of schools provided interventions to students rated 'below' the standards within the classroom programme from 2012 to 2014. For example, just over 70% of schools provided such reading interventions in these years. Similarly, most schools provided reading interventions to students rated 'well below' in addition to the classroom programme, and this proportion increased over time, from 78% in 2012, to 90% in 2014.  In contrast, fewer schools provided writing and mathematics interventions additional to the classroom programme to students rated 'well below' the standards, and this difference can largely be attributed to reading recovery, an intervention that is well embedded in schools' practices.

Results suggest that teaching interventions were delivered in a variety of ways. For example, in 2014 over 62% of principals reported that within the classroom, regular classroom teaching programmes were differentiated to meet students' learning needs (62% in reading, 76% in writing, 68% in mathematics). Principals noted that support external to the classroom programme was provided in 2014 both by qualified teachers (74% in reading, 57% in writing, 61% in mathematics) and teacher aides (41% in reading, 29% in writing and 32% in mathematics).

National Standards achievement data, 2010 to 2014

There have been small increases in the proportions of students rated 'at' or 'above' the Reading, Writing and Mathematics Standards over the first five years of implementation. For example, 64% of students were rated 'at' or 'above' the Writing Standards in 2010, and this increased to 70% in 2014. While these increases are reasonably small, they represent large shifts in the achievement of the population of students. Increases (of at least 10%) in the proportions of students rated 'at' or 'above' were also observed for particular demographic sub-groups: Pasifika students, Year 7 and 8 students, and students at low decile schools. These increases must be interpreted with caution; they represent changes in teachers' judgments of student achievement over time .

Given the magnitude of the improvements in achievement that are suggested by the OTJ data, the evidence that suggests OTJs lack dependability, and evidence about patterns of student achievement in New Zealand from international studies, the OTJ data cannot be taken as evidence that student achievement is improving over time.

Summary

Overall, the project has collected a wide variety of information, from a representative sample of schools, over the first five years of the implementation of National Standards. The data collected has shown reasonably consistent patterns over time which can be taken as an indication that schools have developed ways of the working with the National Standards that are now embedded in their regular practices. It seems likely that these practices will continue uninterrupted for as long as the environment within which schools operate remains the same.

Footnote

  1. The PaCT is an online tool that was developed by the Ministry of Education to improve the reliability and consistency of judgments over time. The mathematics national trial was held as part of its development in 2013. The PaCT became available to all schools in 2015.

Where to find out more

Contact Us

Education Data Requests
If you have any questions about education data then please contact us at:
Email:      Requests EDK
Phone:    +64 4 463 8065