National Standards School Sample Monitoring & Evaluation Project 2010-2013
This report outlines findings from information collected in 2013 and describes trends observed over the four years of implementation to date. Five types of data were collected at two time points. Copies of schools’ 2013 student achievement targets and 2012 analysis of variance reports were collected in the middle of the year. At the end of the year, Overall Teacher Judgments (OTJs) in reading, writing, and mathematics were collected for all students, and judgements from the PaCT mathematics national trial were collected for a sample of students. Copies of end-of-year reports to parents, families and whānau were obtained, and an online survey of principals was conducted.
Author(s): Jenny Ward and Gill Thomas, Maths Technology Limited.
Date Published: May 2015
This report is available as a download (please refer to the 'Downloads' inset box). Please consider the environment before printing the contents of this report.
The National Standards School Sample Monitoring and Evaluation Project describes and evaluates the implementation of National Standards in New Zealand schools. The project has been operating since 2010 when the standards were first introduced. It was initially designed as a three year project, but has been extended for an additional two years to continue monitoring key elements of the implementation of National Standards.
This report outlines findings from information collected in 2013 and describes trends observed over the four years of implementation to date. Five types of data were collected at two time points. Copies of schools' 2013 student achievement targets and 2012 analysis of variance reports were collected in the middle of the year. At the end of the year, Overall Teacher Judgments (OTJs) in reading, writing, and mathematics were collected for all students, and judgements from the PaCT mathematics national trial were collected for a sample of students. Copies of end-of-year reports to parents, families and whānau were obtained, and an online survey of principals was conducted.
Overall Teacher Judgments
The OTJ, as a judgment of each student's achievement in relation to the National Standards, is central to the implementation of the standards initiative overall. The information OTJs provide is reported to parents, families and whānau and to Boards of Trustees. It is also used by schools to tailor teaching programmes and target students for intervention. For these programmes and interventions to successfully raise student achievement, OTJs need to be dependable.
Evidence suggests that increasing proportions of schools moderated OTJs from 2010 to 2013, and that the use of formal moderation processes has become more widespread in this period (for example, 56% of schools used formal processes to moderate reading OTJs in 2010 and this increased to 63% in 2013). Principals reported that a variety of reference points for student achievement were used in moderation discussions at their school, with nearly all principals noting that the National Standards themselves (98%), or learning progressions such as the literacy learning progressions or the Number Framework were used (88%). Just over half of the principals (51%) indicated that school-developed resources such as descriptions of performance or annotated work samples were used. The extent to which these resources accurately describe student capability in relation to the National Standards is unknown and it is possible that the variation in criteria developed by different schools may contribute to any inconsistencies in the judgments being made between schools.
There is a growing body of evidence from this project about the current dependability of OTJs. Considerable variability was observed in the accuracy of teachers' ratings in relation to the writing and mathematics standards in the assessment scenarios in the 2011 and 2012 data collection (61% of teachers' writing judgments and 58% of teachers mathematics judgments were accurate in the 2012 scenarios). Substantial variation has been observed in the OTJs of individual students from 2011 to 2013 (approximately 40% of students receive achievement ratings that vary from those received in previous years), and the OTJs of students in Years 7 and 8 varied by school type in the 2012 and 2013 data collection (for example 77% of Year 8 students at full primary schools were rated 'at' or 'above' the mathematics standards in 2013, compared with 65% of Year 8 students in intermediate schools). One new source of evidence was used to investigate the dependability of OTJs in 2013. The levels of agreement between mathematics OTJs made using schools' regular processes, and mathematics ratings generated from teachers' judgments in the PaCT mathematics national trial were compared, and substantial variation was found (overall, 40% of school OTJs were in agreement with PaCT trial ratings, while 60% differed).
Considered together, these results strongly suggest that OTJs lack dependability, which is problematic as the National Standards system relies on OTJs. It should be noted that these concerns do not mean that all OTJs are inaccurate. While the trends described below support the view that OTJs lack dependability, it is unsurprising that these consistency issues are present, given the recentness of the initiative and the ongoing development of tools to support teachers to make judgments in relation to the National Standards.
Reporting to parents, families, and whānau
Clear reporting to parents, families and whānau is a key element of the National Standards initiative. The intention is that families are well informed about their children's learning and, therefore, better able to support this in the home.
Findings indicate that increasing proportions of schools reported National Standards information to parents, families, and whānau from 2010 to 2013. The proportion of end-of-year reports that refer directly to the National Standards has increased over time (79% in 2010 and 94% in 2013), as has the proportion of reports that sufficiently describe student achievement in relation to the National Standards (60% in 2010 and 70% in 2013). Results suggest that the clarity of reports continues to be of concern, with less than half (45%) of National Standards reports rated as clear in 2013.
Student achievement targets
OTJs are reported to Boards of Trustees and used to inform annual student achievement targets, which guide decisions about the teaching support individual students receive.
Evidence from the project indicates that increasing proportions of schools included targets in their schools' charters that addressed student achievement in relation to the National Standards (75% in 2011 and 95% in 2013). The targets were increasingly differentiated to accelerate progress for specific groups of students (57% of National Standards targets were differentiated in 2011 and an average of 78% of targets were differentiated across the three areas in 2013). The level of challenge inherent in schools' targets appears to be an ongoing cause for concern however, with less than half of schools' 2013 National Standards targets in reading (47%), writing (46%), and mathematics (45%) rated as challenging.
Schools' use of National Standards data
It is intended that schools will use National Standards data to inform the provision of targeted teaching interventions to students, with the ultimate aim of improving student achievement.
Results indicate that increasing proportions of schools collated National Standards achievement data (for example, 76% of schools collated mathematics data in 2011, and this increased to 97% in 2013) and used this to systematically track student achievement in relation to the National Standards (for example, 83% of schools tracked mathematics achievement from the end of 2012 to the end of 2013).
Principals also reported increasingly using National Standards data to inform the provision of tailored teaching interventions for students (an average of 61% across the three areas in 2011 and 90% in 2013). Teaching support was provided in a variety of ways in 2013. Approximately three-quarters of the principals reported that regular classroom teaching programmes were increasingly differentiated to meet students' learning needs (71% in reading, 82% in writing, 76% in mathematics), an increase from approximately half of the principals in 2012. Principals noted that support external to the classroom programme was provided in 2013 both by qualified teachers (74% in reading, 59% in writing, 67% in mathematics) and teacher aides (38% in reading, 27% in writing and 22% in mathematics), and this was similar to 2012 results. The quality of these teaching interventions, or the extent to which they were matched with students' learning needs was unable to be investigated in the scope of this project.
National Standards achievement data, 2010 to 2013
There have been small increases in the proportions of students rated 'at' or 'above' the Reading, Writing and Mathematics Standards over the four years of implementation to date. For example, 72% of students were rated 'at' or 'above' the Reading Standards in 2010, and this increased 2% a year, to 78% in 2013. Substantial increases in the proportions of students rated 'at' or 'above' were observed for some demographic sub-groups: Pasifika students in reading and mathematics, Year 7 and 8 students in writing and mathematics, and students at low decile schools in reading. These increases must be interpreted with caution; they represent changes in teachers' judgments of student achievement over time. Given the magnitude of the improvements in achievement that are suggested by the OTJ data, the evidence that suggests OTJs lack dependability, and evidence about patterns of student achievement in New Zealand from international studies, the OTJ data cannot be taken as evidence that student achievement is improving over time.
Education Data Requests
If you have any questions about education data then please contact us at:
Email: Requests EDK
Phone: +64 4 463 8065