National Standards: School Sample Monitoring and Evaluation Project Publications
Publication Details
This is the first report from the National Standards School Sample Monitoring and Evaluation Project 2010-2013, a three year project on National Standards implementation in a representative sample of schools.
This report presents results from a mid-2010 survey of 82 principals from schools in the project's sample, and an analysis of sample schools’ formats for reporting to parents in 2010 mid-year reports.Author(s): Maths Technology Limited. Report for the Ministry of Education.
Date Published: October 2010
Summary
The School Sample Monitoring and Evaluation Project is a three year project, established to evaluate the implementation of National Standards using a large representative sample of schools. This is the first report from the project and describes results gathered using two methods: an online principal survey and an analysis of the formats schools used for mid-year reporting to parents. As this data was gathered six months after National Standards were first put in place in schools the report provides early information on the implementation of National Standards.
The project has two purposes: to describe the implementation of National Standards within schools, and to monitor and systematically evaluate the effect of National Standards on students, teachers, schools and whānau. The descriptive element of the project is focused around twelve monitoring and evaluation questions, while evaluation is based on seven intentions which describe the intended effects of National Standards. A variety of data sources are employed (see Table 1 for an overview).
Table 1 presents the project's methodology and identifies the two monitoring and evaluation questions addressed in this report:
1. How do schools use information from National Standards to report to and communicate with parents?
7. To what extent are the National Standards understood as a set of common expectations for student achievement?
Note that the relevant data sources and statements of intent for each question are also identified in Table 1.
Table 1: Methodology1,2
- To what extent are the National Standards understood as a set of common expectations for student achievement?
- What processes are employed by schools to maintain consistent application of the National Standards?
- Online survey: Principals and BOT representatives
- Principal interviews
- Schools' achievement targets and analysis of variance reports
- National Standards will provide clear information about student achievement for Boards of Trustees which can be used in decision making and resource allocation processes.
- In what ways do teachers use information from a variety of student assessments to make overall judgments?
- How dependable and consistent are teachers' overall judgments?
- Student achievement data
- Online assessment scenarios.
- Online surveys: teachers and principals
- Principal interviews.
- Teachers will make defensible, trustworthy judgments against the National Standards.
- What changes in student achievement in reading, writing and mathematics are observed as National Standards are introduced?
- What changes in teachers' professional knowledge and practice are observed as National Standards are introduced?
- In what ways is information from National Standards used by schools to set achievement targets?
- In what ways is information from National Standards used by schools to describe student achievement and progress?
- In what ways is information from National Standards used to provide targeted teaching interventions?
- In what ways is information from National Standards used to identify teachers' professional development needs?
- How do schools use information from National Standards to report to and communicate with parents?
- To what extent do parents understand, value, and use National Standards information about their child?
Data Sources
Student Achievement:
- OTJs-standardised
- assessments
Teachers:
- online surveys
Schools:
- achievement targets-analysis of variance reports
- online surveys: principals
- individual interviews: principals
- reporting formats
Whānau:
- online survey: parents
- school report
Intentions
- Information from National Standards assessments will be used by teachers and schools to monitor student progress and achievement against the Curriculum.
- As a result of using National Standards to monitor achievement and progress some students will be provided with targeted teaching interventions.
- Student achievement will improve.
- Schools will use National Standards assessment information to communicate clearly with families about their child's achievement and progress.
- National Standards information will be used to identify teachers' professional development needs. This will enable these to be addressed more effectively.
Sample
All English-medium, full primary, contributing and intermediate state schools were included in the project's sampling frame. A stratified sampling procedure based on three school characteristics was used, with three groups within each characteristic:
- School decile: one to three, four to seven, eight to ten.
- School type: full primary, contributing, and intermediate
- Region: Auckland, North Island excluding Auckland, and South Island
The sample included positions for 126 schools. Each of the 27 different combinations of school characteristics were represented proportionally to ensure the sample was representative of the larger population of schools in New Zealand.
The sample on which this report is based is smaller than the wider sample as school recruitment was still underway at the time of data collection. Included in this report are the survey responses from 82 principals and the school reporting formats of 70 schools which were received by 31 August 2010.
Table 2 shows the characteristics of the 82 schools whose principals responded to the survey. In general, schools that provided school report formats were a sub-set of schools whose principals completed the survey. All but one of the schools that provided their template for reporting to parents also completed the principal survey.
Note:
| ||||||||
School Type | Region | Decile | ||||||
---|---|---|---|---|---|---|---|---|
Years | Sample | Nationally | Region | Sample | Nationally | Deciles | Sample | Nationally |
1 to 8 | 45% (37) | 45% | Auckland | 24% (20) | 23% | 1 to 3 | 26% (21) | 27% |
1 to 6 | 39% (32) | 34% | North Island* | 45% (37) | 48% | 4 to 7 | 38% (31) | 41% |
7 to 8 | 16% (13) | 21% | South Island | 30% (25) | 29% | 8 to 10 | 37% (30) | 32% |
In terms of the three main stratifying variables the sample is representative, although some of the combinations of characteristics are less well-represented. In particular the following schools are under-represented:
- Low decile Year 1-8 schools in the South Island (under-represented by one school)
- Low decile Year 7-8 schools in the South Island (under-represented by one school)
- Low decile Year 7-8 schools in the North Island, excluding Auckland (under-represented by two schools)
Principal survey
The online principal survey contained four sections, each focused on gathering different information: principals' understandings of National Standards, principals' opinions of National Standards, communication about the National Standards, and mid-year reports to parents. Questions required a mix of closed and open responses. A copy of the survey is included as Appendix A.
Eighty-two principals were emailed a link to the survey on 27 July and asked to complete it within two weeks. An additional six principals were emailed the link to the survey as they confirmed their participation in the project; in total 88 principals were invited to complete the survey. The results of the 82 principals who completed the survey by 31 August are included in the analysis; a response rate of 93%.
Descriptive statistics were used to analyse closed responses. Common themes in open comment fields were identified. Themes identified by greater than five percent of respondents are included in the report. Due to rounding percentages in some tables do not total to 100.
School reporting formats
Eighty-eight schools were asked to provide a copy of the format they used to report to parents mid-year. As with the principal survey, requests were made to 82 of these schools on 27 July with requests sent to the remaining six schools as their participation in the project was confirmed. In total 70 schools provided reports by 31 August and these are included in this analysis; a response rate of 80%.
The document analysis was carried out by four raters. Initially the raters worked as a group to establish the characteristics and features of the report formats. Sixteen characteristics were identified and the researchers then worked independently to analyse the school report formats in terms of the presence or absence of these characteristics. Appendix B provides the details of this analysis and inter-rater reliability statistics are included on page fifteen, alongside the relevant results.
Footnotes
- Italics and bold type are used to indicate the two monitoring and evaluation questions which are the focus of this report. For each question the data sources used and intentions addressed are identified.
- The project is grounded in the analytical framework provided by Stake's countenance model. Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 523-540.
Navigation
Where to find out more
Contact Us
Education Data Requests
If you have any questions about education data please contact us:
Email: Requests Data and Insights
Phone: +64 4 463 8065