Summary

The School Sample Monitoring and Evaluation Project is a three year project, established to evaluate the implementation of National Standards using a large representative sample of schools. This is the first report from the project and describes results gathered using two methods: an online principal survey and an analysis of the formats schools used for mid-year reporting to parents. As this data was gathered six months after National Standards were first put in place in schools the report provides early information on the implementation of National Standards.

The project has two purposes: to describe the implementation of National Standards within schools, and to monitor and systematically evaluate the effect of National Standards on students, teachers, schools and whānau. The descriptive element of the project is focused around twelve monitoring and evaluation questions, while evaluation is based on seven intentions which describe the intended effects of National Standards. A variety of data sources are employed (see Table 1 for an overview).

Table 1 presents the project's methodology and identifies the two monitoring and evaluation questions addressed in this report:

1. How do schools use information from National Standards to report to and communicate with parents?
7. To what extent are the National Standards understood as a set of common expectations for student achievement?

Note that the relevant data sources and statements of intent for each question are also identified in Table 1.

Table 1: Methodology¹,²

Sample

All English-medium, full primary, contributing and intermediate state schools were included in the project's sampling frame. A stratified sampling procedure based on three school characteristics was used, with three groups within each characteristic:

1. School decile: one to three, four to seven, eight to ten.
2. School type: full primary, contributing, and intermediate
3. Region: Auckland, North Island excluding Auckland, and South Island

The sample included positions for 126 schools. Each of the 27 different combinations of school characteristics were represented proportionally to ensure the sample was representative of the larger population of schools in New Zealand.

The sample on which this report is based is smaller than the wider sample as school recruitment was still underway at the time of data collection. Included in this report are the survey responses from 82 principals and the school reporting formats of 70 schools which were received by 31 August 2010.

Table 2 shows the characteristics of the 82 schools whose principals responded to the survey. In general, schools that provided school report formats were a sub-set of schools whose principals completed the survey. All but one of the schools that provided their template for reporting to parents also completed the principal survey.

In terms of the three main stratifying variables the sample is representative, although some of the combinations of characteristics are less well-represented. In particular the following schools are under-represented:

• Low decile Year 1-8 schools in the South Island (under-represented by one school)
• Low decile Year 7-8 schools in the South Island (under-represented by one school)
• Low decile Year 7-8 schools in the North Island, excluding Auckland (under-represented by two schools)

Principal survey

The online principal survey contained four sections, each focused on gathering different information: principals' understandings of National Standards, principals' opinions of National Standards, communication about the National Standards, and mid-year reports to parents. Questions required a mix of closed and open responses. A copy of the survey is included as Appendix A.

Eighty-two principals were emailed a link to the survey on 27 July and asked to complete it within two weeks. An additional six principals were emailed the link to the survey as they confirmed their participation in the project; in total 88 principals were invited to complete the survey. The results of the 82 principals who completed the survey by 31 August are included in the analysis; a response rate of 93%.

Descriptive statistics were used to analyse closed responses. Common themes in open comment fields were identified. Themes identified by greater than five percent of respondents are included in the report. Due to rounding percentages in some tables do not total to 100.

School reporting formats

Eighty-eight schools were asked to provide a copy of the format they used to report to parents mid-year. As with the principal survey, requests were made to 82 of these schools on 27 July with requests sent to the remaining six schools as their participation in the project was confirmed. In total 70 schools provided reports by 31 August and these are included in this analysis; a response rate of 80%.

The document analysis was carried out by four raters. Initially the raters worked as a group to establish the characteristics and features of the report formats. Sixteen characteristics were identified and the researchers then worked independently to analyse the school report formats in terms of the presence or absence of these characteristics. Appendix B provides the details of this analysis and inter-rater reliability statistics are included on page fifteen, alongside the relevant results.

Footnotes

1. Italics and bold type are used to indicate the two monitoring and evaluation questions which are the focus of this report. For each question the data sources used and intentions addressed are identified.
2. The project is grounded in the analytical framework provided by Stake's countenance model. Stake, R. E. (1967). The countenance of educational evaluation. Teachers College Record, 523-540.