Appendix A: Test Languages and Sampling

Languages of assessment

Many participating countries administered tests in more than one language in order to cover their whole (Grade 4) student population (see Table A.1).

Table A.1: Countries assessing in more than one language in PIRLS-05/06

Note
See Appendix A.3 in Mullis, et al., 2007 for details of the other countries’ test languages.

Sample design and size

The sample design for New Zealand was organised around the following criteria.

Selecting schools

There was explicit stratification by language, as follows.

• Stratum 1 − schools where more than 80% of instruction was delivered in Māori. Schools were sampled with equal probabilities.
• Stratum 2 − schools where there was at least one class where more than the language of instruction was in Māori. Schools were sampled with equal probabilities.
• Stratum 3 − schools where the main language was English. There was also implicit stratification (or sorting) to ensure a representative sample according to decile band (high, medium, low) and urbanisation (urban, rural) in this stratum. Small schools (where the measure of size, MOS, was less than 16 Year 5 students) were sampled with equal probabilities; otherwise schools were selected with probability proportional to the Year 5 count.
  

Selecting classes/groups

One class or group of Year 5 students was sampled in Stratum 1. Two classes (one where instruction was delivered in English and the other where instruction was in Māori) were sampled from Stratum 2. If the MOS was large for the English-medium class, then two classes were selected randomly. In Stratum 3, if the MOS was 60 or more then two classes/groups were randomly sampled; otherwise, one class/group was randomly sampled.

Tables A.2A and A.2B present details of the original and achieved New Zealand sample sizes in PIRLS-05/06. Achieved samples were then weighted to represent the student population from which they were drawn. That is, the Year 5 student sample was weighted to reflect the population of New Zealand Year 5 students. It also means that even though 50 schools were sampled to cover Māori-medium Level 1 education, the sampling weights adjust the numbers to reflect the overall population. See the Technical Notes for further elaboration. Alternatively, see Joncas (2007) in the PIRLS 2006 technical report referenced at the end of this report. Details of other countries’ sample designs and achieved samples are also described in this report.

Table A.2A: A summary of New Zealand’s achieved school and student samples at Year 5 in PIRLS-05/06

Table A.2B: A summary of New Zealand’s achieved parent, teacher, and school principal samples in PIRLS-05/06

Exclusions

Countries were able to exclude students from the assessment according to very strict internationally defined criteria. Most importantly, exclusions had to be kept to a minimum (i.e., preferably less than 5%). Exclusions could take place at the school level (i.e., a whole school could be excluded) or within schools. As is the practice in all international assessments in which New Zealand has been involved (e.g., TIMSS and PISA), schools/students were excluded according to the following international criteria.

School-level exclusions

These were done on the basis of:

1. schools being in a small, remote geographical region (in New Zealand this included the Correspondence School)
2. the removal of a language group, possibly due to political, organisational, or operational reasons
   
3. schools that were special education schools.
   

Within-school exclusions

Those eligible for exclusion were:

1. functionally disabled students
2. educable mentally disabled students (although it should be noted that students were not to be excluded solely because of poor academic performance or normal discipline problems)
3. students with limited proficiency in the test language  typically, a student who had received less than 1 or 2 years of instruction in the language of the test could be excluded
   
4. other −  in New Zealand this category was for foreign-fee paying students.
   

New Zealand’s final exclusion rates in PIRLS-05/06 are shown in Table A.3, along with the rates for PIRLS-01.

Table A.3: A summary of New Zealand’s exclusions in PIRLS-01 and PIRLS-05/06