PIRLS 2005/2006 in New Zealand: An overview of national findings from the second cycle of the Progress in International Reading Literacy Study (PIRLS)

Section 3: The PIRLS International Benchmarks

Section 3 looks at the performance of Year 5 students in each sub-group in relation to the PIRLS international benchmarks of reading. A benchmark describes the types of comprehension skills and strategies Grade 4 students, or in New Zealand’s case Year 5 students, demonstrated when they encountered particular questions in the PIRLS texts.  

PIRLS international benchmarks of reading

Four points on the reading achievement scale were identified for use as international benchmarks.²¹ These are: the Advanced International Benchmark, the High International Benchmark, the Intermediate International Benchmark, and the Low International Benchmark.²² The descriptions for each international benchmark are summarised in Box 3.1. The benchmarks are also cumulative, in that students who demonstrated the skills and strategies at a given benchmark also demonstrated the skills associated with the lower benchmarks. It is worth remembering that the descriptions do not profess to encompass all reading situations 10-year-olds encounter. However, they do reflect the types of PIRLS texts students were asked to read, the types of questions students were able to answer successfully, and, for multiple-mark constructed response questions, the quality of their responses.

Box 3.1: The PIRLS-05/06 international reading benchmarks for Grade 4 (Year 5 equivalent)

Source: IEA Progress in International Reading Literacy Study (PIRLS) 2006. Adapted from Exhibits 2.4, 2.9, 2.14, and 2.19 in Mullis, et al., 2007.

The following points set out the key results for New Zealand Year 5 students in an international context.

• New Zealand recorded a relatively large proportion (13%) of students reaching the Advanced International Benchmark, almost double the international median of 7 percent.
• Forty-five percent of New Zealand students reached the High International Benchmark, compared with the international median of 41 percent.
• About three-quarters (76%) of New Zealand students reached the Intermediate International Benchmark, the same as the international median.
• Ninety-two percent of New Zealand students reached the Low International Benchmark, a little lower than the international median of 94 percent.
  
• There was no change from 2001 to 2005/2006.
  

PIRLS benchmarks, gender, and ethnicity

Table 3.1 shows the proportions of Year 5 students who reached the PIRLS international benchmarks, by gender and ethnicity. Year 5 girls, Pākehā/European students, and Asian students were well represented among the higher-performing students, with about half or more students in these groups achieving at or above the High International Benchmark (i.e., scored equal to or higher than 550). Māori, but particularly Pasifika students were under-represented among the higher-performing group, with just under one-quarter of Māori students and less than one-fifth of Pasifika achieving at this level. Year 5 boys were also, although to a lesser extent, under-represented, with just two-fifths of students among the higher-performing group. Both Māori and Pasifika students were also less likely than students in the other groups to reach the Intermediate International Benchmark (i.e., achieve a score of 475 or higher), with more than two-fifths of students from each group scoring below this level.

Table 3.1: Percentage of students reaching the PIRLS international reading benchmarks in 2005/2006, by gender and ethnic grouping

Notes
Standard errors appear in parentheses. Because of rounding some figures may appear inconsistent.
* All Year 5 students, including students in other ethnic groups.

Any change between 2001 and 2005/2006?

Table B.5 in Appendix B presents the percentages of students who reached the international benchmarks in 2001. Consistent with the overall pattern observed for New Zealand, there were no changes of note for any of the Year 5 student sub-groups. 

Lower-achieving students

In recent times there has been commentary about the high proportion of students who ‘fail’ in New Zealand’s education system, with the percentage cited sometimes as high as 20 percent. PIRLS, along with TIMSS²³ and PISA,²⁴ are often used to support this notion of ‘failure’, along with the assertion that New Zealand has the widest spread in achievement compared with other countries, enhanced by the so-called ‘large tail’. The international studies are not designed to measure failure; they are designed so that countries can look at, among many other variables, strengths and weaknesses of cohorts of students in particular learning areas. Furthermore, the notion of a large tail in achievement does not hold across all learning areas; nor does it hold for every cohort of students (e.g., Chamberlain, 2007a).

PIRLS is designed to be able to discriminate between those students who demonstrate very well-developed comprehension skills for their age and those who have weak comprehension skills. The skills and strategies are tested through texts and stories, which may or may not be familiar in style, format, and length; PIRLS is not a test of reading per se (decoding).

The purpose of the following analysis is to describe the New Zealand group of students who, using the international context, had weaker comprehension skills, or are lower-achieving students. These students did not reach the PIRLS Intermediate International Benchmark (i.e., scored below 475). Approximately one-quarter of Year 5 students (24%) fell into this category.²⁵ Some of this group also did not reach the Low International Benchmark; 8 percent of Year 5 students scored below 400 overall, while 16 percent scored at least 400 but less than 475.

It should be remembered, though, that New Zealand’s Year 5 students were also well represented among the group of readers with very strong reading comprehension skills.

So what does this mean in terms of reading comprehension as measured by PIRLS? When reading literary texts, lower-achieving students had difficulty with:

• identifying central events, plot sequences, and relevant story details
• making straightforward inferences about the attributes, feelings, and motivations of the main character
• making connections across parts of the text.
  

Some students in this group even had difficulty demonstrating that they could:

• recognise explicitly stated detail
• locate a specified part of the story and make an inference clearly suggested by the text.
  

When reading informational texts, lower-achieving students had difficulty with:

• locating and extracting one or two pieces of information
• making straightforward inferences from a single part of the text
• using subheadings, text boxes, and illustrations to locate parts of the text.
  

Some of this group of students even had difficulty demonstrating they could:

• locate and reproduce explicitly stated information, even from the beginning of a text or in a clearly defined section
• make a straightforward inference clearly suggested by the text.
  

So who was in the lower-achieving group? Figures 3.1 through to 3.3 show the composition of the lower-achievers group according to students’ gender, ethnicity, and gender and ethnicity.²⁶ The approach taken to describe the information is firstly to compare the proportion of lower achievers in each sub-group with the overall proportions in the Year 5 population. Are particular sub-groups over-represented among the group of lower achievers?  The second approach (Figures 3.4 and 3.5) is to show the proportion of each sub-group that were in the lower-achievers group.

Composition of the lower-achievers group

Figure 3.1 shows the composition of the group according to students’ gender. While the Year 5 population was estimated to be 51 percent boys and 49 percent girls, most of those in the lower-achievers group were boys (62%).

Figure 3.1: Gender composition of the Year 5 lower-achievers group in 2005/2006

Notes
Standard errors (SE) appear in parentheses.
The proportion of all Year 5 students who reached the PIRLS Intermediate International Benchmark was 76 percent (SE 1.0%); the proportion who did not reach this benchmark was 24 percent (SE 1.0%).

Figure 3.2 shows the composition of the lower-achievers group according to their ethnic identity. Although the group comprised mostly Pākehā/European students (41%), 38 percent of the group were Māori students, 1.5 times higher than the estimated proportion of Māori in the Year 5 population (21%).²⁷ Pasifika students were also over-represented in this group (14% c.f. 7% in the population).

Figure 3.2: Ethnic composition of the Year 5 lower-achievers group in 2005/2006

Notes
Standard errors (SE) appear in parentheses.
The proportion of all Year 5 students who reached the PIRLS Intermediate International Benchmark was 76 percent (SE 1.0%); the proportion who did not reach this benchmark was 24 percent (SE 1.0%).

The interaction between gender and ethnicity was considered by means of data shown in Figure 3.3. The lower-achievers group comprised mostly Pākehā/European boys (28%) and Māori boys (23%). However, in terms of how these data compared with the proportions in the overall population, Māori boys (c.f. 11% in the overall population) and Pasifika boys (c.f. 4%), Pasifika girls (c.f. 3%), and, to a lesser extent, Māori girls (c.f. 10%) were over-represented in the lower-achievers group.

Figure 3.3: Ethnic and gender composition of the Year 5 lower-achievers group in 2005/2006

Notes
Standard errors (SE) appear in parentheses
The proportion of all Year 5 students who reached the PIRLS Intermediate International Benchmark was 76 percent (SE 1.0%); the proportion who did not reach this benchmark was 24 percent (SE 1.0%).

Proportions of sub-groups in the lower-achievers group

The second approach undertaken to describe the characteristics of lower achievers, as measured by PIRLS, was to look at the proportion of a sub-group that fell into a particular achievement category. Figure 3.4 shows the proportions of Year 5 girls and Year 5 boys who scored below 475 (i.e., did not reach the PIRLS Intermediate International Benchmark). Just under one-fifth of girls were in the lower-achievers group compared with nearly one-third of boys.  

Figure 3.4: Percentage of Year 5 girls and boys who were in the lower-achievers group in 2005/2006

Notes
Standard errors (SE) appear in parentheses.
The proportion of all Year 5 students who reached the PIRLS Intermediate International Benchmark was 76 percent (SE 1.0%); the proportion who did not reach this benchmark was 24 percent (SE 1.0%).

Figure 3.5 shows the proportion of students from each ethnic grouping who were in the lower-achieving group. The proportions of Māori and Pasifika students were nearly three-times those of Pākehā/European and Asian in the lower-achieving category.

Figure 3.5: Percentage of Year 5 students from each ethnic grouping who were in the lower-achievers group in 2005/2006

Notes
Standard errors (SE) appear in parentheses.
The proportion of all Year 5 students who reached the PIRLS Intermediate International Benchmark was 76 percent (SE 1.0%); the proportion who did not reach this benchmark was 24 percent (SE 1.0%).

When looking at gender and ethnicity together, about one-fifth of Pākehā/European boys (21%) were by definition lower achievers in reading literacy, about double that for Pākehā/European girls (11%). The proportions for Asian students were about the same as for Pākehā/European students (i.e., Asian boys, 20%; Asian girls, 12%). Of concern, however, are the figures for Māori and Pasifika students, and in particular boys from these two groups. More than one-third of Māori girls (36%) and two-fifths of Pasifika girls (43%) were in the group of lower-achievers. However, by far the highest proportions were those for Māori boys (51%) and Pasifika boys (49%).

Odds ratios

In order to summarise the information presented in the preceding discussion, the odds of Year 5 students with particular attributes being in the lower-achievers group were determined. Odds are a way of representing the probability or chance of an event.

An odds ratio (OR), a type of effect size, is calculated by dividing the odds of an event (e.g., having a demographic characteristic and being in the lower-achievers group) by the odds of the control event (e.g., not having the demographic characteristic and being in the lower-achievers group). If the OR is greater than 1, the event is more likely to happen than not; if the OR is less than 1, then the chances become less likely, particularly as it approach zero. (See TN 8 in the Technical Notes for further details.)  The ORs and confidence intervals for each demographic characteristic under scrutiny are reported in Tables B.6A and B.6B in Appendix B.

Of note are the ORs for boys, Māori and Pasifika students: they were all greater than 1. That is, the likelihood of being a lower achiever was greater for these students than students who did not share these demographic characteristics. The observations from the analysis can be summarised as follows.

• Year 5 boys had nearly twice the odds of being in the lower-achievers group than Year 5 girls (i.e., 0.42 c.f. 0.23).
• Māori (0.77) and Pasifika (0.86) students had at least three times the odds of being in the lower-achievers group than non-Māori (0.23) and non-Pasifika (0.29) students.
  
• Māori boys (1.04), followed by Pasifika boys (0.97), and Pasifika girls (0.76) had the highest odds of being in the lower-achievers group compared with all other students.
  

Footnotes

21. As in 2001, the scale anchoring method was used by the international researchers and a team of reading experts to develop the descriptions of student performance at the four different points. As well as a quantitative component used to identify the questions that discriminated between successive points on the scale, the process used qualitative methods to develop the descriptions of performance. The methodology is described in the PIRLS 2006 technical report (Martin, et al., 2007).
    
22. It is important to note that these benchmarks are not comparable to the four benchmarks reported for PIRLS-01. For example, the Advanced International Benchmark used in 2005/2006 is not equivalent to the Top 10% Benchmark used in 2001. In 2001, percentiles (specifically, the 25th, 50th, 75th, and 90th) were used to identify the four benchmark points on the scale. Because there was a strong likelihood that the percentiles would change due to more countries participating, and there being a greater variation in performances as new countries join, four new points were identified. These four points were fixed for this and future cycles, which means that countries can determine with more certainty any changes over time. The four new points have also been used retrospectively to see if there has been any change since 2001.
    
23. Trends in International Mathematics and Science Study.
    
24. Programme for International Student Assessment.
25. This proportion compares, for example, with 8 percent of Hong Kong students, 13 percent of Ontario’s students, 14 percent of Singaporean students, and 22 percent of England’s students.
    
26. In Section 5, the composition of the Year 5 lower-achievers group is also considered in terms of whether or not they spoke the test language. In Section 6, the composition is considered by the decile (band) of the school the Year 5 students attended in 2005.
    
27. See Section 2 for details of the estimated Year 5 population by ethnicity.