Section 2: Student Results

Section 2 examines the reading literacy achievement of New Zealand’s Year 5 students from a national perspective. First there is a recap of the results in an international context, followed by a detailed overview of the findings by ethnicity and gender. Comparisons are also made with the achievements of the 2001 year cohort.

Reading literacy achievement in 2005/2006

Figure 2.1 presents the means and distributions for all participating countries.   Because PIRLS has been designed to measure trends in achievement over time, the PIRLS reading achievement scale was set in 2001 to have a mean of 500 and a standard deviation of 100, and will remain constant across the assessment cycles. In addition, while some sets of reading texts and associated questions are released after a cycle, others sets are retained to be used across cycles in order to be able to measure trends. The following points are the key results pertaining to New Zealand Year 5 students in an international context in 2005/2006.

• The mean reading score for New Zealand Year 5 students was 532, which was significantly higher¹¹ than the PIRLS scale mean of 500.
• The mean score for Year 5 students was similar to that of students in three countries – Chinese Taipei, Scotland, and the Slovak Republic − but significantly lower than the mean scores for 17 countries, including England and the United States.
• New Zealand’s Year 5 students achieved at the same level in 2005/2006 as their 2001 counterparts.
  
• Three countries that had a similar performance to New Zealand in 2001 had demonstrated significant improvements in average achievement by 2005/2006. These were Singapore, Hong Kong SAR, and the Russian Federation.¹²
  

To assist readers with understanding the economic and educational context of participating countries, Figure 2.1 includes the value of each country’s Human Development Index provided by the United Nations Development Programme. The index ranges from 0 to 1. Countries with high values on the index have long life expectancy, high levels of participation in education and adult literacy, and a good standard of living as measured by Gross National Product per capita. The majority of countries that scored above the PIRLS scale mean also had index values greater than 0.9 including New Zealand (0.936).

Two of the low-performing countries had the lowest values on the index (approximately 0.64 and 0.653). There were, however, some higher-performing countries with values on the index which ranged from about 0.797 (Russian Federation) to 0.869 (Hungary).

Figure 2.1 also shows the number of years of schooling and the mean age of the students assessed in PIRLS-05/06. Overall, the relationship between the average age of students and countries’ mean achievement was variable. In some countries (and provinces), younger students had higher mean achievements than countries with older students, and vice versa. For example, students in some higher-performing countries such as the Russian Federation and Sweden tended to be, on average, older than New Zealand students, while in Hong Kong SAR and Italy it was not the case. Luxembourg students were typically the oldest. Luxembourg made the decision to assess their Grade 5 students (equivalent to New Zealand Year 6) because of concerns about the students’ preparedness for taking an assessment in one of the two instructional languages (i.e., French and German), given that it is not their home language (Luxembourgish). Students in three of the Canadian provinces were, on average, slightly younger than many of their international counterparts.

Although the mean age of students in New Zealand, England, Scotland, and Trinidad and Tobago was about 10 years, because of the school starting age of 5 years they had also received at least one more year of schooling than many of their international counterparts, who had started school at age 6 or 7.

Figure 2.1: Distribution of countries’ reading achievement in PIRLS-05/06

Notes:

Standard errors appear in parentheses. Because results are rounded, some figures may appear inconsistent. The Canadian provinces took part in PIRLS-05/06 as benchmarking participants.

* Represents years of schooling counting from the first year of ISCED Level 1.

** Taken from United Nations Development Programme's Human Development Report 2006, p. 283-286, except for Chinese Taipei taken from Directorate General of Budget, Accounting and Statistics, Executive Yan, R.O.C Statistical Yearbook 2005. Data for the Belgium (Flemish) and Belgium (French) communities are for the entire country of Belgium. Data for England and Scotland are for the United Kingdom.

† Met guidelines for sample participation rates only after replacement schools were included.

‡ Nearly satisfying guidelines for sample participation rates after replacement schools were included.

2a National Defined Population covers less than 95% of National Desired Population.

2b National Defined Population covers less than 80% of National Desired Population. 

Source: IEA Progress in International Reading Literacy Study (PIRLS) 2006. Adapted from Exhibits 1.1 and 1.2 in Mullis, et al., 2007.

 

Reading literacy achievement and ethnicity

In New Zealand, five broad ethnic classifications are used to describe students’ ethnicity in New Zealand: Pākehā/European, Māori, Pasifika, Asian, and Other ethnic groups.¹³ Figure 2.2 shows the breakdown of the estimated Year 5 population by ethnicity in the two cycles of PIRLS.¹⁴

Figure 2.2: Estimated Year 5 student population in each ethnic grouping in PIRLS-01 and PIRLS-05/06 (weighted percentages)*

 

Notes
Standard errors (SE) appear in parentheses.
Percentages are adjusted for missing responses. Missing ethnicity information was approximately 2 percent in 2001 and 1 percent in 2005.
* See TN 1 for a brief description of the weighting used in PIRLS.

Figure 2.3 presents the mean reading literacy score and the distribution of scores for each of New Zealand’s ethnic grouping in PIRLS-05/06.¹⁵ The average achievement of Pākehā/European students (552) was about the same as for Asian students (550), with around three-quarters (76%) of students in both groups achieving scores equivalent to or above the PIRLS scale mean of 500.

Māori (483) and Pasifika (479) Year 5 students tended to achieve at about the same level, but somewhat lower than the PIRLS scale mean (500). Less than half of Māori students (46%) and Pasifika students (40%) achieved reading scores equivalent to or above the PIRLS scale mean.

Figure 2.3: Distribution of Year 5 students’ reading literacy scores in 2005/2006, by ethnic grouping

 

Notes
Standard errors appear in parentheses.
The 5th and 95th percentiles for Pasifika and Asian students should be interpreted with some caution due to the relatively small (achieved) sample sizes on which these analyses are based. The distribution of scores for Year 5 students in the ‘Other ethnic groups’ category is not shown because of the very small proportion (approximately 2%) they form of the overall population.
See Table B.1 in Appendix A for details of the percentiles and standard errors for 2001 and 2005/2006.

Range of scores

At 290, the range of scores for New Zealand (i.e., the difference between the 5th and the 95th percentiles) was wider − with the exception of England (290) and Bulgaria (276), which had a similarly large spread − than the range for many other higher-performing countries. By way of comparison, the range for the Netherlands was 174 (see Chamberlain, 2007b; Mullis, et al., 2007 for further details). As well as illustrating the variation in achievement across New Zealand’s four main ethnic groupings, Figure 2.3 also highlights the fact that there are high-performing and low-performing students in all ethnic groupings. The range was greater for Māori (290) than for Pākehā/European (266), Pasifika (255), and Asian students (246).

Differences among the ethnic groupings and effect sizes

Both Asian and Pākehā/European students achieved significantly higher reading literacy scores, on average, than Māori and Pasifika students.¹⁶ Effect sizes are a useful way of illustrating the magnitude of the achievement difference between two groups of students.¹⁷ In this report, the effect size is calculated as the difference between the means for two groups in question, divided by the pooled standard deviation of the two groups (i.e., Cohen’s d or the normalised difference between the means). Table B.2 in Appendix B reports the effect sizes for the differences among Pākehā/European, Māori, Pasifika, and Asian students’ mean reading literacy scores for the two PIRLS assessments, 2001 and 2005/2006.

For the purposes of this discussion, the effect size is considered large if the value is greater than 0.75, of medium size if the value is equal to 0.35 or higher but less than 0.75, and small if less than 0.35.¹⁸ Essentially, the calculated effect sizes highlight the large differences in mean reading literacy achievement as measured by PIRLS among the ethnic groupings in 2005/2006. Of note are the effect sizes for the differences between Pākehā/European and Māori (d = 0.84) and between Pākehā/European and Pasifika (d = 0.92).

Any change between 2001 and 2005/2006?

Table 2.1 below shows the mean scores for students in each ethnic grouping for the two cycles. The biggest shift was exhibited for Asian students, who on average achieved 11 scale score points higher than their 2001 counterparts. However, neither the average increase in Asian students’ reading achievement nor the changes found for the other three main ethnic groups were statistically significant.

Table 2.1: Year 5 students’ mean reading literacy scores in 2001 and 2005/2006, by ethnic grouping

Year 5 student group	Mean reading literacy scores for each PIRLS       assessment cycle		Change     2001−2005/2006
	2001	2005/2006
Pākehā/European	552 (3.4)	552 (2.4)	− 1 (4.2)
Māori	481 (5.5)	483 (3.6)	+2 (6.6)
Pasifika	481 (7.2)	479 (6.7)	- 2 (9.9)
Asian	540 (9.9)	550 (5.3)	+11 (11.2)
Other ethnic groups	~ ~	539 (9.6)	N.C. †

Notes
Standard errors appear in parentheses. Because results are rounded, some figures may appear inconsistent.
Tilde (~) indicates the achieved sample size was too small (N < 50) to calculate the mean.See TN 7 in the Technical Notes for details.
^† Not calculated.

Reading literacy achievement and gender

Internationally, both New Zealand Year 5 girls and boys typically achieved above their respective international means. The following points were the highlights for Year 5 students in an international context.

• New Zealand boys (520) scored on average significantly above the international mean for boys (492).
• New Zealand girls (544) scored on average significantly above the international mean for girls (509).
  
• At 24 scale score points, the average difference between New Zealand girls’ and boys’ scores was the fifth largest to be observed internationally (an average difference of 17 scale score points).
  

Figure 2.4 presents the mean reading literacy score and the distribution of scores for New Zealand Year 5 girls and boys separately, along with the (weighted) percentage of girls and boys in the Year 5 population.

Figure 2.4: Distribution of Year 5 students’ reading literacy scores in 2005/2006, by gender

Notes
Standard errors appear in parentheses.
See Table B.3 in Appendix B for the percentiles and standard errors for 2001 and 2005/2006.

Range of scores

As illustrated in Figure 2.4, the range of scores was greater for Year 5 boys (298) than for Year 5 girls (272). The figure also illustrates the weaker performance of some Year 5 boys when compared to that of girls; 5 percent of Year 5 boys scored below 357 (5th percentile), while the corresponding 5th percentile for Year 5 girls was 40 scale score points higher at 399.

Gender differences

The mean reading literacy score for girls in all but two countries in PIRLS was significantly higher than the mean for boys, with the average difference greatest in Kuwait (67 scale score points) and the smallest (and non-significant) in Luxembourg (3) and Spain (4). As noted in the first part of this section, New Zealand recorded one of the largest differences (24) between girls’ and boys’ mean achievement.

Again, effect sizes are a useful way to understand the magnitude (size) of the difference between New Zealand girls’ and boys’ mean achievement. Using the same approach taken to examine differences among the New Zealand ethnic groups, an effect size was calculated to look at the size between New Zealand girls’ and boys’ mean achievement. This was calculated to be d = 0.28 in 2005/2006, which indicates that the difference between girls and boys was relatively small; this was also the case in 2001 (d = 0.29). An examination of the data found that the gender difference was greater among New Zealand’s lower reading achievers than its higher reading achievers. For example, the average difference between the achievement of Year 5 girls and boys who scored below the PIRLS scale mean (500) was 18 scale score points, compared with an average difference of just 6, albeit still significant, for those who scored 500 or more.

An examination of the overall New Zealand distribution also illustrates this observation. The proportion of boys who achieved 592 (i.e., the 75th percentile) or a higher score was 22 percent, 6 percentage points lower than the proportion of girls (28%). At the lower end of the performance range, the proportion of Year 5 boys who scored less than 478 (i.e., the 25th percentile) was 30 percent, compared with 20 percent of Year 5 girls. That is, girls were over-represented among higher achievers and boys were over-represented among lower achievers.

Any change between 2001 and 2005/2006?

The mean scores for Year 5 boys and girls for the two PIRLS cycles are reported in Table 2.2, along with the means for girls and boys calculated for the 26 countries taking part in both PIRLS-01 and PIRLS-05/06 (i.e., the ‘trend’ countries). While there was no significant change in the girls’ mean for the 26 trend countries, the boys’ mean increased by an average of 5 scale score points; the increase was statistically significant.

Table 2.2: Mean reading literacy scores for New Zealand Year 5 students and the 26 trend countries in 2001 and 2005/2006, by gender

Comparison group	Mean reading literacy scores
	Girls		Boys
	2001	2005/2006	2001	2005/2006
New Zealand	542 (4.7)	544 (2.2)	516 (4.2)	520 (2.9)
Trend countries*	525 (0.9)	526 (0.7)	505 (1.1)	510 (0.7)  ▲

Notes
Standard errors appear in parentheses.
* Means calculated for the 26 countries participating in both PIRLS-01 and PIRLS-05/06. Includes New Zealand Data
▲ The increase was statistically significant at the 5 percent level.

The gender difference observed in New Zealand remained relatively large when compared with the differences in other countries. The small increases exhibited by New Zealand girls and boys during the period 2001 to 2005 (2 and 4 scale score points higher respectively) were not found to be of statistical significance.

Reading literacy achievement, ethnicity, and gender

Since ‘girls’ and ‘boys’ both represent diverse groups of students, their performance in PIRLS is viewed in the context of their ethnic identity.

Pākehā/European (564) and Asian (562) girls, on average, performed well above the international girls’ mean of 509. Furthermore, both groups of girls had the greatest proportion (both 82%) of any sub-group scoring 500 (the PIRLS scale mean) or higher.

At 498, the mean score for Māori girls was significantly lower than the international girls’ mean (509) but was about the same as the PIRLS scale mean; 52 percent of Māori girls achieved a score at or above this level (500). Pasifika girls (486), on average, achieved scores below the international mean for girls; furthermore, only about two-fifths (42%) of these students achieved scores at or above the PIRLS scale mean (500).

About 70 percent of Pākehā/European boys and Asian boys scored above the PIRLS scale mean of 500; the means for both groups was 540 (c.f. the international boys’ mean of 492). At 469, Māori boys’ mean score was significantly lower than the international boys’ mean; so too was the mean for Pasifika boys (471). About 40 percent of Māori boys and Pasifika boys achieved scores above the PIRLS scale mean of 500.

Gender differences

Pākehā/European, Māori, and Asian girls generally achieved significantly higher scores than their respective male counterparts.¹⁹ The one exception was that Pasifika girls typically achieved 15 points higher than Pasifika boys, but the difference between the two groups was not statistically significant.²⁰

Were there significant gender differences among those who had ‘above average’ achievement (scored 500 or higher) and those who had ‘below average achievement (scored less than 500) in each of the ethnic groupings?  The finding observed for all Year 5 students noted in the previous section held for Pākehā/European students only (see Figure 2.5). That is, although significant gender difference were found among Pākehā/European students who scored above 500 (an average 6 of scale score points), the gender difference was greater for the group who scored below 500 (an average of 23 scale score points).

Figure 2.5: Mean differences between Year 5 girls’ and boys’ reading literacy achievement scores in 2005/2006, by ethnic grouping

 

Notes
The data points are the mean differences between girls’ and boys’ mean reading literacy achievement in each achievement category (above ▲ and • below the PIRLS international scale mean). A value close to zero indicates that the difference between girls’ and boys’ mean achievement is small. The vertical lines extending from a data point shows the 95 percent confidence interval around the difference of the mean (i.e. ± ▲ 2 standard errors of the difference).
See Appendix B.4 in Appendix B for the mean differences and standard errors.

For Māori, the average difference (a difference of 20 scale score points) was only significant between girls and boys who scored below 500. Gender differences between students who were in the ‘above average’ achievers category and who were in the ‘below average’ category were not found to be significant for the Asian and Pasifika groupings.

Any change between 2001 and 2005/2006?

Tables 2.3 and 2.4 present the mean scores for girls and boys in each ethnic grouping respectively. Pasifika girls in 2005/2006 scored an average of 13 scale score points lower than their 2001 counterparts, while Asian boys scored an average of 14 scale score points higher. These changes were not found to be of statistical significance.

Table 2.3: Year 5 girls’ mean reading literacy scores in 2001 and 2005/2006, by ethnic grouping

Ethnic grouping	Year 5 girls’ mean reading literacy scores for each PIRLS assessment cycle		Change 2001−2005/2006
	2001	2005/06
Pākehā/European	567 (4.7)	564 (2.8)	–4 (5.5)
Māori	495 (7.2)	498 (4.6)	+3 (8.5)
Pasifika	500 (10.1)	486 (6.0)	–13 (11.7)
Asian	560 (13.7)	562 (5.4)	+2 (14.8)
All New Zealand*	542 (4.7)	544 (2.2)	+2 (5.2)

Notes 
Standard errors appear in parentheses. Because of rounding, some results may appear inconsistent.
None of the changes were statistically significant at the 5 percent level.
* All girls. In 2001 there were insufficient data to report the mean by gender. In 2005 the girls’ mean was 542 (12.1).

Table 2.4: Year 5 boys’ mean reading literacy scores in 2001 and 2005/2006, by ethnic grouping

Ethnic grouping	Year 5 boys’ mean reading literacy scores for each PIRLS assessment cycle		Change 2001−2005/2006
	2001	2005/06
Pākehā/European	539 (4.2)	540 (3.3)	+2 (5.3)
Māori	466 (6.5)	469 (4.7)	+3 (8.0)
Pasifika	465 (10.5)	471 (9.4)	+6 (14.1)
Asian	526 (11.9)	540 (7.3)	+14 (14.0)
All New Zealand*	516 (4.2)	520 (2.9)	+4 (5.1)

Notes
Standard errors appear in parentheses. Because of rounding, some results may appear inconsistent.
None of the changes were statistically significant at the 5 percent level.
* All boys. In 2001 there were insufficient data to report the mean by gender. In 2005, the boy’ mean was 536 (14.2).
 

Footnotes

11. As noted in footnote 6, this refers to statistical significance at the 5 percent level. 
    
12. While New Zealand’s average reading literacy achievement did not change, New Zealand’s standing relative to the 25 other countries with comparable data from both cycles changed from 11th in 2001 to 14th in 2005/2006. (This excludes the results from two benchmarking Canadian provinces, Ontario and Quebec, which had been combined for reporting in 2001.)
13. Māori refers to the indigenous people of New Zealand. Pasifika includes people who identify themselves as Cook Islands Māori, Samoan, Tongan, or Niuean. Asian includes people who identify as being Chinese, Indian, Korean, or Vietnamese. The Other ethnic groups include those from Middle Eastern (e.g., Iraqi,) African (e.g., Somali) or South American (e.g., Chilean) backgrounds. Pākehā/European includes people who, for example, identify themselves as of English, Scottish, or Irish heritage, or are of European (such as Dutch or Polish) background.
    
14. In 2001 ethnicity data reflect information supplied by schools; in 2005 the data reflect information supplied by schools and students’ self-identification. The Ministry of Education data for all domestic Year 5 students in 2005 were Pākehā/European, 58%; Māori, 23%; Pasifika, 9%; Asian, 8%; and Other ethnic groups, 2%. (Source: www.educationcounts.govt.nz)
    
15. The results are presented for all Year 5 students and do not reflect their language of instruction.
16. The mean scores for Pākehā/European and Asian students were statistically significantly higher than the mean scores for Māori and Pasifika students (adjusted for multiple comparisons, see TN 5 in the Technical Notes for details.). There was no significant difference between the mean scores for Pākehā/European and Asian students, nor was there a difference between Māori and Pasifika students.
    
17. For details on the calculation and interpretation of effect size, see TN 6 in the Technical Notes.
    
18. This interpretation of large, medium, and small is a variation of the interpretation commonly used for Cohen’s d (large = 0.8; medium = 0.5; small = 0.2).
    
19. The differences between means, with standard errors of the differences, were 23 (3.6), 30 (6.1), and 21 (7.8) respectively.
20. Difference between means, with standard error of the difference, 15 (8.6).