New Zealand students' engagement with the PIRLS 2006 reading passages Publications
Publication Details
This paper summarises the findings from an ‘enjoyment’ survey administered to approximately 6300 New Zealand Grade 4 (Year 5) children. The research investigated whether or not the students engaged with the PIRLS 2006 reading passages, why they liked them, and whether their engagement related to their reading achievement as measured by PIRLS.
Author(s): Megan Chamberlain and Robyn Caygill
Date Published: November 2008
Page Contents:
Introduction
Background
The Progress in International Reading Literacy Study (PIRLS) 2001 showed that, when compared with other countries, New Zealand had one of the largest differences between boys' and girls' mean reading achievement (Mullis, Martin, Gonzalez, & Kennedy, 2003). Although the reported gender difference was consistent with other national and international studies (Crooks & Flockton, 2005; Satherley, 2006; Wagemaker, 1993), it was the magnitude of the mean difference relative to that of other countries which raised the researchers' interest in the area. Was the gender difference in reading literacy achievement exacerbated by the nature of the reading passages used in PIRLS 2001? PIRLS 2006 thus provided an opportunity to determine whether or not New Zealand's Grade 4 students liked the passages they read and then assessed.
The New Zealand Context
PIRLS, one of three international studies in which New Zealand regularly participates that provide information on the education system's performance in an international context, is used extensively by the Ministry of Education to identify strengths and weaknesses in the system. Building on a comprehensive gender-related literature review undertaken by Alton-Lee and Praat (2000) that highlighted differences between boys' and girls' literacy outcomes, a recent synthesis of data from the national and international assessment studies, including PIRLS 2001, emphasised again the differences between girls and boys in favour of girls in this area. Furthermore, significant achievement gaps between girls' and boys' literacy achievement start to appear soon after the beginning of compulsory schooling and persist until the end (Ministry of Education, 2007). However, New Zealand boys and girls are not viewed as homogeneous groups, with gender usually considered in the context of ethnicity and social class. That is, studies look at which girls and which boys have weaker performance, and which ethnic groups have the largest gender differences.
In New Zealand, ethnicity is a key social attribute used along with other features to describe the population. It refers to the ethnic group or groups with which people identify, or to which group they feel they belong. It is a measure of cultural affiliation as opposed to race, nationality, ancestry, or citizenship (Statistics New Zealand, 2005). Five classifications are used to describe ethnicity in New Zealand: Māori (the indigenous people of New Zealand); Pasifika (people who identify themselves culturally with the Pacific Islands); Asian (people who identify with west or east Asian groups); Other ethnic groups (people who identify as having a Middle Eastern, African, or South American background); and Pākehā/European (the largest grouping in New Zealand, which includes people mainly with British Isles or European heritage).
The international reporting on PIRLS 2006 showed that New Zealand's Grade 4 students' average performance in reading literacy was 532, with New Zealand students well represented among those with the strongest comprehensions skills internationally. Although New Zealand girls and boys achieved on average above their respective international means, New Zealand again recorded one of the largest gender differences in achievement, favouring girls (Mullis, Martin, Kennedy, & Foy, 2007). National-level analyses highlight the variation among New Zealand's ethnic groupings, with Pākehā/European (552) and Asian (550) students scoring on average at a much higher level than its indigenous Māori (483) and Pasifika (479) students. In terms of gender, the highest performers, on average, were Asian and Pākehā/European girls, while the weakest performers, on average, were Māori and Pasifika boys. Achievement differences favouring girls were observed between girls and boys in all but one of New Zealand's ethnic groupings; the one exception was the lack of any significant difference between the average achievement of Pasifika girls and boys (Chamberlain, in press).1
Other Research
Closely associated with the notion of reading enjoyment is engagement with the materials students read. "Students are influenced and shaped by the quality and style of curriculum delivery, the choice of content and the suitability of resources" (Crooks & Flockton, 2005, p. 58). Reading interesting books was one of three key factors in a case study undertaken by Cullen (2006), in which she investigated ways of increasing the reading mileage of eight New Zealand Grade 6 students who were identified as weak readers. Along with scaffolding techniques, which underpinned the success of her programme, the students were given incentives to increase their mileage. Cullen herself also chose novels for them to read which had initially been deemed difficult (as measured by readability formulae) for this group. When asked to reflect on their own experiences in the programme, the common theme to come from the students was that the texts chosen by Cullen kept them engaged because they had interesting storylines.
The Children's Reading Choices research undertaken by Coles and Hall (2002) confirmed that although reading patterns and practices were predominantly highly gendered, there were misconceptions about boys' choices of reading materials, with both girls and boys choosing fiction in preference to non-fiction material. The types of fiction books boys read included science fiction and fantasy, spy and war, and humorous fiction, while girls read comparatively more adventure, horror, and animal stories. Clark and Osborne (2007) have also looked at text types that differentiate between students' self-evaluation of either being readers or non-readers. Fiction texts differentiated the two groups, with more than 60% of readers saying they read this genre out of school compared with just 11% of non-readers. Their findings showed that a high percentage of those students who were non-readers did actually read, but their source of reading material included magazines, websites, and emails.
PIRLS in New Zealand and elsewhere is regarded as a "low stakes" assessment, in that there are no implications for individuals or individual schools. At the time of the assessment children are encouraged to do their best, and so in some respects the success of the study relies on the motivation of the participants. Eklöf (2007) has shown that Swedish students were motivated to do their best in another low-stakes international assessment, the Trends in International Mathematics and Science Study (TIMSS) 2003. However, a fundamental difference between the two international assessments is the nature of the content being assessed: reading versus mathematics and science. According to Caygill, Sturrock, and Chamberlain (2007), New Zealand Grade 4 boys are more likely than girls to enjoy learning mathematics and science. In reading it is girls more than boys who enjoy it (Chamberlain, in press). The question then arises: if the reading material is not engaging or is uninteresting to boys, how does this affect their motivation to do their best?
Study Objectives
The purpose of this research was to give an overview of New Zealand students' views of the reading passages used in the Progress in International Reading Literacy Study (PIRLS) 2006, and to examine the relationship between their views of the passages and their reading literacy achievement. In our reporting we consider their gender and their ethnicity.2
Methodology
Background
The assessment framework for PIRLS 2006 was developed and described in detail by Mullis, Kennedy, Martin, and Sainsbury (2006). In brief, the study was developed around three aspects: two purposes of reading (reading for literary experience and reading for informational purposes); the processes of comprehension; and students' attitudes and beliefs about reading. Ten different reading texts - five literary passages and five informational passages - were used to assess the reading purposes and the processes of comprehension. The passages averaged 760 words in length, with a range of 495 to 872 words (Mullis, et al., 2007).
The five literary passages described here were complete short stories or episodes, which were accompanied by supportive illustrations.3 Each story had two main characters and a plot with one or two central events.
- "A Little Lump of Clay" (Lump of Clay) is a story about a little lump of clay that sees itself as worthless until a girl discovers it and turns it into a cup.
- "An Unbelievable Night" (Unbelievable Night) is a fantasy story about a young girl who discovers one night that a crocodile appears to have escaped from her magazine.
- "Flowers on the Roof" (Flowers on Roof) describes how, after leaving her farm to live in the city, an elderly lady makes changes to her new home to make it like her old.
- "Fly Eagle Fly" (Fly Eagle Fly) a farmer raises an eagle as a chicken until the farmer's friend encourages him to release it back into the wild.
- "Shiny Straw" (Shiny Straw) a young wolf, ignoring her mother's warnings, gets caught by some hunters and only escapes after her brother sets her free, only to be caught himself.
The five informational texts described here covered a variety of content, including scientific, biographical, and procedural material. As well as prose, each text included organisational and presentational features such as diagrams, photographs, and text boxes.
- "Introducing Antarctica" (Antarctica) includes facts and figures about Antarctica as well as a letter from a scientist who was working at Scott Base.
- "Day Hiking" (Day Hiking) is a brochure which sets out a check list of what to take on a day hike, a list of safety precautions, and a map showing some hiking routes.
- "Leonardo da Vinci a Man Ahead of His Time" (Leonardo) includes details about his artistic works and inventions.
- "Searching for Food" (Searching for Food) describes three science projects on ants, pill-bugs,4 and worms, which look at the features that assist them to hunt for food.
- "Sharks" (Sharks) is an article about the different types of sharks.
Instrumentation
Students were assigned one of 13 booklets, each with two passages: either one literary text and one informational text; two literary texts; or two informational texts. With the exception of two passages - Unbelievable Night and Searching for Food - each passage appeared in three different booklets and in different positions each time (either first or second position). The aforementioned passages were in the Reader and appeared in the order written above. The PIRLS 2006 technical report, edited by Martin, Mullis, and Kennedy (2007), gives a full account of the development of and procedures used in PIRLS 2006. Student ethnicity data were collected directly from the schools and corroborated by students' self-reports via a national question included in the New Zealand Student Questionnaire.
Enjoyment Survey
At the completion of the PIRLS cognitive assessment, New Zealand Grade 4 students were asked to complete a short questionnaire, referred to as the Enjoyment Survey. Each student was asked, in relation to the two texts they had read and assessed: "Did you like[name of passage]?" Students were asked to tick the box that applied, either Yes or No. In an open question they were then asked to explain why they liked or disliked the text.
Participants
The New Zealand achieved student sample in PIRLS 2006 was 6,256, from a representative sample of 243 schools. Approximately 97% of this achieved student sample completed the Enjoyment Survey. 1 summarises the number of completed responses to the Enjoyment Survey from the participants in PIRLS.
Notes:
| ||||
New Zealand Sub-group | New Zealand Students in PIRLS (N) | Participants Completing Enjoyment Survey | ||
---|---|---|---|---|
(N)a | (%) | |||
Test Language | English | 5,990 | 5,834 | 97 |
Māorib | 266 | 222 | 83 | |
Student Gender | Girls | 3,051 | 2,979 | 98 |
Boys | 3,204 | 3,076 | 96 | |
Missing | 1 | 1 | 100 | |
Student Ethnicity | Pākehā/European | 3,627 | 3,559 | 98 |
Māori | 1,514 | 1,418 | 94 | |
Pasifika | 465 | 456 | 98 | |
Asian | 520 | 504 | 97 | |
Other | 125 | 118 | 94 | |
Missing | 5 | 1 | 25 | |
Total Number of Respondents in PIRLS | 6,256 | 6,056 | 97 |
Qualitative Analysis
A non-hierarchical coding schedule for analysing the students' reasons for either liking or disliking passages was developed during the PIRLS field trial, and was, with some revisions, used for coding students' responses to the Enjoyment Survey administered in the PIRLS main run. The same codes were used for indicating liking or disliking the passage. Seven codes, described in , were used to categorise students' responses.
Code | Description | Examples | ||
---|---|---|---|---|
Note:
| ||||
1 |
Subject Matter /Character: | Students referred to the subject matter or to a character. |
"I like crocodiles" "Granny Gunn is cool" "Because I'm not a farm person" | |
2 | Story Construction: | Reference to aspects of the style of writing/message/moral/sub-plot. |
"The story kept you guessing" "It had good adjectives" "I didn't find that it had very good plot" | |
3 | General Engagement | Use of adjectives to describe their opinions/impressions of a passage. |
"It was unusual" "It was exciting" "It was boring" "It was childish" | |
4 | Level of Difficulty: | Reference to the passages (and/or questions) being easy to read or too short, or, more commonly, difficult to read or too long. |
"It was too easy" "It was too long" "It was too hard to read" | |
5 | Educational: | Reference to the passage was educational, or the student said they learnt something (specific) from reading the passage. |
"I learnt new things" "I never knew Leonard[o] invented the plane" "It had too many facts" | |
6 | Genre | Student referred specifically to the type of passage. |
"I like action stories" "I like stories with a message" "I don't do fiction" | |
7 | Other | Students generally made reference to the format of the text, specifically the brochure format, or the illustrations |
"There were neat illustrations" "I didn't like it because I don't usually read panflites [pamphlets]" |
All data, PIRLS booklet information, whether or not participants liked or disliked the passage and the reasons (verbatim), and the codes assigned to the reasons were entered into an Access database.
Quantitative Analysis
Students' responses to the Enjoyment Survey were aggregated across the 13 booklets in order to summarise the information for each passage without the effect of position in a given booklet. In order to achieve this, two dichotomous variables were generated: "completed passage [name]" and "liked [name]" for each of the 10 passages. The "liked [name]" variable was then included in simple cross-tabulations run by gender, ethnicity, gender and ethnicity, and test language. Chi-square tests were used to determine the association between the categorical variables and the "liked [name]." The position effect was also considered by aggregating students' views according to whether or not the passage was in first position or second position in a given booklet. The (coded) reasons given for why passages were liked (or disliked) were also aggregated across the booklets.
The relationship between student achievement and the "liked [name]" variable, by gender, was undertaken for each passage (i.e., Rasch scores were generated).5 Regression techniques were also applied to look at the effect of liking the passage, gender, position of the passage in a booklet, and the language of the test.6
Data Source and Analysis Tools
The Student Background file (ASGNZLR2) was used to source the gender (ITSEX) and test language (ITLANG) variables, and the weight variable (HOUWGT). A national variable file, NZY5EJS_SCR, containing all raw numeric data pertaining to the Enjoyment Survey, the derived "liked [name]" variable, the ethnicity variable (SCHETHNIC), and the passage Rasch scores, was also used during the analytical work. Analyses were carried out using commercial software (SAS) using IEA custom-written SAS macros, which take into account the complex sample design, sourced from the PIRLS International Database (Foy & Kennedy, 2008).
Findings and Discussion
This discussion section is divided into three parts. Firstly, a summary of whether or not students liked individual passages and the reasons why is presented. The focus of the discussion is generally on the differences and/or similarities between girls and boys. When there was clearly a difference among the ethnic groupings, or an ethnicity and gender interaction, this has also been noted. Since we were not attempting to make inferences about the New Zealand Grade 4 population, no measures of uncertainty are reported for the calculated percentages of students who endorsed the passages; the results reflect only the views of the 6,300 students involved in the study. Secondly, an overview of the effect of passage location is discussed. Finally, students' opinions of the passages are examined in the context of their achievement.
In brief, just under two-thirds of New Zealand's Grade 4 students reported they liked both the passages (62%) on which they were assessed. Girls, in particular Asian girls, were most likely to favourably endorse both their passages (72%); Pasifika boys were less so (53%). Proportionately more students assessed in English liked both passages (63%) than students assessed in Māori (53%). Very few students reported they did not like either passage (2%).
Literary Passages
Table 3 shows the percentage of Grade 4 students who reported that they liked a particular literary passage. The literary passage that received the highest level of endorsement from New Zealand Grade 4 students was Unbelievable Night (86%),followed by Flowers on the Roof (83%), with girls in both cases more likely to give their approval to the passage than boys. It was also apparent that for two other passages - Lump of Clay and Fly Eagle Fly - girls were more likely than boys to endorse them. The one exception was Shiny Straw, which attracted a slightly higher proportion of boys liking the passage (82%) than girls (80%). <
Note:
| |||||
New Zealand Student Group | Students Who Liked the Passage (%) | ||||
---|---|---|---|---|---|
Lump of Clay | Flowers on the Roof | Fly Eagle Fly | Shiny Straw | Unbelievable Night | |
Girls | 83 | 89 | 85 | 80 | 93 |
Boys | 69 | 77 | 75 | 82 | 80 |
All NZ | 76 | 83 | 80 | 81 | 86 |
With two exceptions, students' views were similar regardless of their ethnicity. The exceptions were Fly Eagle Fly, which was liked by proportionately more Asian students (90%) (and the same percentage of boys and girls) than students from any other ethnic grouping; and Shiny Straw, which was more likely to be endorsed by Māori students (82%) (and the same percentage of boys and girls) than students in the other groupings.
An overview of students' reasons why they liked (or did not like) a literary passage is presented in Table 4. The reasons New Zealand students gave for liking the literary passages generally fell under the category of general engagement (i.e., the use of a particular adjective to endorse a passage). Examples of the reasons students gave for endorsing Unbelievable Night included "it was cool," "it was imaginative," "because it was fascinating," and "I liked the story because it was very frakey [freaky]." Students liked the humour of Flowers on the Roof, with "because it was funny" and "because it was interesting and grabed my attetion [grabbed my attention]" typical of responses under the general engagement category.
Note:
| ||||||||||
Category of Reason | Percentage of Grade 4 Students Reporting Yes or No, and the Reason | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Lump of Clay | Flowers on the Roof | Fly Eagle Fly | Shiny Straw | Unbelievable Night | ||||||
Yes | No | Yes | No | Yes | No | Yes | No | Yes | No | |
Subject/Character | 13 | 7 | 16 | 12 | 20 | 9 | 26 | 7 | 14 | 5 |
Story Construction | 35 | 7 | 22 | 8 | 28 | 10 | 20 | 14 | 16 | 5 |
General Engagement | 41 | 55 | 50 | 53 | 39 | 45 | 44 | 36 | 56 | 52 |
Difficulty | 5 | 22 | 6 | 16 | 5 | 25 | 3 | 36 | 4 | 20 |
Educational | 2 | 1 | 1 | 1 | 2 | 2 | 1 | < 1 | 1 | 4 |
Genre | 1 | 4 | 1 | 3 | 2 | 1 | 4 | < 1 | 6 | 5 |
Other | 3 | 5 | 3 | 8 | 3 | 7 | 2 | 6 | 4 | 9 |
Story construction, specifically the style of writing, was a common category of reason for liking Fly Eagle Fly. Examples of some reasons under this category include "I liked it because I liked the describing words" and "because it had lots of beautiful words like majestically".
Shiny Straw was the only literary passage liked by proportionately more boys than girls. As well as general engagement, students − and in particular boys − liked the subject matter/character. "Because it was about wolves and my favourite animal is a wolf," "because blue wolf reminds me of my koros [grandfather's] dog," and "I liked it because the carcthers wrer [characters were] cool" are typical of the types of responses under this category. A preference for one of the two main characters (e.g., "cause I liked Blue Wolf") was also mentioned.
Reasons for liking Lump of Clay also fell under the story construction category, with many students making reference to the feelings it evoked or the effective vocabulary; for example, "I liked it because it tells you how a clay might feel," "I like the story because it gives you enough detail so you can make out the story in your mind," and "I liked it because it had lots of exprestions [expressions]."
Little Lump of Clay was also the literary passage with the lowest level of endorsement among New Zealand students, with nearly one-third of boys (31%) and a little under one-fifth of girls (17%) not liking the passage. This pattern held, with the exception of Pasifika students, across the ethnic groupings. The reasons presented for not liking this passage, as with the other passages, fell under the general engagement category, with "boring" the most commonly used adjective in their reasons; for example, "because it was boring and it didn't rely [really] click me," "because it was boring I mean who wants to listen to a boring bit of clay," and "because it was boring story with not much excitement in it."
Informational Passages
shows the percentages of New Zealand Grade 4 students who endorsed each of the five informational passages. At 85%, Antarctica received the highest level of approval by New Zealand Grade 4 students, and was approved by girls (88%) more than by boys (81%). The highest levels of endorsement came from Pākehā/European and Māori girls (88% and 90%). Sharks (82%) was also viewed positively, more so among boys (87%) than girls (77%). This observation held across the ethnic groupings. Interestingly, Leonardo attracted the highest level of endorsement from almost all Pasifika girls (98%) and boys (86%). Pasifika girls also really enjoyed Day Hiking (93%), as did Asian girls (92%).
Note:
| |||||
NZ Student Group | Grade 4 Students Who Liked the Passage (%) | ||||
---|---|---|---|---|---|
Antarctica | Day Hiking | Leonardo | Searching for Food | Sharks | |
Girls | 88 | 84 | 78 | 72 | 77 |
Boys | 81 | 75 | 81 | 75 | 87 |
All NZ | 85 | 79 | 79 | 74 | 82 |
A summary of the reasons students gave for liking the informational passages is presented in . The educational aspect of Antarctica was the reason most students liked this passage. Examples of students' responses to Antarctica included: "I liked it because I learnt something new," and "because I didn't know some really amazing facts." Students' reasons for liking the Sharks passage also fell under the educational category: "Because it had information in it that I did not know," "I like gaining knowledge," and "now I can learn more about sharks teeth and what they want to eat" were typical of the type of responses students offered. The subject matter was the other reason for liking Sharks: "because I like sharks," "because sharks have really good abaitys [abilities]," and "because sharks are fast strong and fiarec [fierce] and also my favourite sea animal" illustrate the types of responses in this category.
Note:
| ||||||||||
Category of Reason | Percentage of Grade 4 Students Reporting Yes or No, and the Reason | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
Antarctica | Day Hiking | Leonardo | Searching for food | Sharks | ||||||
Yes | No | Yes | No | Yes | No | Yes | No | Yes | No | |
Subject/Character | 14 | 19 | 32 | 17 | 33 | 8 | 16 | 26 | 26 | 43 |
Story Construction | 3 | 2 | 2 | 1 | 3 | 2 | 2 | 1 | 1 | 1 |
General Engagement | 21 | 37 | 25 | 42 | 26 | 29 | 28 | 36 | 15 | 19 |
Difficulty | 3 | 18 | 8 | 26 | 4 | 45 | 2 | 19 | 4 | 27 |
Educational | 54 | 12 | 25 | 7 | 30 | 7 | 37 | 8 | 51 | 7 |
Genre | 2 | 7 | 1 | 3 | 3 | 5 | 13 | 6 | 1 | 2 |
Other | 3 | 6 | 6 | 4 | 1 | 3 | 2 | 5 | 3 | 1 |
The subject matter of Day Hiking (e.g., "because tramping is fun," "because I like tramping and seeing animals")7 and Leonardo (e.g., "because I like inventions and I'm a big fan of da Vinci," "because I like envetions [inventions] and painting,") was also offered as the reason why these two passages appealed to New Zealand students.
Searching for Food was the informational passage that received the lowest level of endorsement from New Zealand students, regardless of gender and ethnicity. Students' reasons for liking this passage came under the general engagement category, with the adjective "interesting" typically used in their explanation (e.g., "because it was interesting," "because it was fun and it was interesting.") The reasons cited by students for why they did not like the passage fell under either the general engagement category (e.g., "it was boring," "because it was creepy and not that interesting") or the subject matter category (e.g., "I do not like creepy croules [creepy crawlies]," "I like science but I hate bugs," "because I'm not to big on worms and ants.")
Passage Position8
As noted in the Instrumentation section, each student was assigned one of 13 booklets, each containing two reading passages - passage 1 and passage 2. Analyses were run separately for passage 1 and passage 2 to determine whether or not the order of the passage (position 1 or position 2) had an effect on students' views. Table 7 shows the percentages of Grade 4 students who reported they liked each passage, arranged in descending order, for passage 1 and passage 2.
Position in the Test Booklet | |||||
---|---|---|---|---|---|
First Position | Students Liked Passage (%) | Second Position | Students Liked Passage (%) | ||
Notes:
| |||||
1. Antarctica | 88 (2) | 1. Flowers on the Roof | 84 (8) | ||
2. Unbelievable Night* | 86 (-) | 2. Antarctica | 83 (1) | ||
3. Sharks | 84 (5) | 3. Fly Eagle Fly | 79 (5) | ||
4. Shiny Straw | 83 (6) | 4. Day Hiking | 78 (7) | ||
5. Fly Eagle Fly | 83 (3) | 5. Sharks | 77 (3) | ||
6. Leonardo | 83 (9) | 6. Shiny Straw | 77 (4) | ||
7. Day Hiking | 81 (4) | 7. Lump of Clay | 75 (9) | ||
8. Flowers on the Roof | 81 (1) | 8. Searching for Food† | 74 (-) | ||
9. Lump of Clay | 77 (7) | 9. Leonardo | 73 (6) |
Antarctica was the most liked passage in position 1 and second most liked in position 2 while Lump of Clay was the least liked in position 1 and third least liked in position 2, just ahead of Searching for Food and Leonardo. Interestingly, Flowers on the Roof was the only passage with a higher percentage of liking it in position 2 of the booklet compared with position 1; all the others had decreased ratings in the second position. This may be a reflection of the other passages with which Flowers on the Roof was paired. For example, in booklet 1 Flowers on the Roof in position 2 was paired with Little Lump of Clay (position 1), which would suggest students compared the two texts.
Students' Views of the Passages and Their Achievement
In this last section we examine the relationship between New Zealand students' opinions of the passages and their achievement (Rasch scores) on the passages, by gender.9 Figure 1 provides a graphical representation of the average differences in achievement between students who liked the passages and those who did not like them, by gender. On average, girls who liked the literary passage on which they were assessed achieved higher scores than girls who did not like the passage. However, the differences between their mean scores were, with one exception, small and not statistically significant (at the 5% level). The exception was Shiny Straw, where the average difference of 3.6 score points was statistically significant (t = 2.93). The achievement pattern for boys was similar to that of girls, with boys who liked the literary passage achieving at a higher level, on average, than boys who reported they did not like it. However, only the average differences observed for Little Lump of Clay (2.6 points; t = 2.75) and Shiny Straw (3.2 points; t = 2.50) were statistically significant.
Figure 1: Mean achievement differences on the literary and informational passages, by gender
Note:
- A data point with a value close to zero indicates that the difference between students who liked the passage and students who did not is small. The vertical lines extending from a data point show the 95% confidence interval around the difference of the mean (i.e., ± 2 standard errors of the difference).
The figure also illustrates that achievement differences between students (both girls and boys) who either liked or disliked the informational passages were more evident than was the case for the literary passages. Significant differences between the mean scores of girls who liked and girls who did not like three of the five informational passages were detected: Sharks (2.3 points; t = 2.71), Leonardo (2.7 points; t = 2.67), and Antarctica (3.0 points; t = 2.45). Boys who liked Day Hiking (2.3 points; t = 2.43) and Searching for Food (3.3 points; t = 2.90) achieved significantly higher scores on average than boys who did not like these two passages. However, it was the 6.1 (average) point difference between the boys who liked Leonardo and the boys who did not which was so marked (t = 4.74).
In order to confirm the relationships shown between achievement, gender and liking the passage, two multiple regression models were built. The regressions were run separately for the literary and informational passages, using the Rasch passage score as the outcome variable. The independent variables were various combinations of gender, liking the passage, position variables, and the language of the test. For the resulting "literary" model, significant b coefficients (p < 0.05) were found for the variables: [assessed in Māori], [boys], [liked the passage], [boys*position 2], and [boys*position 2*liked passage]. As well as boys and students assessed in Māori tending to have lower scores, the model demonstrated that students who liked the passage were likely to have higher scores. Achievement on the second passage was likely to be lower for boys than for girls, although this effect was negated if boys liked the second passage. A slightly different pattern was observed for the "informational" model: [assessed in Māori], [boys], [liked the passage], [Leonardo*liked*boys] were the only variables with significant b coefficients. Boys and students assessed in Māori tended to have lower scores on the informational passages, while liking had a positive effect on achievement particularly for those boys who liked the passage, and especially, if they liked Leonardo.
In keeping with our approach at looking at the individual passages, simple models were built for each passage. The outcome variable in each case was the Rasch passage score. The independent variables were: [liked passage], [assessed in Māori], [boy liked passage], [position 2], and [<ethnic boy>] where <ethnic boy> took the value of "boy" in each ethnic grouping. The standardised coefficients for these regressions are reported in .
Notes:
| ||||||||||
Passage | Literary Passages | Informational Passages | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
L of C | F on R | F E F | S S | U Na | A | D H | L | S | S f Fb | |
N | 1218 | 1241 | 1205 | 1211 | 1165 | 1193 | 1197 | 1213 | 1192 | 1149 |
β1[Liked Passage] |
2.1 (1.0)* |
3.5 (1.2)** |
3.3 (1.1)** |
2.9 (1.0)** |
2.4 (0.8)** | |||||
β2[Assess in Māori] |
-14.9 (1.5)** |
-12.1 (1.2)** |
-9.9 (1.3)** |
-12.9 (1.3)** |
-15.2 (2.5)** |
-13.0 (1.8)** |
-10.9 (1.2)** |
-10.5 (1.4)** |
-11.7 (1.4)** |
-10.4 (2.7)** |
β3[Boy Liked Passage] |
3.6 (1.6)* | |||||||||
β4[Position 2] |
-1.1 (0.5)* | N.A. |
-1.3 (0.7)* |
1.5 (0.7)* | N.A. | |||||
β5[Pākehā Boy] |
-2.9 (1.4)* | |||||||||
β6[Māori Boy] | -7.5 (1.3)** | -7.7 (1.6)** | -4.6 (1.5)** | -5.5 (1.5)** | -6.5 (1.9)** | -4.5 (1.5)** | -7.1 (1.5)** | -8.9 (1.5)** | -3.9 (1.7)* | -7.8 (1.4)** |
β7[Pasifika Boy] | -11.3 (2.0)** | -8.7 (2.0)** | -6.6 (1.8)** | -6.6 (2.0)** | -5.6 (1.9)** | -5.3 (1.9)** | -9.8 (2.0)** | -10.2 (2.4)** | -5.7 (2.0)** | -5.1 (2.1)* |
β8[Asian Boy] | -3.2 (1.6)* | -4.9 (1.9)** | ||||||||
R2 | 0.127 | 0.128 | 0.09 | 0.108 | 0.087 | 0.108 | 0.109 | 0.121 | 0.074 | 0.081 |
The significant (negative) b coefficients common to all passage models were the variables [assessed in Māori], [being a Māori boy], and [being a Pasifika boy]. In addition, the b coefficient for the [liked passage] was significant (and positive) in the models for Fly Eagle Fly, Shiny Straw, Antarctica, Leonardo, and Sharks. Interestingly, significant (negative) passage position effects on student achievement were observed for Lump of Clay and Antarctica, and consistent with the finding from the "informational model" described previously, there were significant positive effects of passage position and being a boy who liked Leonardo.
Conclusion and Implications
New Zealand students were on the whole very positive about the PIRLS 2006 reading passages, with almost all students reporting they liked at least one of the passages. The fantasy passage Unbelievable Night and the informational passage Antarctica[1]received the most endorsement, while Little Lump of Clay and Searching for Food were the least endorsed. The reasons most often given by students for liking the literary passages generally fell into the overarching category of general engagement. Students' reasons for liking the information passages mainly fell into three categories: educational, general engagement, and subject matter. Not surprisingly, there were differences between girls' and boys' views. Unbelievable Night was found to be the most popular among New Zealand girls, while Sharks appeared to be the most popular among boys. For some passages, there were small effects of liking the passage on student achievement. Furthermore, there did appear to be some position effect on achievement for some passages, and more so for boys. It is unlikely however that these findings would explain entirely New Zealand's relatively large gender difference shown in PIRLS.
Implications
There are two main implications. Firstly, the findings reinforce the passage rotation practice already adhered to by the international research team. However, during the development phase for the next cycle of PIRLS there are opportunities (for example, in the field trial) to use the findings from this research to determine if there is any potential bias or lack of engagement with particular passages, and if so, how that could be alleviated across booklets, and particularly the Reader. Particular attention could also be paid to the positioning of any engagement question, with the placement at the end of the reading passage (and before the set questions relating to the passage), rather than at the end of the assessment.
Secondly, there are interesting results here for New Zealand classroom teachers in terms of identifying types of materials that will engage our reluctant readers or non-readers (10see footnote 10). If reading for enjoyment is one of the key levers for improving the skills of weaker readers, which in New Zealand's case are often boys, then having evidence of what engages these students is essential. As was noted by the New Zealand Ministry of Education (2003, p. 12), "Effective practice involves using and creating rich texts. These relate to students' interests, draw on and affirm their social and cultural identities, use authentic language, and motivate and challenge them as learners."
References
- Alton-Lee, A., & Praat, A. (2000). Explaining and addressing gender differences in the New Zealand compulsory school sector. Wellington: Research Division, Ministry of Education.
- Caygill, R., Sturrock, F., & Chamberlain, M. (2007). Mathematics and science achievement in New Zealand: Tracking the changes of year 5 students in TIMSS 1994–2000. Wellington: Research Division, Ministry of Education
- Chamberlain, M. (in press). PIRLS 2005/2006 in New Zealand: An overview of national findings from the second cycle of the Progress in International Reading Literacy Study (PIRLS). Wellington: Research Division, Ministry of Education.
- Clark, C., & Osborne, S. (2007). Demystifying the reluctant reader. Literacy Today, 53, December, pp.22–23.
- Coles, M., & Hall, C. (2002). Gendered readings: Learning from children's reading choices. Journal of Research in Reading, 25(1), 96–108.
- Crooks, T., & Flockton, L. (2005). National Education Monitoring Project: Reading and speaking assessment results 2004. Dunedin: Educational Assessment Research Unit, Otago University.
- Cullen, T. (2006). Thinking outside the square to enable reluctant readers to undertake high reading mileage. Reading Forum New Zealand, 21(2), 24−33.
- Eklöf, H. (2007). Test-taking motivation on low-stakes tests: A Swedish TIMSS 2003 example. The Second IEA International Research Conference: Proceedings of the IRC-2006. (Vol. 1 pp. 135–143). Amsterdam: International Association for the Evaluation of Educational Achievement.
- Foy, P., & Kennedy, A. M. (Eds.). (2008). PIRLS 2006 user guide for the international database. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
- Martin, M. O., Mullis, I.V. S,, & Kennedy, A. M. (2007) PIRLS 2006 technical report. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
- Ministry of Education. (2003). Effective literacy practice years 1 to 4. Wellington: Learning Media.
- Ministry of Education. (2007). Boys' achievement: A synthesis of the data. Wellington: Learning Policy Frameworks, Ministry of Education.
- Mullis, I. V. S., Kennedy, A. M., Martin, M. O., & Sainsbury, M. (2006). PIRLS 2006 assessment framework and specifications (2nd ed.). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
- Mullis, I. V. S., Martin, M. O., Kennedy, A. M., & Foy, P. (2007). PIRLS 2006 international report: IEA's Progress in International Reading Literacy Study in primary schools in 40 countries. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
- Satherley, P. (2006). Student outcome overview 2001–2005. Wellington: Research Division, Ministry of Education.
- Statistics New Zealand. (2005). Statistical standard for ethnicity 2005.
- Wagemaker, H. (Ed.). (1993). Achievement in reading literacy: New Zealand's performance in a national and international context. Wellington: Research Section, Ministry of Education.
- Mullis, I. V. S., Martin, M. O., Gonzalez, E. J., & Kennedy, A. M. (2003). PIRLS 2001 international report: IEA's study of reading literacy achievement in primary schools in 35 countries. Chestnut Hill, MA: International Study Center, Boston College.
Footnotes
- New Zealand children were also found to be relatively positive about reading. However, about one in five students never read for fun in their own time. Boys, particularly Māori and Pasifika boys, were over-represented in this group of non-readers.
- In keeping with our original proposal, we also investigated the passages which engaged the group of New Zealand students who found reading boring and who had reported they read for fun. Because of reasons of brevity we have not reported this part of our work.
- An abbreviated descriptor appears in italics after the full title.
- In New Zealand, "pill-bugs" are known colloquially as "slaters", and so this term was used in this passage.
- Note that the Rasch scores are based on a relatively small number of items. The authors would like to acknowledge Oliver Neuschmidt from the IEA Data Processing Center, Hamburg, for generating these scores.
- We do not, however, report students' views by the language (English or Māori) of the assessment in this paper.
- "Tramping" is the word commonly used in New Zealand for hiking or bush-walking.
- The authors wish to acknowledge Dr Ian Schagen (Chief Research Analyst, Ministry of Education, New Zealand, formerly of the National Foundation for Educational Research, England), who undertook this piece of analysis as part of a more extensive analysis of passage position.
- For the most part our analyses showed more differences between girls' and boys' views overall and within ethnic groupings than between ethnic groupings. We therefore, focus on just gender differences in this section.
- An examination of the children's views of the passages in relation to those who reported that they never read for fun showed that these students also tended to hold less favorable views about the passages. Flowers on the Roof, Unbelievable Night, and Antarctica featured as popular passages among New Zealand's non-readers.
Navigation
Contact Us
PIRLS
For queries about the Progress in International Reading Literacy Study (PIRLS) please email the: PIRLS Mailbox