PISA 2009: Digital readers at age 15: Results from the PISA 2009 Electronic Reading Assessment

Publication Details

The Electronic Reading Assessment is a new and innovative component of PISA that acknowledges the increasing relevance of electronic text and recognises its importance as a feature of reading. Those students who took part in the Electronic Reading Assessment (ERA) were given a 10 minute practice session and then asked to complete a 40 minute computer based assessment. The texts used in this assessment were hypertexts. Hypertexts are where the user has navigation tools and features that allow them to move through pages of text freely in numerous ways.

Author(s): Sarah Kirkham

Date Published: June 2011

What is the PISA 2009 Electronic Reading Assessment (ERA)?

The Programme for International Student Assessment (PISA) 1 is an international study that assesses how well countries are preparing their 15-year-old students to meet real-life opportunities and challenges. PISA assesses three key areas — reading, mathematical and scientific literacy. Students participating in PISA to date have all responded to print (paper) based assessments. In 2009, the main PISA study continued to assess in this way but it also offered countries the option of assessing some of the participating students in reading using a computer based assessment — the Electronic Reading Assessment (ERA).


The ERA is a new and innovative component of PISA that acknowledges the increasing relevance of electronic text and recognises its importance as a feature of reading. Those students who took part in the ERA were given a 10 minute practice session and then asked to complete a 40 minute computer based assessment. The texts used in this assessment were hypertexts. Hypertexts are where the user has navigation tools and features that allow them to move through pages of text freely in numerous ways.

Who took part in this study?

In New Zealand, 145 of the randomly selected schools with 15-year-old students took part in PISA 2009. The selection process was run by the international research consortium. While 4,643 selected students in New Zealand took part in the print based assessment, a smaller subset of students randomly selected from each school took the ERA (1,752 students in total).


Altogether 19 countries or economies took part in the ERA component of PISA 2009. The Organisation for Economic Co-operation and Development (OECD) countries that participated in the ERA were Australia, Austria, Belgium, Chile, Denmark, France, Hungary, Iceland, Ireland, Japan, Korea, New Zealand, Norway, Poland, Spain, and Sweden. The partner countries or economies that took part were Colombia, Hong Kong-China, and Macao-China. Around 36,500 15-year-old students from these 19 countries or economies participated in this assessment.

What additional information is gathered?

Background information is also gained in each PISA cycle from questionnaires completed by students and school principals. Of particular relevance to the ERA, many countries collected information on students’ knowledge about and use of information and communication technology (ICT). This covered such things as students’ familiarity with computers, the internet, and various types of software, and tasks that can be carried out electronically. The information gathered from these questionnaires enables the relationship between contextual information and achievement to be examined.

How well did New Zealand 15-year-olds do in the international context?


Figure 1: Distribution of digital reading achievement for ERA countries

pubID-101736-fig1

Notes:  1. Asterisks(*) indicate partner countries/economies.
           2. Standard errors2 are shown in parentheses


The average (mean) digital reading literacy score for New Zealand 15-year-olds was 537, significantly higher than the print reading literacy score (521).

The New Zealand average was:

  • significantly higher than the OECD average of 4993
  • significantly higher than the average score for students in 16 countries or economies
  • the same as the average score for students in Australia
  • significantly lower than the average score for students in Korea


Compared with other high-performing countries or economies participating in the ERA, New Zealand had a relatively large proportion of students who demonstrated very advanced digital reading skills as well as a relatively high proportion of students who demonstrated poor skills.


Figure 2: Distribution of New Zealand 15-year-olds’ digital reading achievement by gender
pubID-101736-fig2

Note: Standard errors are shown in parentheses.


New Zealand 15-year-olds typically achieved above the OECD averages for girls and for boys. However, the difference in performance in favour of girls that was observed in the print based assessment in PISA 2009 was also evident in the computer based assessment. The 40 point difference between New Zealand girls and boys was the largest amongst countries participating in the ERA.

How did sub-groups of New Zealand 15-year-olds do in the ERA?

 

Figure 3: Distribution of New Zealand 15-year-olds’ digital reading achievement by ethnicity
pubID-101736-fig3

Notes:  1. The students that were identified in the ethnic group “Other” have been left out of this table as the small size of the group
                makes it difficult to carry out viable analyses.

            2. Standard errors are shown in parentheses.



Students from all ethnic backgrounds were represented among the high and low achievers. Pākehā/European (556) and Asian (538) students generally scored at a higher level than Māori (502) and Pasifika (472) students.4 Māori students achieved at a similar level to the OECD average for those countries who participated in the ERA option (499).


Overall girls had stronger achievement than boys from the same ethnic background. This was particularly marked for the Māori and Pasifika students with differences of 47 and 52 respectively.

                                   
Table 1: Mean digital reading achievement by ethnicity
GenderFemaleMaleFemale MaleFemaleMaleMaleFemale
Ethnicity Pākehā/EuropeanMāoriPasifikaAsian
Mean574 (2.7) 538 (3.6)527 (5.5)480 (5.8) 499
(7.7)
447 (6.9) 555 (6.4)523 (7.5)
Difference between female and male 36475232


Did students’ achievement vary based on their home background?

Digital reading achievement was significantly higher, on average, among students who regularly spoke English at home. Students who had at least one parent born in New Zealand had significantly higher digital reading achievement, on average, than those whose parents were not born in New Zealand.


New Zealand students from higher socio-economic backgrounds5 tended to have significantly higher mean achievement than those from lower socio-economic backgrounds.

Access to ICT and its use

Students taking part in PISA were asked about their access to and use of computers at home and at school. The percentage of New Zealand students with access to computers at home rose by 17% between 2000 and 2009 (refer Table 2), reflecting a similar rise across the OECD countries that took part in the ERA.


The proportion of students with access to the internet at home saw an even bigger increase between 2000 and 2009. The digital reading achievement for those students with access to the internet at home was significantly higher than those without.

                        
Table 2: Proportions of students with access to computers and internet at home
Computer at home *(%)Internet at home (%)
2000
20092000
2009
New Zealand79 (0.8)96 (0.3)62(1.0) 92(0.5)
OECD Average672 (0.2)94 (0.1) 45(0.2)89 (0.1)
Note: Standard errors are shown in parentheses.



Students were also asked how frequently they participated in certain online activities at home. The most popular of these activities for New Zealand students were browsing the internet for fun (79% reported doing this at least once a week), using email (71%), chatting online (63%), and downloading music, films, games or software from the internet (60%).7 Additionally, 68% reported doing homework on the computer at home more than once a week and 52% reported browsing the internet at home for schoolwork frequently.


Almost all New Zealand students reported in 2009 that they have access to computers and the internet at school (less than two percent reported no access to either at school). The most frequent use for school computers by New Zealand students was searching the internet for schoolwork (50% reported doing this at least once a week).8


In terms of computer based activities, New Zealand students felt most confident about creating a presentation by themselves or with some help from someone and were least confident about creating a database by themselves or with help.9


Despite regular use of computers by a relatively high proportion of students, students in New Zealand on average expressed a less positive attitude towards computers than the OECD average.  Boys in New Zealand expressed more positive attitudes towards computers, a pattern reflected in a number of OECD countries and partner countries and economies.

How does digital reading differ from print?

Digital reading is considered a subset of reading literacy as a whole but it has some distinct and unique features such as non-linear navigation through pages of text. However, the basic processes of reading, such as word identification and ability to recognise and understand grammatical structures, are applicable to both online and print reading.

More detailed information on the digital reading results is available in the OECD PISA 2009 report Students On Line: Digital Technologies and Performance, Volume 6.

                                 
Table 3: List of countries and economies participating in PISA 2009 Electronic Reading Assessment
AustraliaHong Kong-China*Iceland
AustriaHungaryNorway
BelgiumIrelandPoland
ChileJapanSpain
Columbia*KoreaSweden
DenmarkMacao-China*
FranceNew Zealand
Notes:  1. Asterisks(*) indicate partner countries/economies.
           2. Standard errors are shown in parentheses

Footnotes

  1. PISA is an initiative of the OECD and a collaborative effort of the participating countries. The Comparative Education Research Unit within the Ministry of Education’s Research Division is responsible for the administration of PISA in New Zealand.
  2. Standard errors provide a measure of the precision of the estimate; in this case, the mean.
  3. The digital reading assessment scale was constructed to match the PISA 2009 print reading results for the 16 OECD countries that took part.
  4. Students were asked to identify the ethnic groups they belonged to. Those who identified with more than one group were counted in each of those groups.
  5. Using the PISA index of economic, social and cultural status.
  6. Across the 27 countries that administered this question in both 2000 and 2009.
  7. The other activities asked about were: play one-player games; play collaborative online games; publish and maintain a personal website, weblog or blog; and participate in online forums, virtual communities or spaces. Less than 45% of New Zealand students reported doing these frequently.
  8. The other activities were: chat online at school; use email at school; download, upload or browse material from the school’s website; post your work on the school’s website; play simulations at school; practice and drilling, such as for foreign language learning or mathematics; do individual homework on a school computer; use school computers for group work and communication with other students. Less than 30% of New Zealand students reported doing these frequently.
  9. The other activities asked about were editing digital photographs or other graphic images, using a spreadsheet to plot a graph, creating a multi-media presentation (with sound, pictures, video).