Summary

Background: the PISA survey

PISA surveys the knowledge and skills of 15-year-olds in the principal industrialised countries. The product of collaboration between participating governments through the OECD, it draws on leading international expertise to develop valid comparisons across countries and cultures.

PISA 2003 is the second assessment in the series. In this survey:

• Well over a quarter of a million students in 41 countries took part. All 30 OECD member countries participated, as well as ‘partner countries’ in Asia, Africa, Eastern Europe, and Latin America.
• Each student took a two-hour written test in their school.
• Students were assessed in mathematics, reading, science, and problem solving. Mathematics was the main focus in 2003, while reading was the main focus in the 2000 survey. The next PISA assessment in 2006 will focus on student performance in science.

The key features of the PISA approach are :

• Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons.
• The innovative ‘literacy’ concept, which is concerned with the capacity of students to apply knowledge and skills in key subject areas and to analyse, reason, and communicate effectively as they pose, solve, and interpret problems in a variety of situations.
• Its inclusion of assessment that is not restricted to particular areas of the school curriculum. The assessment of ‘problem-solving’ in 2003 was the first such ‘cross-curricular’ assessment.
• Its regularity, which will enable countries to monitor their progress in meeting the key learning objectives over time.
• Its consideration of student performance alongside the characteristics of students’ homes and schools in order to explore some of the main features associated with educational success. Each participating student and school completed a questionnaire that allowed a wide range of background information to be considered alongside student performance.
• Its breadth of geographical coverage, as the countries that have participated so far represent one-third of the world’s population and almost nine-tenths of the world’s gross domestic product (GDP).

The comparisons made below set New Zealand students’ performance alongside that of students in the participating OECD countries, which are the countries in the survey most comparable to New Zealand: it excludes countries such as Tunisia, Peru and Russia from the analysis. A total of 29 OECD countries reported results in 2003. (One country, the United Kingdom, participated but did not meet the sampling requirements.)

Assessing reading literacy in PISA 2003

When looking at 15-year-olds’ reading abilities, PISA adopts the broad concept of ‘reading literacy’. Reading is no longer considered to be simply the ability to read and write, which is acquired in childhood as a single well-defined skill. Today, it is viewed as an advancing set of knowledge, skills, and strategies that individuals develop and build on throughout life, through experience, not just formal education.  According to the agreed PISA definition, reading literacy is:

… understanding, using, and reflecting on written texts in order to achieve one’s goals, to develop one’s knowledge and potential, and to participate in society.

PISA 2003 Learning for Tomorrow’s World, OECD: p. 27

This definition goes beyond the traditional notion of decoding information and literal interpretation of what is written towards more applied tasks.

The first dimension, the text format, refers to the type of written material:

• Continuous prose – such as a book or an article;
• Non-continuous – texts, such as graphs, tables, lists, or diagrams.

The reading tasks set in PISA were based on a number of units, each with several questions following from a text that the students were required to read. The texts were drawn from a variety of forms.

The second dimension, types of reading tasks, represents the things that students need to do as readers:

• Retrieve information – that is, to locate single or multiple pieces of information in a text.
• Interpret texts – that is, to construct meaning and draw inferences from written information.
• Reflect on and evaluate texts – that is, to relate written information to their prior knowledge, ideas, and experiences.

In PISA 2000, student performances in these three types of tasks were reported on a separate scale. In 2003, however, less assessment time was allocated to reading and the results are reported only on a single reading literacy scale that combines the three types of tasks.

The third dimension, the situation or context, reflects the categorisation of texts based on the author’s intended use, the relationship with other persons implicitly or explicitly associated with the text, and the general content. A number of situations were represented in PISA, and they were selected to maximise the diversity of content included in the reading literacy assessment.

• Reading for private use (personal);
• Reading for public use;
• Reading for work (occupational); and
• Reading for education.

The questions set in PISA varied in format: some were multiple choice; others required written answers. Of the latter, some required one correct answer whereas others were more ‘open’ and could be answered correctly in a variety of ways. In some cases, partial credit could be given for responses that showed some, but not all, of the required degree of understanding.

Reading scores and reading proficiency in PISA 2003

Reading questions represented a wide range of difficulty, from extremely simple identification of information to highly complex interpretations drawing on multiple pieces of non-explicit information. Each task was given a score according to its difficulty and the students were assigned individual scores according to the highest difficulty of task that they were likely to be able to perform. The types of task set at varying levels of difficulty are shown in shaded boxes throughout this report.

The scale is devised with an average score for all the OECD countries of 500 and a standard deviation of 100, meaning that about two-thirds of students score between 400 and 600 points.

To aid interpretation, tasks were ranked at five ‘proficiency levels’ and each student was assigned to the highest level at which they could perform the required tasks. Specifically, a student must be capable of getting a majority of tasks at a given proficiency level correct in order to be ranked at that level. Some students do not reach even the lowest level, Level 1, but this does not mean that they are unable to perform any PISA reading tasks. However, it does mean that they would get the majority of simple tasks wrong.

Table 1 shows the meaning of reading proficiency at different levels in PISA. For a more detailed description of each level refer to Appendix A at the end of this report.