Summary

Background: the PISA survey

PISA surveys the knowledge and skills of 15-year-olds in the principal industrialised countries. The product of collaboration between participating governments through the OECD, it draws on leading international expertise to develop valid comparisons across countries and cultures.

PISA 2003 is the second assessment in the series. In this survey:

• Well over a quarter of a million students in 41 countries took part. All 30 OECD member countries participated, as well as ‘partner countries’ in Asia, Africa, Eastern Europe, and Latin America.
• Each student took a two-hour written test in their school.
• Students were assessed in mathematics, reading, science, and problem-solving. Mathematics was the main focus in 2003, while reading was the main focus in the first survey in 2000. The next PISA assessment in 2006 will focus on student performance in science.

The key features of the PISA approach are:

• Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons.
• The innovative ‘literacy’ concept, which is concerned with the capacity of students to apply their knowledge and skills in key subject areas and to analyse, reason, and communicate effectively as they pose, solve, and interpret problems in a variety of situations.
• Its inclusion of assessment that is not restricted to particular areas of the school curriculum. The assessment of ‘problem solving’ in 2003 was the first such ‘cross-curricular’ assessment.
• Its regularity, which will enable countries to monitor their progress in meeting the key learning objectives over time.
• Its consideration of student performance alongside the background characteristics of students’ at home and school in order to explore some of the main features associated with educational success. Each participating student and school completed a questionnaire that allowed a wide range of background information to be considered alongside student performance.
• Its breadth of geographical coverage, as the countries that have participated so far represent one-third of the world’s population and almost nine-tenths of the world’s gross domestic product (GDP).

The comparisons made below set New Zealand students’ performance alongside that of students in the OECD countries, which are the countries in the survey most comparable to New Zealand: it excludes countries such as Tunisia, Peru, and Russia from the analysis. A total of 29 OECD countries reported results in 2003. (The United Kingdom participated but did not meet the sampling requirements.)

Assessing scientific literacy in PISA 2003

Students in PISA are asked to apply science knowledge and skills to real-life situations. According to the agreed PISA definition, science literacy is:

…the capacity to use scientific knowledge, to identify questions and to draw evidence-based conclusions in order to understand and help make decisions about the natural world and the changes made to it through human activity.

PISA 2003 Learning for Tomorrow’s World, OECD: p. 24

Most of the assessment items from PISA 2000 were also used in PISA 2003, allowing direct comparison between the two cycles.

The questions set in PISA varied in format: some were multiple choice; others required written answers. Of the latter, some required one correct answer and others were more ‘open’ and could be answered correctly in a variety of ways. In some cases, partial credit could be given for responses that showed some, but not all, of the required degree of understanding.

Science scores in PISA 2003

Performance in science is marked on a single scale with an average score of 500 score points and a standard deviation of 100 score points. Approximately two-thirds of students across the OECD countries score between 400 and 600 points. This report uses mean scores to compare the science performance of New Zealand students with those in other OECD countries.

Because science has been a minor domain, with less testing time in both of the 2000 and 2003 cycles of PISA, the scale cannot be defined as yet in terms of proficiency levels, which help to describe what kinds of skills are needed to reach a particular score. In the third PISA cycle in 2006, science will be the major domain of testing and a full instrument for measuring and reporting science will enable proficiency levels to be defined. However, criteria for harder and easier tasks can still be described in relation to items associated with different points on the science scale. Examples of the kinds of tasks involved at different levels are provided in Boxes A and B later in this report.