Summary

Background

PISA surveys the knowledge and skills of 15-year-olds in the principal industrialised countries. The product of collaboration between participating governments through the OECD, it draws on leading international expertise to develop valid comparisons across countries and cultures.

PISA 2003 is the second assessment in the series. In this survey:

• Well over a quarter of a million students in 41 countries took part. All 30 OECD member countries participated, as well as ’partner countries’ in Asia, Africa, Eastern Europe, and Latin America.
• Each student took a two-hour written test in their school.
• Students were assessed in mathematics, reading, science, and problem-solving. Mathematics was the main focus in 2003, while reading was the main focus in the first survey in 2000. The next PISA assessment in 2006 will focus on student performance in science.

The key features of the PISA approach are:

• Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons.
• The innovative ’literacy’ concept, which is concerned with the capacity of students to apply knowledge and skills in key subject areas and to analyse, reason, and communicate effectively as they pose, solve, and interpret problems in a variety of situations.
• Its inclusion of assessment that is not restricted to particular areas of the school curriculum. The assessment of ’problem-solving’ in 2003, the subject of this report, was the first such ’cross-curricular’ assessment.
• Its regularity, which will enable countries to monitor their progress in meeting the key learning objectives over time.
• Its consideration of student performance alongside the background characteristics of students at home and school in order to explore some of the main features associated with educational success. Each participating student and school completed a questionnaire that allowed a wide range of background information to be considered alongside student performance.
• Its breadth of geographical coverage, as the countries that have participated so far represent one-third of the world’s population and almost nine-tenths of the world’s gross domestic product (GDP).

The comparisons made below set New Zealand students’ performance alongside that of students in the participating OECD countries, which are the countries in the survey most comparable to New Zealand: it excludes countries such as Tunisia, Peru and Russia from the analysis. A total of 29 OECD countries reported results in 2003. (The United Kingdom participated but did not meet the sampling requirements.)

Assessing problem-solving in PISA 2003

The collection of information about problem-solving skills in PISA 2003 was carried out to address concerns from the OECD countries that students’ capabilities in reading, mathematics, and science do not tell us enough about their overall capacity to solve problems in real-life situations beyond the context of the school curriculum.


According to the agreed PISA definition, problem-solving is:

… an individual’s capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious and where the literacy domains or curricular areas that might be applicable are not within a single domain of mathematics, science or reading.
PISA 2003 Learning for Tomorrow’s World, OECD: p. 26

The problem-solving component within PISA 2003 examines how well young adults are prepared to meet the challenges of today’s knowledge societies, focusing on young people’s ability to use their knowledge and skills to meet real-life challenges beyond the classroom.

The concept of problem-solving in PISA is defined by five dimensions:

• Problem types: PISA 2003 assesses three types of problems: decision-making, system analysis and design, and trouble-shooting. For a description of what these three types of problems involve, see Appendix B. Examples of the three different types of problems are given in Boxes A to C of this report:
  • A decision-making problem requiring students to find a suitable time and date to go to the cinema, when shown a timetable and a set of constraints (Box A, pages 9-11).
  • A systems analysis and design problem requiring students to organise dormitory allocations in a children’s camp, given a set of rules and other information (Box B, pages 13-15).
  • A trouble-shooting problem requiring students to diagnose a problem with the flow of water through an irrigation network (Box C, pages 19-21).

• Problem context: problems are set in contexts outside the classroom and school curriculum, such as personal life, work and leisure, and community and society.
• Disciplines : the domain covers a wide range of disciplines and thus complements the main PISA domains of mathematical, scientific, and reading literacy.
• Problem-solving processes: the assessment is designed to find out the extent to which students are able to understand, characterise, represent and solve problems effectively. Students should also be able to reflect on the solution and to communicate it to others.
• Reasoning skills: beyond drawing on students’ knowledge, good problems also draw on their reasoning skills.

Problem-solving scores and problem-solving proficiency in PISA 2003

As in other PISA domains, problem-solving in PISA is scored on a scale assigning a point score to each student. Each task is given a score according to its difficulty, and the students are assigned individual scores according to the highest difficulty of task that they are likely to be able to perform.

The scale is devised with an average score for all the OECD countries of 500 and a standard deviation of 100, meaning that about two-thirds of students score between 400 and 600 points.

To aid interpretation, tasks are ranked at three ‘proficiency levels’ with each student assigned to the highest level at which they could perform the required tasks. Specifically, a student must be capable of getting a majority of tasks at a given proficiency level correct in order to be ranked at that level. Some students do not reach even the lowest level, Level 1. This does not mean that they are unable to perform any PISA problem-solving tasks but it does mean they would get the majority of simple tasks wrong. Table 1 provides detailed information on the three proficiency levels.

Students below Level 1 (less than 405 score points) are able to deal only with straightforward problems and structured tasks that require students to give responses based on facts or to make observations with few or no inferences. They are characterised as weak or emergent problem-solvers.

Appendix A gives more detail about the characteristics of problem-solvers at these levels. The boxes in this report give examples of problem-solving tasks at varying levels of difficulty.