PISA 2003: Problem solving for tomorrows' world Publications
Publication Details
This PISA03 thematic report explores the problemsolving skills of New Zealand 15yearold students. The problemsolving component examines their overall capacity to solve problems in reallife situations beyond the context of the school curriculum.
Author(s): Ministry of Education.
Date Published: November 2009
Summary
Background
PISA surveys the knowledge and skills of 15yearolds in the principal industrialised countries. The product of collaboration between participating governments through the OECD, it draws on leading international expertise to develop valid comparisons across countries and cultures.
PISA 2003 is the second assessment in the series. In this survey:
 Well over a quarter of a million students in 41 countries took part. All 30 OECD member countries participated, as well as ’partner countries’ in Asia, Africa, Eastern Europe, and Latin America.
 Each student took a twohour written test in their school.
 Students were assessed in mathematics, reading, science, and problemsolving. Mathematics was the main focus in 2003, while reading was the main focus in the first survey in 2000. The next PISA assessment in 2006 will focus on student performance in science.
The key features of the PISA approach are:
 Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons.
 The innovative ’literacy’ concept, which is concerned with the capacity of students to apply knowledge and skills in key subject areas and to analyse, reason, and communicate effectively as they pose, solve, and interpret problems in a variety of situations.
 Its inclusion of assessment that is not restricted to particular areas of the school curriculum. The assessment of ’problemsolving’ in 2003, the subject of this report, was the first such ’crosscurricular’ assessment.
 Its regularity, which will enable countries to monitor their progress in meeting the key learning objectives over time.
 Its consideration of student performance alongside the background characteristics of students at home and school in order to explore some of the main features associated with educational success. Each participating student and school completed a questionnaire that allowed a wide range of background information to be considered alongside student performance.
 Its breadth of geographical coverage, as the countries that have participated so far represent onethird of the world’s population and almost ninetenths of the world’s gross domestic product (GDP).
The comparisons made below set New Zealand students’ performance alongside that of students in the participating OECD countries, which are the countries in the survey most comparable to New Zealand: it excludes countries such as Tunisia, Peru and Russia from the analysis. A total of 29 OECD countries reported results in 2003. (The United Kingdom participated but did not meet the sampling requirements.)
Assessing problemsolving in PISA 2003
The collection of information about problemsolving skills in PISA 2003 was carried out to address concerns from the OECD countries that students’ capabilities in reading, mathematics, and science do not tell us enough about their overall capacity to solve problems in reallife situations beyond the context of the school curriculum.
According to the agreed PISA definition, problemsolving is:
… an individual’s capacity to use cognitive processes to confront and resolve real, crossdisciplinary situations where the solution path is not immediately obvious and where the literacy domains or curricular areas that might be applicable are not within a single domain of mathematics, science or reading.
PISA 2003 Learning for Tomorrow’s World, OECD: p. 26
The problemsolving component within PISA 2003 examines how well young adults are prepared to meet the challenges of today’s knowledge societies, focusing on young people’s ability to use their knowledge and skills to meet reallife challenges beyond the classroom.
The concept of problemsolving in PISA is defined by five dimensions:
 Problem types: PISA 2003 assesses three types of problems: decisionmaking, system analysis and design, and troubleshooting. For a description of what these three types of problems involve, see Appendix B. Examples of the three different types of problems are given in Boxes A to C of this report:
 A decisionmaking problem requiring students to find a suitable time and date to go to the cinema, when shown a timetable and a set of constraints (Box A, pages 911).
 A systems analysis and design problem requiring students to organise dormitory allocations in a children’s camp, given a set of rules and other information (Box B, pages 1315).
 A troubleshooting problem requiring students to diagnose a problem with the flow of water through an irrigation network (Box C, pages 1921).
 Problem context: problems are set in contexts outside the classroom and school curriculum, such as personal life, work and leisure, and community and society.
 Disciplines : the domain covers a wide range of disciplines and thus complements the main PISA domains of mathematical, scientific, and reading literacy.
 Problemsolving processes: the assessment is designed to find out the extent to which students are able to understand, characterise, represent and solve problems effectively. Students should also be able to reflect on the solution and to communicate it to others.
 Reasoning skills: beyond drawing on students’ knowledge, good problems also draw on their reasoning skills.
Problemsolving scores and problemsolving proficiency in PISA 2003
As in other PISA domains, problemsolving in PISA is scored on a scale assigning a point score to each student. Each task is given a score according to its difficulty, and the students are assigned individual scores according to the highest difficulty of task that they are likely to be able to perform.
The scale is devised with an average score for all the OECD countries of 500 and a standard deviation of 100, meaning that about twothirds of students score between 400 and 600 points.
To aid interpretation, tasks are ranked at three ‘proficiency levels’ with each student assigned to the highest level at which they could perform the required tasks. Specifically, a student must be capable of getting a majority of tasks at a given proficiency level correct in order to be ranked at that level. Some students do not reach even the lowest level, Level 1. This does not mean that they are unable to perform any PISA problemsolving tasks but it does mean they would get the majority of simple tasks wrong. Table 1 provides detailed information on the three proficiency levels.
Level & Corresponding Range of Point Scores  ProblemSolving Proficiency  

Level 1 405499 
Basic problemsolvers 
Students proficient at Level 1 solve problems where they have to deal with only a single data source containing discrete, welldefined information. Students at this level are generally not able to solve multifaceted problems that involve more than one data source or those that require reasoning. Students below Level 1 (less than 405 score points) are able to deal only with straightforward problems and structured tasks that require students to give responses based on facts or to make observations with few or no inferences. They are characterised as weak or emergent problemsolvers. 
Level 2 499592 
Reasoning, decisionmaking problemsolvers  Students proficient at Level 2 use reasoning and analytic processes and can solve problems requiring decisionmaking skills. Students apply various types of reasoning to analyse situations and solve problems that require them to make a decision from welldefined alternatives. 
Level 3 592 or above 
Reflective, communicative problemsolvers  Students proficient at Level 3 typically think about underlying relationships in problems and relate these to solutions as well as analysing situations and making decisions and communicating solutions clearly to others. 
Note:

Students below Level 1 (less than 405 score points) are able to deal only with straightforward problems and structured tasks that require students to give responses based on facts or to make observations with few or no inferences. They are characterised as weak or emergent problemsolvers.
Appendix A gives more detail about the characteristics of problemsolvers at these levels. The boxes in this report give examples of problemsolving tasks at varying levels of difficulty.
Navigation
Where to find out more
Contact Us
PISA
If you have any questions about PISA (Programme for International Student Assessment) please email: PISA Mailbox