PISA 2003: Problem solving for tomorrows' world Publications
Publication Details
This PISA-03 thematic report explores the problem-solving skills of New Zealand 15-year-old students. The problem-solving component examines their overall capacity to solve problems in real-life situations beyond the context of the school curriculum.
Author(s): Ministry of Education.
Date Published: November 2009
Summary
Background
PISA surveys the knowledge and skills of 15-year-olds in the principal industrialised countries. The product of collaboration between participating governments through the OECD, it draws on leading international expertise to develop valid comparisons across countries and cultures.
PISA 2003 is the second assessment in the series. In this survey:
- Well over a quarter of a million students in 41 countries took part. All 30 OECD member countries participated, as well as ’partner countries’ in Asia, Africa, Eastern Europe, and Latin America.
- Each student took a two-hour written test in their school.
- Students were assessed in mathematics, reading, science, and problem-solving. Mathematics was the main focus in 2003, while reading was the main focus in the first survey in 2000. The next PISA assessment in 2006 will focus on student performance in science.
The key features of the PISA approach are:
- Its policy orientation, with design and reporting methods determined by the need of governments to draw policy lessons.
- The innovative ’literacy’ concept, which is concerned with the capacity of students to apply knowledge and skills in key subject areas and to analyse, reason, and communicate effectively as they pose, solve, and interpret problems in a variety of situations.
- Its inclusion of assessment that is not restricted to particular areas of the school curriculum. The assessment of ’problem-solving’ in 2003, the subject of this report, was the first such ’cross-curricular’ assessment.
- Its regularity, which will enable countries to monitor their progress in meeting the key learning objectives over time.
- Its consideration of student performance alongside the background characteristics of students at home and school in order to explore some of the main features associated with educational success. Each participating student and school completed a questionnaire that allowed a wide range of background information to be considered alongside student performance.
- Its breadth of geographical coverage, as the countries that have participated so far represent one-third of the world’s population and almost nine-tenths of the world’s gross domestic product (GDP).
The comparisons made below set New Zealand students’ performance alongside that of students in the participating OECD countries, which are the countries in the survey most comparable to New Zealand: it excludes countries such as Tunisia, Peru and Russia from the analysis. A total of 29 OECD countries reported results in 2003. (The United Kingdom participated but did not meet the sampling requirements.)
Assessing problem-solving in PISA 2003
The collection of information about problem-solving skills in PISA 2003 was carried out to address concerns from the OECD countries that students’ capabilities in reading, mathematics, and science do not tell us enough about their overall capacity to solve problems in real-life situations beyond the context of the school curriculum.
According to the agreed PISA definition, problem-solving is:
… an individual’s capacity to use cognitive processes to confront and resolve real, cross-disciplinary situations where the solution path is not immediately obvious and where the literacy domains or curricular areas that might be applicable are not within a single domain of mathematics, science or reading.
PISA 2003 Learning for Tomorrow’s World, OECD: p. 26
The problem-solving component within PISA 2003 examines how well young adults are prepared to meet the challenges of today’s knowledge societies, focusing on young people’s ability to use their knowledge and skills to meet real-life challenges beyond the classroom.
The concept of problem-solving in PISA is defined by five dimensions:
- Problem types: PISA 2003 assesses three types of problems: decision-making, system analysis and design, and trouble-shooting. For a description of what these three types of problems involve, see Appendix B. Examples of the three different types of problems are given in Boxes A to C of this report:
- A decision-making problem requiring students to find a suitable time and date to go to the cinema, when shown a timetable and a set of constraints (Box A, pages 9-11).
- A systems analysis and design problem requiring students to organise dormitory allocations in a children’s camp, given a set of rules and other information (Box B, pages 13-15).
- A trouble-shooting problem requiring students to diagnose a problem with the flow of water through an irrigation network (Box C, pages 19-21).
- Problem context: problems are set in contexts outside the classroom and school curriculum, such as personal life, work and leisure, and community and society.
- Disciplines : the domain covers a wide range of disciplines and thus complements the main PISA domains of mathematical, scientific, and reading literacy.
- Problem-solving processes: the assessment is designed to find out the extent to which students are able to understand, characterise, represent and solve problems effectively. Students should also be able to reflect on the solution and to communicate it to others.
- Reasoning skills: beyond drawing on students’ knowledge, good problems also draw on their reasoning skills.
Problem-solving scores and problem-solving proficiency in PISA 2003
As in other PISA domains, problem-solving in PISA is scored on a scale assigning a point score to each student. Each task is given a score according to its difficulty, and the students are assigned individual scores according to the highest difficulty of task that they are likely to be able to perform.
The scale is devised with an average score for all the OECD countries of 500 and a standard deviation of 100, meaning that about two-thirds of students score between 400 and 600 points.
To aid interpretation, tasks are ranked at three ‘proficiency levels’ with each student assigned to the highest level at which they could perform the required tasks. Specifically, a student must be capable of getting a majority of tasks at a given proficiency level correct in order to be ranked at that level. Some students do not reach even the lowest level, Level 1. This does not mean that they are unable to perform any PISA problem-solving tasks but it does mean they would get the majority of simple tasks wrong. Table 1 provides detailed information on the three proficiency levels.
Level & Corresponding Range of Point Scores | Problem-Solving Proficiency | |
---|---|---|
Level 1 405-499 |
Basic problem-solvers |
Students proficient at Level 1 solve problems where they have to deal with only a single data source containing discrete, well-defined information. Students at this level are generally not able to solve multifaceted problems that involve more than one data source or those that require reasoning. Students below Level 1 (less than 405 score points) are able to deal only with straightforward problems and structured tasks that require students to give responses based on facts or to make observations with few or no inferences. They are characterised as weak or emergent problem-solvers. |
Level 2 499-592 |
Reasoning, decision-making problem-solvers | Students proficient at Level 2 use reasoning and analytic processes and can solve problems requiring decision-making skills. Students apply various types of reasoning to analyse situations and solve problems that require them to make a decision from well-defined alternatives. |
Level 3 592 or above |
Reflective, communicative problem-solvers | Students proficient at Level 3 typically think about underlying relationships in problems and relate these to solutions as well as analysing situations and making decisions and communicating solutions clearly to others. |
Note:
|
Students below Level 1 (less than 405 score points) are able to deal only with straightforward problems and structured tasks that require students to give responses based on facts or to make observations with few or no inferences. They are characterised as weak or emergent problem-solvers.
Appendix A gives more detail about the characteristics of problem-solvers at these levels. The boxes in this report give examples of problem-solving tasks at varying levels of difficulty.
Navigation
Where to find out more
Contact Us
PISA
If you have any questions about PISA (Programme for International Student Assessment) please email: PISA Mailbox