PISA 2006: Mathematical Literacy - How ready are our 15-year-olds for tomorrow's world?

Introduction

This report examines the mathematical literacy results for New Zealand students from PISA 2006. The international findings for PISA 2006 were published by the OECD in two volumes in 2007 (OECD 2007a and 2007b). A summary of key New Zealand results from this study was published in December 2007 (Telford & Caygill 2007). Other reports in this series will focus on reading literacy, scientific literacy, school contexts, and attitude and engagement factors.¹

This report begins by providing an overview of the mathematical literacy domain, including what was assessed and how the results can be interpreted. Following this, the overall performance of New Zealand’s 15-year-olds in PISA 2006 is examined in comparison with other participating countries and over time. Finally, results for groups within the New Zealand population are presented according to different characteristics: gender, ethnic grouping, immigrant status, language spoken at home, and socio-economic status.

Definition of mathematical literacy

The PISA assessment frameworks (OECD 2006) define mathematical literacy as follows.

Mathematical literacy is an individual’s capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgements and to use and engage with mathematics in ways that meet the needs of that individual’s life as a constructive, concerned and reflective citizen.

Mathematical literacy questions in PISA are designed to cover a range of knowledge, skills, and abilities. The knowledge component of mathematical literacy includes knowledge of mathematical terminology, facts and procedures, while skills in performing mathematical operations and methods are also assessed. The ability to pose, formulate, solve and interpret problems using mathematics in a variety of situations or contexts is a crucial component of mathematical literacy. Within the mathematical literacy domain, each problem in the assessment has three components: the context, the mathematical content, and the competencies that students bring to answering questions.

How mathematical literacy was measured in PISA 2006

Each student was assessed for two hours with a pencil-and-paper test containing both multiple-choice and constructed-response questions. Background information was also collected by way of questionnaires completed by students, parents and school principals. Students were given one of thirteen assessment booklets containing different combinations of science, mathematics and reading tasks. Less testing time overall was provided for the two minor domains, mathematical and reading literacy, than the major domain, scientific literacy.²

The overall pool of mathematics tasks comprised a mix of tasks covering the three components of context, content, and competencies. In terms of the mathematical content of a question, each of the content areas was roughly one-quarter of the questions (23% space and shape, 23% uncertainty, 27% quantity, and 27% change and relationships). The context of these questions included questions from an educational or occupational setting (17%), a personal setting (19%), a scientific setting (25%), and a public setting (38%). In terms of the competencies involved, roughly one-quarter (23%) of questions required reproduction of practised knowledge, half of the questions required connections to be made in taking problem-solving to situations that are not routine, and just over one-quarter (27%) required reflections to be made by the student about the processes needed or used to solve a problem.³

Selected test questions

Figures A.1 to A.6 in Appendix 2 present examples of the types of questions used in PISA to assess mathematical literacy. Proportions of New Zealand students who correctly answered each question in PISA 2003 are given, along with results from a selection of countries. These items were released to the public after the 2003 cycle and so were not included in the PISA 2006 assessment. However, they do illustrate the difficulty of questions in PISA and are linked to proficiency levels (see below for an explanation).

How is PISA reported?

In PISA 2003, student performance in mathematics was reported separately for each of the four content areas - space and shape, uncertainty, quantity, and change and relationships - as well as the combined mathematics scale. In PISA 2006, however, because mathematics was a minor focus and had a smaller proportion of the testing time, results are only reported on a single combined scale. An OECD mean score of 500 points was established for PISA 2003 as the benchmark against which mathematics performance has since been measured.

Description of proficiency levels

In 2003, PISA developed proficiency levels to illustrate the range in mathematical literacy across 15-year-old students. These proficiency levels describe the types of tasks students achieving at each level were able to do and were linked to score points on the achievement scale. (See Table 1 for a brief description of the levels, with the associated score points at the boundary of the levels; also see Appendix 3 for a detailed proficiency description.) Note that students were considered to be proficient at a particular level if, on the basis of their overall performance, they could be expected to answer at least half of the questions in that level correctly. Typically, students who were proficient at higher levels had also demonstrated their abilities and knowledge at lower levels.

What can PISA results tell us?

PISA allows us to compare the performance of New Zealand 15-year-olds in mathematical literacy against that of their counterparts in 56 other countries. The minor domain results offer an update on overall performance rather than the in-depth analysis permitted by major domain results. Because the results from the 2006 assessment are only a second data point for mathematical literacy, any changes are indicative rather than indications of a longitudinal trend.

Two main measures⁴ will be examined in this report:

• the mean scores of particular groups of students on the combined mathematical literacy scale;
• the proportions of students within particular groups achieving at each proficiency level.
  

Table 1: Summary of PISA mathematical literacy proficiency levels

Source: Adapted from OECD 2007a. See Appendix 3 for the detailed proficiency level map.

Footnotes

1. Only the reading report was published at the time this report was released (Marshall et al. 2008), the rest are in press.
2. See Table A.2 in Appendix 1 for details of how the three-yearly PISA assessments are structured.
3. Values come from OECD 2007a, Table A5.3, p. 365.
4. Please refer to ‘Definitions and technical notes’ at the end of this report for further details.