Who can PISA tell us?

• PISA monitors overall student performance and well-being so that we can observe whether literacy is improving, remaining stable or declining over time.
• PISA also allows us to examine New Zealand’s performance relative to the rest of the world to learn from the experiences of other countries and for us to spread our current practices.
• The PISA questionnaire collects data on different factors in the system that contribute to student achievement over time including aspects of well-being, family background, the school environment, what happens in the classroom, and students’ own mindsets, experiences, and perceptions.
• International and national organisations trust and use PISA data. For example, UNESCO uses PISA data to measure countries’ progress towards the 2030 Sustainable Development Goal #4 which aims for inclusive and equitable quality education for all students regardless of gender or wealth.
• PISA cannot measure everything we value, so will only ever show specific aspects of a student’s competences. Therefore, it is important to combine PISA findings with our understanding of performance from other research that uses other methodologies.

What does PISA 2022 assess?

• In 2022, mathematical literacy was the major focus and reading and scientific literacy were minor focuses.  Each subject framework is refreshed every nine years to ensure it accurately reflects the essential skills needed in the current day and this cycle it was mathematics.
• The term literacy captures the “capacity of students to apply knowledge and skills in key subjects, and to analyse, reason and communicate effectively as they identify, interpret and solve problems in a variety of situations”.
• Developed by experts around the world, the PISA 2022 mathematics framework aligns well with the New Zealand Curriculum (NZC, 2007) and Te Mātaiaho. For instance, there are direct connections between the learning areas of number and algebra, geometry and measurement and statistics to the four PISA knowledge areas, as well as between the Te Mātaiaho key competencies and PISA 21st Century skills.
• Most 15-year-olds are in Year 11 and are expected to be working at Level 5 to Level 6 of NZC. Of the PISA mathematics items that were open-ended, the PISA test markers concluded that 99% were at curriculum Level 6 or lower, 88% were Level 5 or lower, and 50% were at Level 4 or lower.
• Creative thinking was the innovative domain assessed for the first time in 2022. Creative thinking can help students adapt to a constantly and rapidly changing world, and one that demands flexible workers equipped with ‘21st century’ skills that go beyond core literacy and numeracy.
• The test questions are not what you would find in a typical standardised test. They do not measure memorisation of facts or curriculum, but rather demand that students draw on their knowledge and use their problem-solving skills to respond to real-world scenarios (see example).

• 

• Students and principals were asked to complete a background questionnaire that provides further information to help better understand the data. Students cannot learn everything they need to know in school. To be effective lifelong learners, young people need not only knowledge and skills, but also an awareness of why and how they learn. That is why PISA asks students about their motivations, beliefs about themselves and learning strategies.

Who takes part and how are they selected?

• Approximately half a million students from 81 countries/economies, including 37 OECD countries, took part in PISA 2022. Countries all over the world took part including South, Central and North America, Europe, Asia, the Middle East, the Pacific, and Oceania.
• In New Zealand, almost 4,700 students from 169 English-Medium schools took part in the study in Term 3 of 2022. Both school and student identities remain confidential.
• Schools are selected by the international statistical organisation, Westat, from a list of all schools in New Zealand with PISA-eligible students enrolled. Schools are selected to be representative across a range of characteristics: school size (number of students), decile, location (major urban or smaller communities), authority (state or independent) and type (coeducational or single sex).
• Students are selected randomly from a list of all PISA-eligible students in each school using software. Students are eligible for PISA if they are between 15 years 3 months and 16 years 2 months (hereafter, ’15-year-olds’) and enrolled in Year 8 and above in the school. Most students are in Year 11 with about 10% in Year 10, and 10% in Year 11.

How is PISA developed?

• PISA is the result of a collaboration between the OECD Secretariat, international contractors, experts, and teams at participating countries and economies.
• First, assessment and questionnaire frameworks are developed under the guidance of expert groups. Participating countries/economies are then invited to submit questions that are then added to items developed by the OECD’s experts. The questions are reviewed by the by participating countries/economies and are carefully checked for cultural bias. For instance, the PISA research team in Aotearoa New Zealand employs teachers and professional development partners to review each item’s cultural relevance and if it is cognitively appropriate for 15-year-olds. Only those questions that pass these checks are used in PISA.
• The questions are then adapted to individual countries’ contexts and if needed, translated to the language of instruction. Adaptations ensure students read about scenarios within contexts they are familiar with, and the language and names reflect Aotearoa New Zealand.
• Countries who translate go through a strict process where they perform two independent translations which are reconciled and then verified by a third party. This is to ensure any differences in performance cannot be attributed to language differences.
• Before the main test is conducted there is a trial test run in all participating countries and economies. If any test questions prove to be too easy or too difficult in certain countries and economies, for reasons not related to the overall level of proficiency of students (as demonstrated on the remaining questions), they are dropped from the main test in all countries and economies.

How is PISA administered?

• PISA uses high quality technical standards which all countries must follow. Read the PISA 2022 Technical Standards here.
• Test administrators are hired to lead the test and survey sessions. They follow identical instructions and script to ensure the test conditions are uniform around the world.
• Independent quality monitors conduct ‘surprise’ site visits to schools to ensure that the rules are followed globally.
• Stats NZ has closely scrutinised the methodological rigor underpinning PISA which is why Cabinet approved it as a Tier 1 statistic. That means PISA meets high expectations of impartiality and statistical quality, has long-term continuity, and provides international comparability.