Summary

What is PISA?

• The Programme for International Student Assessment (PISA) is an international study of the knowledge, skills, and competencies of 15-year-old students in the domains of reading literacy, mathematics literacy, science literacy, and in 2022, creative thinking. Read more here: https://www.oecd.org/pisa/.

How are students and schools selected?

• To ensure that the data accurately reflects the true performance, experiences and opinions of 15-year-olds, schools and students are randomly selected through a two-stage stratified sampling design.
• For PISA 2022, a representative sample of 227 Aotearoa New Zealand English-Medium schools (with 15-year-old students) from a total of 450 was selected by the international PISA Sampling organisation Westat. Larger schools had a greater chance of being selected because they had more 15-year-olds.
• After the schools agreed to participate, 25 to 75 students (depending on the school size) were then randomly selected from each school, by the Aotearoa New Zealand PISA National Centre, using custom-written software for the study.
• PISA 2022 was conducted during challenging times (Term 3 of 2022), and Aotearoa New Zealand was unable to meet the very strict international participation standards of 85% of schools and 80% of students taking part (72% of selected schools and students participated).

What is a non-response bias analysis?

• To determine how well the achieved PISA 2022 sample reflects the 15-year-old population, or the quality of the achieved sample, we compared demographic characteristics of the students taking part with known characteristics of the 15-year-old student population.
• For instance, did more boys participate than girls? If more boys than girls participated, we would say our data is biased towards boys and our data may not accurately represent all 15-year-olds.

What did we find out?

• Our school sample of 169 schools represented schools well across different deciles, authority (state vs. independent/private), gender (co-educational vs. boys vs. girls), location (rural vs. urban) and size.
• Our student sample was found to be slightly biased. The analysis showed slightly more high achieving students participated. This means the results will likely show Aotearoa New Zealand in a more positive light than if we had an unbiased sample, potentially by about 10 PISA points.
• Unsurprisingly, students who were chronically absent around the time PISA 2022 was conducted in New Zealand were less likely to take part, and students who were regular attenders were more likely to take part.
• It is important to note that the bias may not affect trend or country comparisons to its full extent, because a similar (although, likely, smaller) bias may also affect past results and the results of other countries and economies.

How do we interpret the PISA 2022 Results?

• In survey research, no matter how successful it is, there is always some uncertainty in the precision of what we are measuring and the possibility of some form of bias in our results.
• Typically, the calculation of standard errors and confidence intervals for population estimates such as mean PISA scores allow for small amounts of bias or uncertainty that may be within a sample. Since our non-response bias analysis suggests a potentially larger amount of bias than standard errors can account for, more caution should be used in interpreting findings.
• However, the fact that bias is identified means that we are better equipped when reading the results while still being cautious before arriving at conclusions. It means that:
  • If the data shows better outcomes than in the past, on performance or in survey data, we cannot be certain of the extent to which it is due to the full population improving or higher achievers being over-represented in the data.
  • Where results show worse outcomes than in the past, we can be fairly confident that the outcomes are truly poorer.
  • We cannot be certain that if there are changes, what the size of the change is. For example, country ‘rankings’ become difficult to conclude.

Conclusion

The adversity that families and students have gone through in the past few years was immense, and so were the impacts of attending school in the context of a public health crisis. In that context, our response rate of 72% was a success and a true credit to the hard work of participating schools.

Taken all together, the PISA 2022 results can still help paint a broad picture of the performance of the education system and help us understand the impact that the COVID-19 pandemic had on student learning and engagement. We can triangulate the results with other research and frame the results in the context of what was occurring in Term 3 of 2022. Most importantly, the data can be used as a benchmark of where we started from immediately following the height of the pandemic, as the education system recovers from adversity, and inform us on our progress towards better outcomes for our learners.

External factors like COVID-19 disruptions and weather events influenced participation rates, but initiatives aimed at improving student attendance and engagement, such as the Attendance and Engagement Strategy, may positively affect participation in future studies.