Science achievement: What we know from New Zealand's participation in TIMSS 2014/15 and PISA 2015 Publications
Publication Details
In 2014 and 2015 New Zealand took part in two large international studies – the Trends in International Mathematics and Science Study (TIMSS), and the Programme for International Student Assessment (PISA). TIMSS focuses on the maths and science achievement of Year 5 and Year 9 students. PISA focuses on the reading, maths, and science literacy of 15-year-old students. Both studies also examine the context in which learning takes place.
TIMSS and PISA assessment frameworks have slightly different approaches to science – TIMSS focuses on curriculum, teaching, and learning, while PISA focuses on science literacy for life beyond school for the average student. While still assessing all three areas of literacy (reading, maths, and science), each cycle of PISA has a main focus on only one of these areas – in this cycle it was science.
Author(s): Comparative Education Research Unit, Ministry of Education.
Date Published: May 2017
Achievement over time for New Zealand students
Year 5 students had higher science achievement in 2014 than in 2011.
TIMSS was first conducted in 1994, and PISA began in 2000. TIMSS is conducted every four years, while PISA is conducted every three years. The trend in science achievement of Year 5 and Year 9 students has been measured since 1994, while PISA provides data on the science achievement of 15-year-old students since 2006. Although science was assessed in 2000 and 2003, 2006 was the first time science was the focus area of the PISA assessment1.
Exhibit 1 shows the trends in science achievement for students across the cycles of both studies.
- The average2 science achievement of Year 5 students in 2014/15 was the same as it wasfor the students 20 years ago, but significantly3 higher than in 2010/11.
- The average science achievement of Year 9 students in 2014/15 was the same as it was for the students 20 years ago, and was also the same as in 2010/11.
- For Year 5 and Year 9 students, TIMSS 2002/03 results remain the highest among all six cycles, even though the score difference for Year 9 students between 2002/03 and 2014/15 was not statistically significant.
- The average science achievement of 15-year-old students in 2015 was significantly lower than that of the students in 2006, but the same as it was in 2012. The majority of the decline occurred between 2009 and 2012.
Exhibit 1: Trends in science achievement for TIMSS and PISA
Note: A score of 500 is not the same across the year levels. However, within each year level, a score of 500 is the same across assessment years.
Range of science achievement
The range of achievement, that is, the gap between low and high achievers, is an important measure to use to examine equity in education. Rather than using the highest and lowest scores, range is commonly measured as the difference between the score at which only 5% of students achieved a lower score, and the score at which only 5% of students achieved a higher score. However, even when different measures of range are used, in PISA, the range for 15-year-old students in science has been wider than for many other countries, while in TIMSS, the range has varied over time. In recent cycles, TIMSS results have shown the range to be fairly moderate compared to other countries. However, in this latest cycle of TIMSS, there have been some changes.
- At the Year 5 level, the range of achievement in 2014/15 has not changed since 2010/11, but is not as wide as it was in 1994/95 and 1998/99.
- At the Year 9 level, the range of science achievement has increased. In 2014/15 this range was much wider than in 2002/03; the change occurred at both ends of the spectrum, with the top higher and the bottom lower than in 2002/03.
Low to advanced performance in science
13% of 15-year-old students were top performers in science.
Both TIMSS and PISA classify students by their performance, and link this performance to descriptions of the types of science tasks students could do.4 In TIMSS these classifications are called benchmarks and there are four: the advanced at 625 scale score points; the high at 550; the intermediate at 475; and the low at 400 scale score points.
In PISA these are called proficiency levels, and are numbered 1 to 6. Students at Level 6 are capable of advanced science thinking and reasoning, and are adept at using their scientific knowledge in a variety of complex situations, whereas those at Level 1 can only complete relatively basic science tasks. Level 2 is considered to be a baseline level at which students begin to demonstrate the competencies that will enable them to participate actively in science-related life situations.
- Six percent of Year 5 students, and ten percent of Year 9 students reached the advanced benchmark in 2014 (see Exhibit 2). Thirteen percent of 15-year-olds were top performing students who were at Level 5 or above in 2015 (3% were at Level 6).
- Just under half of the 15-year-old students who were top performers in PISA science were also top performers in reading and maths.
- At the other end of the spectrum, 12 percent of Year 5 and Year 9 students did not reach the low benchmark, and 17 percent of 15-year-old students performed below Level 2 in science. These students did not perform the simple science tasks designed for students at their grade and age grouping.
Exhibit 2: Proportion of high and low achievers in science in TIMSS and PISA
Note: Benchmarks and proficiency levels are not directly comparable across year levels, but are indicative of the range of achievement within New Zealand.
Changes over time
There have been some changes at each year level in the proportions of students at each benchmark – these changes tend to reflect the movement in average science achievement.
- The proportions of Year 5 students reaching each benchmark in 2014/15 was significantly lower than in 2002/03 (the TIMSS cycle with the best results for New Zealand students). However, the proportion of Year 5 students classified as intermediate
or high performers increased significantly between 2010/11 and 2014/15, driving the increase in average achievement for Year 5 students. - The proportion of Year 9 students not reaching the low benchmark was high in 2014/15 (12%), compared with 2002/03 (6%). However, the proportion of Year 9 students reaching each benchmark has not changed significantly between 2010/11 and 2014/15.
- There has been little change in the proportion of 15-year-old students in lower and higher proficiency levels in science since PISA 2012. However, there was an increase in the proportion of poor performers, from 14 percent in 2006, to 17 percent in 2015. Over the same period, the proportion of top performers has declined from 18 percent in 2006, to 13 percent in 2015.
How does New Zealand compare with other countries?
Compared to other countries, there was a relatively large proportion of Year 5 students who had low science performance.
Compared with other countries, New Zealand students' average science achievement ranked relatively low for primary students, but was relatively better for secondary students (see Exhibit 3). Rankings for New Zealand change as other countries enter or leave a study or if they improve or see a decline in their scores. However, rankings help form part of the picture of science achievement.
- New Zealand Year 5 students' average science achievement in 2014/15 was significantly higher than 14 countries (out of 47 countries), but lower than 30 countries, including all the other predominantly English-speaking countries who participated at the middle primary level.5
- Out of 39 countries, New Zealand Year 9 students' average science achievement in 2014/15 was significantly higher than 20 countries, but lower than 14 countries.
- The average science score of New Zealand 15-year-old students was significantly higher than 53 countries (out of 69 countries), and lower than 9 countries.
- For 15-year-old students, the difference between New Zealand and the OECD average has increased from 14 points above the OECD average in 2012, to 20 points above in 2015. This has arisen from decreases in average achievement for several OECD countries, meaning that compared to PISA 2012, New Zealand's position relative to other participating countries has improved.
High and low performers
Compared with other countries, New Zealand Year 5 students were around the middle when considering the proportion of advanced performers in science at the middle primary level. In contrast, New Zealand's proportions of Year 9 advanced performers, and 15-year-old top performers, were higher than the international median and the OECD average respectively.
As might be expected given the relatively higher average science achievement among secondary students, the proportion of Year 9 students and 15-year-olds who were lower performers was smaller than the international median and the OECD average respectively. In contrast, at the Year 5 level the proportion of lower performers was bigger than the international median.
Exhibit 3: Comparisons of average science scores for TIMSS and PISA countries
Notes:
- The shading on these lists shows those countries significantly above or below New Zealand in a darker shade, while those that are not significantly different are shaded the same as New Zealand.
- B-S-J-G (China) refers to the four PISA participating China provinces: Beijing; Shanghai; Jiangsu; and Guangdong.
- FYR Macedonia refers to the Former Yugoslav Republic of Macedonia.
- Standard errors are presented in parentheses.
Strengths and weaknesses of New Zealand students within science
New Zealand Year 5 and Year 9 students have consistently shown greatest strength in Earth science and life science (biology) questions, compared to physical science questions (Year 9 students had lower achievement particularly in chemistry questions in physical science). Similarly, in PISA 2006, Earth and space systems was an area of relative strength, while physical systems was an area of relative weakness in New Zealand. However, in 2015, 15-year-old students performed equally well on all content areas.
Both TIMSS and PISA also examine the thinking skills students are most likely to employ when answering a question or working on a problem. Interestingly, in TIMSS, New Zealand students show a relative strength in questions requiring reasoning, and in PISA, for New Zealand 15-year-olds, evaluating and designing scientific enquiry was an area of relative strength, compared to other scientific competencies. There has been an improvement in the averages at the Year 5 level for questions requiring knowing science, as well as questions requiring reasoning, between 2010/11 and 2014/15.
Science achievement of New Zealand boys and girls
Boys and girls performed about the same in science in both TIMSS and PISA.
Within all age groups, in the most recent TIMSS and PISA assessments, boys and girls had similar average science achievement; however, boys had a wider range than girls (see Exhibit 4).
This lack of difference has not always been the case over the years. For example, after a decrease for Year 9 girls between 2002/03 and 2010/11, they had lower average science achievement than Year 9 boys. However, this decrease was reversed by 2014/15, so girls' average science achievement was the same as that of boys. The PISA 2012 cohort largely overlapped with the TIMSS 2010 cohort. As the achievement data shows, Year 9 girls made an improvement by the time they turned 15 to reduce the gender gap in scores.
Exhibit 4: Average science achievement of boys and girls in TIMSS and PISA
Notes:
- The darker colour bar indicates the average score for that gender is significantly higher than the other gender.
- Standard errors are presented in parentheses.
Science achievement of students in different ethnic groupings
Both TIMSS and PISA asked students which ethnic groupings they belonged to, with the option of selecting more than one grouping. Both studies identified four major ethnic groups: Māori; Pasifika; Pākehā/ European; and Asian. All other ethnic identities were grouped under Other. There was a wide variation in science achievement, with advanced and very low achievers in all ethnic groupings.
Across both TIMSS and PISA, Māori and Pasifika students had lower average science achievement than non-Māori and non-Pasifika students. Analyses show that this achievement gap narrows when socio-economic factors are taken into account.
Changes over time
The average science achievement within Year 5 ethnic groupings has not changed significantly since 2010/11. However, for secondary students, there were some significant changes.
- The average science achievement of Year 9 students within the four major ethnic groupings has not changed significantly since 2010/11. The achievement of those classified as Other has decreased significantly in this period.
- The average science scores for 15-year-old students within Māori and Pasifika ethnic groupings have not changed significantly since 2006 and 2012. Students in the Asian and Pākehā/ European groupings had significantly lower scores in 2015 than in 2006, but similar to 2012.
Science achievement and socio-economic status
In PISA, compared to earlier cycles, a student's socio-economic background is not such a strong predictor of their achievement.
Regardless of the measure of socio-economic status, TIMSS and PISA both show that students in homes with many resources for learning had higher science achievement, on average, than those whose homes had fewer resources. Similarly, at a school level, science achievement was higher, on average, for students in schools with more affluent students than those in schools with more economically disadvantaged students. In both studies, the difference in science achievement between these two groupings within New Zealand was higher than in most other countries. However, in PISA, compared to earlier cycles, a student's socio-economic background is not such a strong predictor of how well they have achieved.
Footnotes
- Science was a minor focus and had only a limited number of test questions in the first two cycles of PISA in 2000 and 2003.
- The 'average' refers to the mean unless otherwise stated.
- The word 'significant' in this report is used to describe statistical significance. Statistical significance means we are 95% certain that we would get the same results if the study was repeated with a different sample.
- Full descriptions of benchmarks are provided in Martin, Mullis, Foy, & Hooper (2016), and descriptions of proficiency levels are provided in PISA 2015 Assessment and Analytical Framework (OECD, 2016a).
- Year 5 is referred to as middle primary and Year 9 is referred to as lower secondary where international comparisons are drawn.
References
- Caygill, R., Singh, S., & Hanlar, V. (2016). Science Year 5: Trends over 20 years in TIMSS. Wellington: Ministry of Education.
- Caygill, R., Hanlar, V., & Singh, S. (2016). Science Year 9: Trends over 20 years in TIMSS. Wellington: Ministry of Education.
- May, S., with Flockton, J., & Kirkham, S. (2016). PISA 2015 New Zealand summary report. Wellington: Ministry of Education.
- Martin M. O., Mullis, I. V. S., Foy, P., & Hooper, M. (2016). TIMSS 2015 international results in science.
- OECD. (2016a). PISA 2015 assessment and analytical framework: Science reading, mathematic and financial literacy. Paris: Author.
- OECD. (2016b). PISA 2015 results (volume 1): Excellence and equity in education. Paris: Author.
Navigation
Where to find out more
Contact Us
PISA
If you have any questions about PISA (Programme for International Student Assessment) please email: PISA Mailbox