TIMSS and the New Zealand science curriculum

Science curriculum levels and the TIMSS content domains

In order to gain greater understanding of the relationship between the Science in the New Zealand Curriculum (SciNZC)¹ levels and student achievement in TIMSS, New Zealand teachers were asked at which level of the SciNZC most of the students in their class were currently working for each of the strands: Living World, Material World, Physical World, Planet Earth and Beyond. Note that the information was not collected for individual students but for the majority of the TIMSS students the teacher taught. For the purpose of analysis, a teacher’s response has been assigned to each individual student in the class. Figure 7 shows that while the majority of Year 5 students were working at level 3 of the curriculum, there were still a significant number of students in classes working at level 2.

Since the TIMSS domains were similar to the SciNZC strands in terms of content, with Material World and Physical World combined similar to the physical science domain, the figure also maps the mean TIMSS domain score for the students estimated to be working at each level of the SciNZC. For example, the mean score in the TIMSS life science domain for the 59 percent of students in classes estimated to be working at level 3 of the Living World strand of the SciNZC was 526 scale score points. The figure shows that students whose classes were working at higher levels of the curriculum have higher achievement on the associated TIMSS content domain. Note that no attempt is being made here to infer a causal link – that is, we are not saying the higher mean achievement is because they are working at the higher level.

It is interesting to look at these results in an international context and observe that if only those students working at level 3 of the curriculum were included in the TIMSS testing, New Zealand’s overall science score would still have been below that of the high-performing countries, Singapore, Chinese Taipei, Hong Kong SAR, and Japan. For example, the mean score for Singaporean students on the life science domain was 582 scale score points, while New Zealand students working at level 3 had a mean score of 526 scale score points.

Figure 7: Mean content area achievement by New Zealand curriculum strands

Note:
The bars on the graph represent the proportions of Year 5 students while the points represent mean scores. Lines extending from the points represent the 95% confidence interval, i.e. the range within which we are 95 percent confident that the true population value lies.
In the cases where there are no mean scores, the ‘other’ grouping, there were too few students to report achievement. Scores presented for levels 1, 4, and combined 2 &3 should be treated with caution as the proportion of students is in each of these groups is small.
Standard errors are presented in parentheses.

Curriculum match

Questions about international studies often focus on the appropriateness of the assessment questions for New Zealand students. New Zealand is not unique in asking this question; other countries are also concerned with appropriateness of the tests. The TIMSS assessment questions are developed through a collaborative process that begins with the development of an assessment framework. The TIMSS 2007 assessment frameworks (Mullis, Martin, et al., 2005) were designed to specify the important aspects of science that participating countries agreed should be the focus of an international assessment of science achievement. However it is inevitable that the tests included questions that were unfamiliar to some students in some countries. In order to investigate the extent to which the TIMSS 2006/07 assessment was relevant to each country’s curriculum, TIMSS conducted a Test-Curriculum Matching Analysis (TCMA). The TCMA was also used to investigate the impact of selecting only appropriate questions on a country’s performance.

For the TCMA, each question was examined using the following two criteria:

• whether or not the topic of the question is in the intended curriculum for the majority of middle primary students (in our case Year 5) – that is, more than 50 percent; and
• whether or not the question topic is intended to be encountered by the middle primary students prior to the TIMSS testing (testing of New Zealand Year 5 students occurred in the beginning of November).

While all questions, regardless of this analysis, were included in any overall results reported for TIMSS, this analysis was used to ascertain the level to which the results might change for New Zealand if only questions judged appropriate were included in the tests. The analysis also included an examination of how students in other countries would fare if given only the "New Zealand-appropriate" test.

Table 10 shows the proportion of questions considered appropriate to the New Zealand curriculum in each of the TIMSS content areas. However, it should be noted that New Zealand's science curriculum provides some challenges for deciding whether at least half Year 5 students are likely to have met the question topics in the TIMSS test.² The curriculum is not prescriptive, instead providing some broad guidelines of science concepts and skills that schools can choose to cover. Schools are encouraged to design science programmes that are relevant to their students and communities. Consequently, when schools plan their science programmes there is considerable variation between them. Another challenge is that the broad achievement objectives are grouped in Levels which cover approximately two years of schooling. As shown in the previous section, New Zealand Year 5 students were generally working at levels 2 and 3 of the curriculum so information from levels 1, 2, and 3 was used to guide judgements on the TCMA.³

Table 10: Appropriateness of the TIMSS tests to the New Zealand curriculum

Note: Life science corresponds to the Living World strand of the curriculum, physical science corresponds to a combination of the Physical and Material World strands of the curriculum, and earth science corresponds to the Planet Earth and Beyond strand of the curriculum.

Although only around two-thirds of the questions were judged appropriate for New Zealand students, the TIMSS TCMA analysis shows that some of the higher-performing countries would have done better on the ‘New Zealand’ test than New Zealand Year 5 students, as shown in Table 11.

Table 11: Performance of middle primary students from selected countries on the ‘New Zealand’ appropriate test

Source: Adapted from Exhibit C.1 in Martin, Mullis, and Foy, 2008.

Footnotes

1. This was the curriculum in place at the time of testing.
   
2. Note that for the TCMA, the curriculum document used was the 1993 Science in the New Zealand Curriculum (Ministry of Education, 1993).
   
3. Thanks to Chris Joyce and Ally Bull from NZCER for this work.