TIMSS and the New Zealand mathematics curriculum

Mathematics curriculum levels and the TIMSS content domains

In order to gain greater understanding of the relationship between the Mathematics in the New Zealand Curriculum (MiNZC)¹ levels and student achievement in TIMSS, New Zealand teachers were asked at which level of the MiNZC most of the students in their class were currently working for each of the strands: Number, Algebra, Measurement, Geometry, and Statistics.² Note that the information was not collected for individual students but for the majority of the TIMSS students the teacher taught. For the purpose of analysis, a teacher’s response has been assigned to each individual student in the class. Figure 7 shows that while the majority of Year 5 students were working at level 3 of the curriculum, there were still a significant number of students in classes working at level 2, particularly in the Algebra and Measurement strands.

Since the TIMSS domains were similar to the MiNZC strands in terms of content, with Measurement and Geometry combined similar to the geometric shapes and measures domain, and Number and Algebra combined similar to the number domain, the figure also maps the mean TIMSS domain score for the students estimated to be working at each level of the MiNZC. For example, the mean score in the TIMSS number domain for the 57 percent of students in classes estimated to be working at level 3 of the Number strand of the MiNZC was 501 scale score points. The figure shows that students whose classes are working at higher levels of the curriculum have higher achievement on the associated TIMSS content domain. Note that no attempt is being made here to infer a causal link – that is, we are not saying the higher mean achievement is because they are working at the higher level.

It is interesting to look at these results in an international context and observe that if only those students working at level 3 of the curriculum were included in the TIMSS testing, New Zealand’s overall mathematics score would still have been below that of the high-performing countries, Singapore, Chinese Taipei, Hong Kong SAR, and Japan. For example, the mean score for Singaporean students on the number domain was 611 scale score points, while New Zealand students working at level 3 had a mean score of 501 scale score points.

Figure 7: Mean content area achievement by New Zealand curriculum strands

Note:
The bars on the graph represent the proportions of Year 5 students while the points represent mean scores. Lines extending from the points represent the 95% confidence interval, i.e. the range within which we are 95 percent confident that the true population value lies.
In the cases where there are no mean scores, the ‘other’ grouping, there were too few students to report achievement. Scores presented for levels 1, 4, and combined 2 & 3 should be treated with caution as the proportion of students is in each of these groups is small.
Standard errors are presented in parentheses.

Curriculum match

Questions about international studies often focus on the appropriateness of the assessment questions for New Zealand students. New Zealand is not unique in asking this question; other countries are also concerned with appropriateness of the tests. The TIMSS assessment questions are developed through a collaborative process that begins with the development of an assessment framework. The 'TIMSS 2007 assessment frameworks' (Mullis, Martin, et al., 2005) were designed to specify the important aspects of mathematics that participating countries agreed should be the focus of an international assessment of mathematics achievement. However it is inevitable that the tests included questions that were unfamiliar to some students in some countries. In order to investigate the extent to which the TIMSS 2006/07 assessment was relevant to each country’s curriculum, TIMSS conducted a Test-Curriculum Matching Analysis (TCMA). The TCMA was also used to investigate the impact of selecting only appropriate questions on a country’s performance.

For the TCMA, each question was examined using the following two criteria:

• whether or not the topic of the question is in the intended curriculum for the majority of middle primary students (in our case Year 5) – that is, more than 50 percent; and
• whether or not the question topic is intended to be encountered by the middle primary students prior to the TIMSS testing (testing for New Zealand Year 5 students occurred in the beginning of November).
  

While all questions, regardless of this analysis, were included in any overall results reported for TIMSS, this analysis was used to ascertain the level to which the results might change for New Zealand if only questions judged appropriate were included in the tests. The analysis also included an examination of how students in other countries would fare if given only the "New Zealand-appropriate" test.

Table 10 shows the proportion of questions considered appropriate to the New Zealand curriculum in each of the TIMSS content areas. However, it should be noted that New Zealand's mathematics curriculum provides some challenges for deciding whether at least half of Year 5 students are likely to have met the question topics in the TIMSS test.³ The curriculum is not prescriptive, instead providing some broad guidelines of mathematics concepts and skills that schools can choose to cover. Schools are encouraged to design mathematics programmes that are relevant to their students and communities. Consequently, when schools plan their mathematics programmes there is considerable variation between them. Another challenge is that the broad achievement objectives are grouped in levels which cover approximately two years of schooling. As shown in the previous section, New Zealand Year 5 students were generally working at levels 2 and 3 of the curriculum, so information from levels 1, 2, and 3 was used to guide judgements on the TCMA.⁴

Table 10: Appropriateness of the TIMSS tests to the New Zealand curriculum

Note: Number corresponds to the Number and Algebra strands of the curriculum, geometric shapes and measures corresponds to the Measurement and Geometry strands of the curriculum, and data display corresponds to the Statistics strand of the curriculum.

Although only around 85 percent of the questions were judged appropriate for New Zealand students, the TIMSS TCMA analysis shows that some of the higher-performing countries would have done better on the ‘New Zealand’ test than New Zealand Year 5 students, as shown in Table 11.

Table 11: Performance of middle primary students from selected countries on the ‘New Zealand’ appropriate test.

Source: Adapted from Exhibit C.1 in Mullis, Martin, and Foy, 2008. 

Footnotes

1. This was the curriculum in place at the time of testing.
   
2. The Mathematical Processes strand of the MiNZC relates more to the TIMSS cognitive domains and so is not included in this discussion.
3. Note that for the TCMA, the curriculum document used was the 1992 Mathematics in the New Zealand Curriculum (Ministry of Education, 1992).
   
4. Thanks to Chris Joyce and Ally Bull from NZCER for this work.