Trends in New Zealand mathematics achievement 1994 to 2006

Trends in means and ranges since 1994

New Zealand has participated in TIMSS since its inception in 1994. In 1998, although no assessment was offered internationally at the middle primary level, New Zealand opted to repeat the 1994 assessment. Therefore, we now have information from four different assessments of mathematics achievement. Figure 1 presents the distributions of mathematics achievement of New Zealand Year 5 students over the four cycles of TIMSS.

The results from an examination of mathematics achievement since 1994 (see Figure 1) show that mean mathematics achievement in 2006 is higher than 1994, the first cycle of TIMSS. Although the mean score for 2006 is numerically lower than 2002, the difference between 2002 and 2006 is not significant.

It is also useful to look at the range of achievement as represented by the outer limits of achievement. The lowest outer limit presented in Figure 1 is the 5th percentile – the score at which only five percent of students achieved a lower score and 95 percent of students achieved a higher score. The highest outer limit is the 95th percentile – the score at which only five percent of students achieved a higher score and 95 percent of students a lower score. In addition, the 25th and 75th percentiles are presented in Figure 1, along with the inter-quartile range.

As shown in Figure 1, the range of achievement was narrower in 2006 than 1998 and 1994, but not as narrow as in 2002. The positive aspect of this change is that fewer students are demonstrating very low achievement, while a similar proportion of New Zealand students are gaining very high scores.

Figure 1: Distribution of New Zealand Year 5 mathematics achievement in TIMSS from 1994 to 2006

Note:
For trend purposes, only students tested in English are included in the results for 2002.
Standard errors are presented in parentheses

Trends in benchmarks for mathematics

In order to describe more fully what achievement on the mathematics scale means, the TIMSS international researchers have developed benchmarks. These benchmarks link student performance on the TIMSS mathematics scale to performance on the mathematics questions and describe what students can typically do at set points on the mathematics achievement scale. The international mathematics benchmarks are four points on the mathematics scale; the advanced benchmark (625), the high benchmark (550), the intermediate benchmark (475), and the low benchmark (400). The performance of students reaching each benchmark is described in relation to the types of questions they answered correctly. Table 1 presents the descriptions of the international benchmarks of mathematics achievement.

Table 1: TIMSS 2006/07 international benchmarks of mathematics achievement

Advanced international benchmark – 625

Students can apply their understanding and knowledge in a variety of relatively complex situations and explain their reasoning. They can apply proportional reasoning in a variety of contexts. They demonstrate a developing understanding of fractions and decimals. They can select appropriate information to solve multi-step word problems. They can formulate or select a rule for a relationship. Students can apply geometric knowledge of a range of two- and three-dimensional shapes in a variety of situations. They can organize, interpret, and represent data to solve problems.

High international benchmark – 550

Students can apply their knowledge and understanding to solve problems. Students can solve multi-step word problems involving operations with whole numbers. They can use division in a variety of problem situations. They demonstrate understanding of place value and simple fractions. Students can extend patterns to find a later specified term and identify the relationship between ordered pairs. Students show some basic geometric knowledge. They can interpret and use data in tables and graphs to solve problems.

Intermediate international benchmark – 475

Students can apply basic mathematical knowledge in straightforward situations. Students at this level demonstrate an understanding of whole numbers. They can extend simple numeric and geometric patterns. They are familiar with a range of two-dimensional shapes. They can read and interpret different representations of the same data.

Low international benchmark – 400

Students have some basic mathematical knowledge. Students demonstrate an understanding of adding and subtracting with whole numbers. They demonstrate familiarity with triangles and informal coordinate systems. They can read information from simple bar graphs and tables.

Source: Exhibit 2.1 from Mullis, Martin, and Foy, 2008.

Table 2 presents the proportions of New Zealand Year 5 students that reached each of the benchmarks in each cycle from 1994 to 2006. Note that the proportion shown for the low benchmark also includes students who performed at the advanced, high, and intermediate benchmarks. This is because, by definition, students who could do the more complex questions associated with, for example, the high benchmark, would also be able to complete the easier questions associated with the intermediate and low benchmarks.

The proportion of New Zealand students reaching the high, intermediate, and low benchmarks has been steadily rising since 1994 although the advanced benchmark has not changed significantly during this time. Five percent of students reached the advanced benchmark in 2006. The proportion of students reaching the high, intermediate and low benchmarks, which peaked in 2002 (27%, 62%, and 86% respectively), has been maintained in the 2006 results. The differences between the benchmarks in 2006 compared with 2002 are not of statistic significance.

There was also a group of Year 5 students in each cycle who did not reach the low benchmark. In terms of the benchmark definitions, these were students who did not demonstrate some basic mathematical knowledge. This group was proportionally largest in 1994 (22%) and smallest in 2002 (14%).

Table 2: Trends in proportions of Year 5 students at each benchmark from 1994 to 2006

Year	Percentage of Year 5 students reaching each benchmark
	Advanced	High	Intermediate	Low
2006	5 (0.5)	27 (1.0)	61 (1.1)	85 (1.0)
2002	5 (0.5)	27 (1.2)	62 (1.3)	86 (1.0)
1998	5 (0.9)	24 (1.9)	55 (2.5)	81 (1.8)
1994	4 (0.6)	20 (1.4)	51 (1.9)	78 (1.7)

Note: Standard errors are presented in parentheses.

Trends on the test questions

At the end of each cycle of TIMSS, test questions are released into the public domain. At the beginning of the next cycle, new questions are developed to replace the released questions. In addition, each cycle of TIMSS includes some questions from the previous cycle(s) to provide a trend measure over time. This section presents an analysis of the trend questions included in both TIMSS 2002/03 and TIMSS 2006/07. Note that no questions from TIMSS 1994/95 were included in the TIMSS 2006/07 assessment.

There were 79 questions common to both the 2002/03 and 2006/07 cycles. Of these 79 questions, 10 questions had similar proportions of students correctly answering them across two cycles (as shown in Table 3). There were a number of questions (39) that proportionally fewer students correctly answered in 2006 compared with 2002. In contrast, there were 30 questions that proportionally more students correctly answered in 2006 compared with 2002. When the change in proportions of students correctly answering was averaged across all the common questions, this represented a decrease of 0.6 percent.

This analysis reiterates that the decrease of New Zealand’s mean mathematics score by 3 scale score points (from 495 to 492) is not statistically significant.

Table 3: Trends in question statistics for mathematics questions common to 2002/03 and 2006/07

Change between 2002/03 and 2006/07	Decrease by 5% or more	Decrease by between 1% and 5%	Increase or decrease by 1% or less	Increase by between 1% and 5%	Increase by 5% or more
Number of questions	14	25	10	21	9

It is interesting to note that of the 14 questions in the group that decreased by 5 percent or more (when the proportion of students correctly answering in 2006 was compared with 2002), there were proportionally more of the geometric shapes and measures, and data display questions than number questions. In contrast, proportionally more questions from the number domain were in the group that increased and far fewer geometric shapes and measures questions.

Trends in mathematics content and cognitive domains

The mathematics assessment in TIMSS is organised around two dimensions, a content dimension and a cognitive dimension, as described in the 'TIMSS 2007 assessment frameworks' (Mullis, Martin, Ruddock, O’Sullivan, Arora, and Erberber, 2005). The content dimension comprises three content domains that describe the subject matter to be assessed:

• number;
• geometric shapes and measures; and
• data display.
  

The three content domains can be mapped onto the three strands of the current New Zealand Mathematics and Statistics curriculum, Number and Algebra, Geometry and Measurement, and Statistics, which themselves are combinations of the five strands of the previous Mathematics curriculum (Number, Algebra, Measurement, Geometry, and Statistics).

The cognitive dimension comprises three cognitive domains that describe the thinking processes that students must use as they engage with the content:

• knowing;
• applying; and
• reasoning.
  

TIMSS assessment questions were categorised by the content and cognitive domains, and content and cognitive achievement scales were constructed separately for each domain. In order to simplify comparisons across domains, the scales were constructed to have the same average difficulty (set at 500 scale score points). As well as looking at achievement in each of these domains, the results can then be used to ascertain relative strengths for participating countries.

As Table 4 shows, New Zealand Year 5 students achieved relatively better at data display questions and relatively worse at number questions in 2006. Although the 2002 domains of number and patterns and relationships have been combined in 2006, and likewise the 2002 domains of measurement and geometry have been combined, the trend is very similar. In 2002, the data domain (522) was relatively higher than the other domains and the number (475) and patterns and relationships (495) domains were relatively lower than the other domains.

In the cognitive domains, New Zealand Year 5 students achieved relatively better at tasks that required them to use their reasoning and relatively worse at questions that required demonstrating their knowledge in 2006. In 2002, New Zealand year 5 students showed a relative strength in the reasoning domain (503) and a relative weakness in the applying domain (486).

Table 4: Year 5 mean mathematics scores on the content and cognitive domains in 2006

Content domain	Mean domain score	Cognitive domain	Mean domain score
Number	478 (2.7)	Knowing	482 (2.5)
Geometric shapes and measures	502 (2.3)	Applying	495 (2.3)
Data display	513 (2.6)	Reasoning	503 (2.8)

Note: Standard errors are presented in parentheses.

Table 5 shows the number of test questions (and the associated raw score points) in each of the content and cognitive domains. As can be seen from the table, score points were not evenly distributed across domains. This distribution of questions across domains reflects the content and cognitive emphasis of many of the curricula of participating countries.

Looking at Tables 4 and 5 together, it is important to note that the content domain where New Zealand Year 5 students show the greatest strength, data display, had the least number of questions. Similarly, the cognitive area of greatest strength, reasoning, had the least number of questions. The distribution of mathematics questions across the content domains was similar in 2006 to 2002, with a slight increase in data display questions (data in 2002) and corresponding decrease in number questions (number and patterns and relationships in 2002).

Table 5: Number of questions in each of the content and cognitive domains

Content domain	Total number of questions	Total number of score points	Cognitive domain	Total number of questions	Total number of score points
Number	93	98	Knowing	69	73
Geometric shapes and measures	60	65	Applying	70	75
Data display	26	29	Reasoning	40	44
Total	179	192	Total	179	192

Note: In scoring the tests, correct answers to most questions were awarded one point. However, responses to some constructed-response questions were evaluated for partial credit with a fully correct answer awarded two points. Thus, the number of score points exceeds the number of questions in the test.