What do international rankings tell us about the performance of New Zealand universities?

3. Analysis of international rankings

3.1 ARWU rankings⁷

Table 3 presents the New Zealand universities’ performance in the 2009 ARWU rankings. The first seven columns represent a relative score out of 100, with 100 representing the performance by the top university in that measure. The final column is the overall ranking of the university. Note that the individual rankings for institutions outside the top 100 are not published. Instead, universities are placed in broad ranges of rankings in alphabetical order. The actual ranking of the New Zealand universities which are all outside the top 100 has been determined by the author from the underlying data.

The University of Auckland achieved the highest ranking (214), followed by the University of Otago (279) and the University of Canterbury (422).⁸ The world’s top ranked university was Harvard University from the United States.

The overall score used to determine the rankings indicates that the performance of the top ranked New Zealand university, Auckland, was 15.9 percent that of Harvard University. To put this in perspective, the distribution of the overall scores of the Top 500 universities in 2009 are presented in Figure 1 with the location of the New Zealand universities marked on the graph. It is clear that the majority of universities in the Top 500 (including the five New Zealand universities) are located towards the lower end of overall relative scores. Harvard, the top performing university, is well ahead of the second placed university, Stanford.

Therefore, although the relative score of the New Zealand universities is well below that of the top-placed university, they are much closer to the relative performance of the majority of universities in the Top 500 than is indicated by raw relative overall scores.

Figure 1: Distribution of overall scores in the 2009 ARWU Top 500 rankings

The relative scores for each individual component of the rankings can identify areas where the New Zealand universities perform relatively well and not so well. The data in Table 3 shows that the strongest area of relative performance is in the number of indexed publications. The average relative score in this measure is 28.5, which compares with the overall average of 37.2 for the Top 500 universities. The weakest relative performance is in the area of awards to faculty. No current staff member of a New Zealand university has been awarded a Nobel or Field prize.

One of the individual component scores, per capita performance, can be used to show the impact the size of a university can have on the ARWU rankings. This measure adjusts the other five measures by the number of academic staff at a university and presents a figure more in line with traditional reporting standards used in New Zealand and a more conventional performance indicator. For example, the Performance-Based Research Fund (PBRF) Quality Evaluation results are presented on a per full-time equivalent staff basis, not a total score basis.

Table 4 compares the rankings of the five New Zealand universities in the Top 500 in 2009 using the overall measure and the per capita measure. On average, there is an improvement of 80 places for the New Zealand universities if the per capita measure is used to rank institutions, with the University of Otago now the top-ranked New Zealand university and the University of Auckland now in second place.⁹ The biggest movement is by the University of Canterbury, which improves 154 places. Only the University of Auckland exhibits a decrease in ranking, with a fall of 49 places. So taking into account the number of academic staff at a university presents a quite different picture of the performance of New Zealand universities.

The ranking data for 2009 provides an up-to-date snapshot of university performance, but does not show how New Zealand university performance has been changing over time. Often, the focus when the rankings are released is on how the ranking of an individual university has changed from one year to the next. Table 5 presents the ranking of the New Zealand universities within the Top 500 between 2006 and 2009.¹⁰ We can see that the rankings of the New Zealand universities have remained relatively stable over this time, which is not surprising, as it is unlikely that major change in performance would occur within a short space of time.

The biggest change in ranking occurred at the University of Canterbury, which improved 26 places. Of the other universities, Auckland (2 places), Massey (26 places), Victoria (14 places) and Otago (9 places) showed a deterioration in their ranking.¹¹

But the rankings in Table 5 don’t tell the whole story. If closing the gap to the top-performing university is the objective, then arguably, the overall relative score should be the focus of attention, rather than the rankings themselves. Table 6 presents the overall relative score for participating New Zealand universities between 2006 and 2009, while Figure 2 graphs the same data.

With the focus now on performance relative to the top university, the data shows that over time the University of Canterbury has been improving its performance compared to the top university (Harvard). Victoria University of Wellington had a similar score in 2009 to that it achieved in 2006, while the other three universities all had lower overall scores in 2009, compared with 2006. Generally, the size of the change in relative performance by New Zealand universities was relatively modest, once again reflecting the difficulty in achieving significant change within a short time-span.

The analysis of overall scores can also show how the ranking of a university may improve while its performance relative to the top university may decline. For example, the University of Auckland achieved a ranking of 216 with a score of 16.1 in 2006, but was ranked higher in 2009 at 214 with a lower overall score of 15.9. In other words, the relative performance of the University of Auckland compared with Harvard University fell, but not as much as some of the universities that were previously ranked higher than Auckland. Depending on what is the focus of measurement, one could justly claim that Auckland’s performance either improved or deteriorated over the period.

Figure 2: Overall relative scores of New Zealand universities in the 2006-2009 ARWU Top 500 rankings

Comparing the overall score and ranking can also show that it is easier for universities with lower overall scores to move up and down the rankings. For example, Massey University and the University of Auckland saw their overall relative score decrease by a similar amount between 2008 and 2009 (0.9 points for Auckland and 0.8 points for Massey), but Auckland’s ranking deteriorated by 12 places only while Massey’s deteriorated by 38 places.

To show the key drivers of change in the rankings over time, Table 7 presents the average component scores for the five New Zealand universities in the Top 500 between 2006 and 2009. This data is graphed in Figure 3.

The data shows that the relative scores of the number of indexed publications, highly cited researchers and per capita performance of the New Zealand universities improved on average between 2006 and 2009. However, this has been offset by a decline in the Alumni score and Nature & Science (N&S) score over this time.

Figure 3: Average relative score of New Zealand universities in the 2006-2009 ARWU Top 500 rankings by component

Another way of comparing performance to that of the top-placed university is to examine the trends in per capita score over time. The relative overall per capita score is presented in Table 8 and graphed in Figure 4. The biggest improvement in the relative per capita score between 2006 and 2009 was achieved by the University of Canterbury (8.3 percent), while the smallest increase was shown by the University of Otago (0.4 percent).¹²

What becomes clearer, once you take size of institution into account, is that the relative performance of all New Zealand universities has improved between 2006 and 2009, compared with the top-placed university (California Institute of Technology - Caltech). This compares with the results in Table 6, where just one of the New Zealand universities exhibited an improvement in relative overall score over the same period.

The impact of size on performance can also be illustrated by a hypothetical merger of the five New Zealand universities in the 2009 ARWU rankings. Without any change in individual university performance, this action would result in a single university being ranked around 14^th in the world.

Figure 4: Per capita relative score of New Zealand universities in the 2006-2009 ARWU Top 500 rankings

The question of what benchmark New Zealand universities should be measured against deserves further discussion. Whether it is prudent to benchmark to Harvard’s or Caltech’s performance is moot, given the massive resources at the disposal of these American universities. Arguably, a better benchmark might be against the Australian universities, in particular the Group of Eight (G8).¹³

The G8 comprises eight large metropolitan universities that are research intensive in nature and are commonly the subject of benchmarking for New Zealand universities.¹⁴ For the purposes of this analysis, one of the G8 universities, Australian National University (ANU), is excluded from the analysis. ANU is predominantly focused on postgraduate-level teaching and research, so comparing the performance of ANU with New Zealand universities is arguably not appropriate. This leaves the other seven universities in the group – called the ‘G7’ – as the benchmark.

Figure 5 presents the overall scores of the New Zealand universities, relative to the Australian G7 average overall score between 2006 and 2009. The University of Auckland is closest to the G7 average in terms of performance with an overall relative score of 76 in 2009, followed by the University of Otago (64). The other three universities in Figure 5 display relatively similar performance at just under half the G7 average in 2009.

Although the period between 2006 and 2009 is relatively short, the data in Figure 5 suggests that the University of Auckland, University of Otago, Massey University and Victoria University of Wellington have been losing ground to the G7 universities, while the University of Canterbury exhibited improved relative performance.

Figure 5: Overall relative scores of New Zealand universities in the 2006-2009 ARWU Top 500 rankings compared with the Australian G7 average score

Figure 6 compares the relative performance of New Zealand universities with the G7 using the per capita measure. Once again, by adjusting for the size of the academic workforce the relative performance of New Zealand universities improves. In 2009, the University of Otago shows a per capita relative score (98) around the G7 average. Although the relative performance of the University of Auckland declines slightly when using the per capita measure, the remaining three universities in Figure 6 all improve their performance to the G7 average.

Between 2006 and 2009, the University of Canterbury once again shows an improvement in performance relative to the G7 average using per capita relative score. Victoria University of Wellington improved each year until exhibiting a drop in relative performance in 2009, as did the University of Auckland. The per capita data for the University of Otago fluctuates over time, while Massey University exhibited a similar relative score in 2009 as it achieved in 2006. Overall, when you take account of institutional size the data would suggest that New Zealand universities are, on average, maintaining their relative position with the G7 universities.

Figure 6: Per capita relative score of New Zealand universities in the 2006-2009 ARWU Top 500 rankings compared with the Australian G7 average per capita score

The ARWU rankings also consider information on the gross domestic product (GDP) and population of countries in the Top 500. They use this to measure country performance by comparing measures such as the share of Top 500 universities with the share of world GDP. This can help to adjust the results for the fact that more wealthy nations can support their higher education systems to a greater degree through either greater government funding or private endowments. Figure 7 shows the ratio of the percentage share of universities in the Top 500 per percentage share of the total GDP for each of the countries with universities in the Top 500.

New Zealand is the top-performing country in terms of this ratio – our share of Top 500 universities was over 4 times New Zealand’s share of total GDP. So once the size of an economy is taken into account, the New Zealand universities perform very well in the ARWU.

Figure 7: Ratio of the percentage of universities in the 2009 ARWU Top 500 per percentage share of total GDP by country

However, there is still an issue with relative performance. For example, a country could have 10 universities in the Top 500, but this would say nothing about their relative position. They could be ranked between 491 and 500. To control for this, we sum the points used to determine the overall ranking at the country level and compare it to that country’s share of total GDP.

Figure 8 presents this data. Once the relative position of New Zealand’s universities is taken into account, we slip from number 1 to number 2 in the rankings, reflecting the fact that none of our universities are in the ‘top tier’ in the raw rankings. Nevertheless, this still represents strong performance by the country’s universities.

Figure 8: Ratio of the percentage of overall summed scores in the 2009 ARWU Top 500 per percentage share of total GDP by country

Although New Zealand performs well when adjusting for the size of the economy, a somewhat different picture emerges if performance is adjusted for the size of the population. Figure 9 compares country performance in the ARWU Top 500 when adjusted by share of GDP with country performance adjusted by share of population.

Broadly, the results show a reasonable degree of correlation between country performance adjusted for share of GDP and country performance adjusted for share of population. However, New Zealand, along with Israel, appear to be significant outliers compared to the other countries in the Top 500. Although both countries perform very well in the GDP adjusted measure, they are lower performers when taking into account population size. For example, in the share of universities in the Top 500 metric, New Zealand is ranked first on the GDP adjusted measure (with a ratio of 4.4) but ranked just eighth on the population adjusted measure (with a ratio of 6.3).

3.2 THES rankings¹⁵

The results of the 2009 THES rankings are presented in Table 9. The top New Zealand university in terms of overall rankings is Auckland (61) followed by Otago (125) and Canterbury (188).¹⁶ The top university overall, as was the case in the ARWU rankings, is Harvard University.

It is clear from these results that the ranked New Zealand universities perform better in the THES than in the ARWU rankings. But, what is the source of this better performance? The University of Auckland’s top ranking is based on strong performance in the academic and employer peer review measure. Not surprisingly, given its per capita performance in the ARWU rankings, the University of Otago is the top-ranked New Zealand university in terms of the research measure, cites/FTE. It also performs particularly strongly in terms of international faculty, as do all of the New Zealand universities. Ranked New Zealand universities do less well in the number of students per academic staff member.

As noted earlier, the THES rankings are biased in favour of English-speaking countries, especially those countries that promote export education. Alongside the concerns raised about the survey methods used by THES, this means that the performance of New Zealand universities in these measures should be viewed with caution.

The relative performance of the three New Zealand universities in the THES Top 200 is shown in Figure 10, which shows the distribution of the Top 200 universities by the overall relative score. One notable feature is that the distribution is less skewed than the ARWU, which reflects the approach of THES in using z scores to calculate relative scores, reducing dispersion.

Figure 10: Distribution of overall scores in the 2009 THES Top 200 rankings

Table 10 presents THES rankings of the New Zealand universities over time. Because of changes to methodology, only the 2007 to 2009 rankings can reliably be compared.¹⁷ As one might expect over such a short a time frame, the rankings have been reasonably stable for New Zealand universities. The biggest change is exhibited by Massey University, which dropped 57 places between 2007 and 2009.¹⁸ Smaller drops were exhibited by the University of Auckland and the University of Otago, while the University of Waikato and Victoria University of Wellington improved their ranking slightly.

Table 11 presents the overall score used to determine the rankings of the three New Zealand universities in the THES Top 200 between 2007 and 2009. This data is graphed in Figure 11. It suggests that, compared to the top-performing university (Harvard), the performance of all three New Zealand universities has declined. Note that the University of Canterbury maintained the same ranking in 2009 as it achieved in 2007, despite a slight drop in overall relative score.

Figure 11: Overall relative score of New Zealand universities in the 2007-2009 THES Top 200 rankings

The average performance of the three universities in the Top 200 in each of the components in the THES between 2007 and 2009 is presented in Figure 12. This shows that the main areas of improvement have been better employer peer review and an increase in the proportion of international faculty, while the number of citations per FTE academic staff member has declined significantly. However, a change to the way that the citations were extracted from 2008 was likely to have been detrimental to the relative performance of the New Zealand universities, so, how much of this change is real rather than a result of the methodological change is hard to determine. The improvement in the number of EFTS per FTE academic staff between 2008 and 2009 is a result of the drop in international students at New Zealand universities. However, this has had the perverse effect of causing the performance in the international student measure to drop during this period.

Figure 12: Average relative score of New Zealand universities in the 2007-2009 THES Top 200 rankings by component

Note: The average is between the University of Auckland, University of Otago and the University of Canterbury.

The performance of three New Zealand universities to the G7 average is presented in Figure 13. In 2009, the University of Auckland (96) achieved a relative score very similar to the G7 average, with the Universities of Otago (82) and Canterbury (71) displaying lesser relative performance.

The data also shows that the ranked New Zealand universities have generally maintained their position relative to the G7 average between 2007 and 2009, compared to the decline in performance against the top university shown in Figure 11.

Figure 13: Relative scores of the ranked New Zealand universities in the 2007-2009 THES Top 200 compared with the G7 average score

In Figure 14, the performance of each country in the THES is examined. For each country this shows the ratio of the percentage of universities in the Top 200 to the percentage share of total GDP. New Zealand is the top country using this measure by some margin. The share of New Zealand universities in the Top 200 was more than 7 times New Zealand’s share of total GDP.

Figure 14: Ratio of the percentage of universities in the 2009 THES Top 200 per percentage share of total GDP by country

The relative ranking of the universities is not taken into account by the analysis in Figure 14. Figure 15 does take this into account, by measuring the ratio of the percentage share of the Top 200 overall points by universities in a particular country to that country’s percentage share of world GDP. The data shows that the share of points in the Top 200 by the New Zealand universities is 7 times New Zealand’s share of GDP. Once again, the New Zealand universities perform well. However, given the concerns about the methodology used to generate the THES rankings, this does not necessarily mean that any satisfaction can be taken from this good performance.

Figure 15: Ratio of the percentage of overall summed scores in the 2009 THES Top 200 per percentage share of total GDP by country

As was the case in section 3.1, the country results can also be examined by adjusting for share of population rather than share of GDP. Figure 16 presents country performance in the THES Top 200 when adjusted by share of GDP compared with country performance adjusted by share of population.

The results show that once again, New Zealand shifts from being an excellent performer when adjusting for share of GDP, to being more in the middle of the pack when adjusting for share of population. In the percentage share of universities in the Top 200 metric, New Zealand is ranked first (with a ratio of 7.5) when adjusting for share of GDP, but falls to a rank of fifth equal (with a ratio of 8.5) when adjusting for share of population.

Figure 16: Ratio of country performance per share of population and GDP in the 2009 THES Top 200 by measure of performance

Footnotes