What do international rankings tell us about the performance of New Zealand universities?

2.1 How the rankings are determined

ARWU ranking

The first ARWU rankings were published in 2003 and were developed initially as a benchmarking exercise for the Shanghai Jiao Tong University. In its current configuration, the 2009 ARWU rankings use six measures of performance to generate an overall ranking for each university. These measures are defined in Table 1.

There are two key points to note about the measures used in the ARWU rankings. First, the ARWU indicators measure the research performance of a university, with no indicators of teaching performance. Second, five of the six measures are totals of either people or research outputs. Only one indicator is calculated on a per academic staff member basis. This means that the AWRU is, to a certain extent, a measure of volume of research, with larger institutions at an advantage. How this impacts on the rankings of the New Zealand universities is explored later in this report.

To generate a final ranking, the performance of each university in each measure is expressed as a percentage of the top-performing university. Then, the weightings shown in Table 1 are applied to these relative measures and a total score calculated. To obtain the final ranking, each university score is then expressed as a percentage of the score achieved by the top university. For example, a score of 80 means that the overall weighted performance of that university was 80 percent that of the top performing university.

THES ranking

The first THES rankings were published in 2004 and were designed to inform readers of the THES about the comparable performance of the world’s universities through measuring a number of dimensions of university performance. Up to and including 2009, the THES rankings were compiled by QS Quacquarelli Symonds Ltd.

The measures used to compile the 2009 THES rankings include: survey results from academics and employers that captures the perceptions of the quality of a university, measures of class size, research impact and the proportions of international faculty and students at an institution (see Table 2). A key point to note is that the THES is heavily reliant on surveys about the perceived quality of universities, with the academic and employer survey contributing 50 percent of the weighted performance score.

From 2007, the THES rankings have been calculated in a way that reduces the dispersion in performance in the various measures, reducing the impact of outliers. This process involves the use of z scores⁴ to calculate the relative performance of universities compared to the top performing university. The weightings in Table 2 are then applied to the score in each measure to arrive at a relative score, which is then expressed as a percentage of the top performing university. This overall score is then used to determine the rankings.

2.2 Critiques of the rankings

Both the ARWU and THES rankings have come in for criticism. A key criticism of both ranking systems is that they both aggregate the component scores to create a final overall ranking. This aggregation is regarded as flawed (Brooks 2005). It is argued that universities may have different areas of specialisation, so a better way of applying rankings is to compare each university against other universities on each specialisation. Also, the generation of a final ranking measure requires the use of arbitrary weightings, the selection of which introduces subjectivity into the rankings process (Usher and Savino 2006). In this regard, Usher and Savino praise the approach of the Centre for Higher Education Development rankings which does not seek to arrive at a final single ranking.

Another criticism is that higher education institutions based in English-speaking countries have an advantage, given that the bibliometric measures used in both the THES and ARWU are sourced mainly from English-speaking journals (Marginson 2007). In the case of the THES rankings, the citation data used to measure research performance is not normalised for different subject disciplines. There are vastly different rates of citation across subject disciplines, so universities with a focus on science and/or the presence of a medical school will fare better in this measure.⁵

Additional criticisms that apply to each of the individual rankings are examined below.

THES rankings

Marginson (2007) argues that the large weighting applied to survey responses (50 percent) captures how a university is perceived, not how they actually perform. The result is that universities are being judged on credentialism and not actual learning and research outcomes. In addition, he argues that there is a lack of transparency in these measures and a problem with regional bias. Marginson argues that universities in the UK and USA will have better name recognition. This is compounded by a low response rate (around 1 percent) for the academic peer review measure.

Marginson has also criticised the use of the number of students to academic staff as a proxy for teaching quality, with a lower ratio supposedly reflecting higher quality. Marginson argues that teaching quality cannot be accurately assessed using a resource quantity indicator such as this.

Although the ranking authors suggest that the proportion of international students reflects the prestige with which an institution is viewed, Marginson argues it simply rewards volume building. This measure will also favour universities in English speaking countries and those in countries where there is a centrally mandated export education policy.

The THES rankings are being substantially revised for 2010, and a new analytical contractor has been appointed by THES. The THES rankings will reappear in a quite different form from 2010.⁶

ARWU rankings

As mentioned previously, the ARWU rankings focus solely on research-based measures. Therefore, this ranking system does not present a picture of university performance across multiple dimensions. Also, institutions of larger size will be at an advantage as four of the five measures (comprising 90 percent of the final score) use totals of researchers or research outputs in calculating relative performance.

Of the specific measures used in the ARWU rankings, the inclusion of Nobel prize winners is controversial as it is largely science-based and can be subject to politicking (Marginson 2007). It also means that institutions earn points from people who may have long since ceased to be associated with that institution through a halo effect. This historic nature also applies to the HiCite measure, which examines citations of individuals between 1981 and 1999 (Holmes 2006).

Conclusion

It is clear that neither the ARWU nor the THES rankings are free from criticism. Nevertheless, a number of analyses of the THES and ARWU rankings argue that the ARWU rankings are a better indicator of institutional excellence (see Taylor and Braddock 2007; Marginson 2007). However, it should be remembered that the ARWU focuses solely on research performance and does not provide any information on the teaching activities of institutions. Therefore, a good understanding of the coverage and limitations of these systems is crucial to interpreting institutional performance.

Footnotes

4. A z score indicates how many standard deviations an observation is above or below the mean value.
5. The authors of the ARWU ranking have responded to criticism of the bias of bibliometric databases by applying a weighting of 2 to papers published in the social sciences index.
6. See http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycode=408980. [accessed 8 March 2010]