How university rankings work

University league tables can be seen as a necessary evil that may or may not help students make informed choices, writes Stephen Matchett.

League tables that rank universities are proliferating the world over and prospective students are increasingly turning to them to inform their decision about where to study.

The creators of the various rankings make extravagant claims about what their league table achieves, while the opponents of league tables – and there are many – express outrage at the methodologies used, and even the idea of holding universities to account in this way.

But the market needs information on universities, since they are generally opaque when it comes to comparing their institution’s performance against their peers. That means rankings will not go away.

While arguing they do more harm than good, the European Universities Association accepts this market reality, “because of their simplicity and consumertype information.”

But how useful is the information provided in various rankings?

For a start, most rankings measure research, with a bias towards natural science and medicine, rather than the quality of undergraduate or postgraduate courses or graduate employment.

So Professor Andrejs Rauhvargers, secretary general of the Latvian Rectors’ Conference, drilled down into the methodologies used in many of the rankings to explain how they work.

It’s up to you to work out whether that information helps you to choose where to study.

ACADEMIC RANKING OF WORLD UNIVERSITIES

* Assesses only 1000 research universities worldwide and publicly ranks just 500.

* Uses Nobel Prize winners in science or Fields medallists in maths as a proxy for quality of education.

* Research in the hard sciences, technology, engineering and maths is emphasised, leaving social sciences, arts and humanities pretty well ignored.

THE -THOMSON REUTERS 2010 WORLD UNIVERSITY RANKING

* Ranking indicators include teaching (30 per cent), based mainly on reputation, survey, research volume, income, reputation (30 per cent), and research influence, measured by citations – or how many times a research paper is mentioned in other research papers.

* Research output as measured by number of staff , publications and so on.

QS – WORD’S BEST UNIVERSITIES RANKING

* This is the old Times Higher Education (THE) – QS rankings (last issued 2009) rebadged with US News and World Report.

* Lists fewer than 1000 of the world’s 17,000 universities.

* Measures research and teaching quality plus graduate employability and “international outlook”.

* Marks teaching via student/staff ratio.

* Uses international staff and students as proportions of total strengths as proxies for international outlook.

* Citations per faculty and peer-review measure research, with peer review being based on responses to an internet survey. Graduate employability also relies on an online survey.

WEBOMETRICS RANKING OF WORLD UNIVERSITIES

* Produced by a subsidiary of Spain’s National Research Council.

* Assumes web presence is a proxy for quality and counts number of pages on university academic websites plus published papers retrieved from Google Scholar and rich files (pdf and equivalent) published.

There are dozens of others, including an entire battalion of rankings for business schools around the world.

Rauhvargers also reports on bibliometric measures which list university research output.

And he mentions the EU’s own in-development Multirank system, which is intended to include the US, Asia, Europe and Australia and assess overall university performance plus research.

There is no doubting he detests both the way league tables are used and how they are compiled.

“Rankings, it is claimed, make universities more transparent,” he writes. However, methodologies of the existing rankings, and especially those of the most popular league tables, still lack transparency themselves. It is difficult, if not impossible, to follow the calculations made from raw data and, from there, to the overall score, just by using publicly available information,” he writes.

But there is no point in denouncing league tables without offering an alternative.

Perhaps the EU’s Multirank might become a global gold standard – but you have to wonder whether anything authored by European officialdom will ever be much use to anybody who does not speak fluent bureaucrat.

Can a league table of university league tables be far away?