Ellen Hazelkorn takes a look at the accuracy of university rankings from an international perspective.

Global rankings have decisively shifted the nature of the conversation around higher education to  emphasise universities’ performance in knowledge economies. How did that happen and why are rankings becoming increasingly important?

The emergence of global rankings coincided with the acceleration of globalisation at the turn of the millennium. This is because higher education is a global game. Its success (or failure) is integral to and a powerful indicator of the knowledge-producing and talent-attracting capacity of nations. But, the landscape in which HE operates today has become extremely complex; there are many more demands and many constituencies which have an impact on and a voice in shaping higher education’s role and purpose. While rankings have been around since the early 1900s, global rankings represented a significant transformation in their range and influence.

The arrival of the Shanghai Academic Rankings of World Rankings (ARWU) in 2003 set off an immediate chain-reaction that speaks to pent-up demand in the political system, and arguably more widely, for more meaningful and internationally comparative measures of quality and performance. In other words, once the number of people participating in and served by higher education expands, so as to begin to comprise and affect the whole of society rather than a narrow elite, then matters of higher education governance and management, and performance and productivity necessarily come to the fore.

Over recent years, rankings have become a significant actor and influencer on the higher education landscape and society more broadly, used around the world by policymakers and decision-makers at government and higher education institution (HEI) level, as well as by students and parents, investors, local communities, the media, and others. There are over 18,000 university-level institutions worldwide. Those ranked within the top-500 would be within the top 3% worldwide. Yet, by a perverse logic, rankings have generated a perception amongst the public, policymakers and stakeholders that only those within the top 20, 50 or 100 are worthy of being called excellent. Rankings are seen to be driving a resource-intensive competition worldwide. But, it is also apparent that putting the spotlight on the top universities, usually referred to as world-class, has limited benefit for society overall or all students. Despite much criticism about their methodology, data sources and the concept of rankings, their significance and influence extends far beyond higher education – and is linked to what rankings tells us (or are perceived to tell us) about national and institutional competitiveness and the changing geopolitical and knowledge landscape. Their persistence is tied to their celebration of “elites” in an environment where pursuit of mobile talent, finance and business are critical success factors for individuals and societies.

How accurate are university rankings, in your opinion? In what sense do indicators vary among different university rankings?

Rankings’ popularity is largely related to their simplicity – but this is also the main source of criticism. The choice of indicators and weightings reflect the priorities or value judgements of the producers. There is no such thing as an objective ranking nor a reason why indicators should be either weighted (or given particular weights) or aggregated. Although rankings purport to measure higher education quality, they focus on a limited set of attributes for which (internationally) comparable data is available. They are handicapped especially by the absence of internationally comparable data for teaching and learning, student and societal engagement, third mission, etc. This means that most global rankings focus unduly on research and reputation.

Many of the issues being measured are important for institutional strategic planning and public policy but annual comparisons are misguided because institutions do not and cannot change significantly from year to year. In addition, many of the indicators or their proxies have at best an indirect relationship to faculty or educational quality and could actually be counter-productive. Their influence derives from the appearance of scientific objectivity.

Rankings are both consistent and inconsistent. Despite a common nomenclature, they appear to differ considerably from each other. “Which university is best” is asked differently depending upon which ranking is asking the question; there are methodological differences as to how similar aspects are counted (e.g. definitions of staff and students or international staff and students). And because much of the data relies on institutional submissions there is room for error and “malpractice”. This presents a problem for users who perceive the final results as directly comparable. Nonetheless, the same institutions tend to appear at or near the top; differences are greatest beyond the top 100 or so places.

Over the years, ranking organisations have become very adept at expanding what has become a profitable business model. Recent years has seen an increasingly close corporate relationship, including mergers, between publishing, data/data analytics and rankings companies.

There is a proliferation of different types of rankings, for different types of institutions, world regions, and aspects of higher education. Most recently, global rankings have sought to focus on teaching, engagement and the UN’s SDGs. Despite these “improvements”, the same problems which afflict earlier formats and methodologies remain. Many of the indicators focus on inputs which are strongly correlated to wealth (e.g. institutional age, tuition fees or endowments/philanthropy), as a proxy for educational quality. Both QS and THE use reputational surveys as a means of assessing how an institution is valued by its faculty peers and key stakeholders. This methodology is widely criticized as overly-subjective, self-referential and self-perpetuating in circumstances where a respondent’s knowledge is limited to that which they already know, and reputation is conflated with quality or institutional age

What is the difference between global and national rankings? What are the strengths and weaknesses of these two different assessments?

Global rankings rely on internationally comparable information. However, there are greater differences in context, data accuracy and reliability, and data definitions. The ranking organisations do not audit the data, and even if they did context remains important.

Despite claims to “compare the world’s top universities” (Quacquarelli Symonds World University Rankings, 2019) or “provide the definitive list of the world’s best universities evaluated across teaching, research, international outlook, reputation and more” (Times Higher Education, 2018), in truth, global rankings measure a very small sub-set of the total 18,000 higher education institutions worldwide.

National rankings in contrast have access to a wider array of data, as well as having the capacity to quality control it – albeit problems still persist. Today, there are national ranking in more than 70 countries; they can be commercial or government rankings. They may be a manifestation of other evaluation processes, such as research assessment, whereby the results are put into a rankings format either by media organisations or the government itself. National rankings are used by students/parents but also by governments to drive policy outcomes such as classification, quality assurance, improving research and driving performance broadly.

It is  claimed that universities learn how to apply resources in exactly the right place and submit data in exactly the right way and are thus able to manipulate rankings. A prominent scandal was the Trinity College Dublin data scandal, where researchers were allegedly instructed in how to answer ranking questionnaires. How easy is it to manipulate such rankings?

Higher education leaders and policymakers both claim that rankings are not a significant factor in decision-making, but few are either unaware of the rank of their universities or that of national or international peers. The increasing hype surrounding the now multi-annual publication of rankings is treated with a mixture of growing alarm, scepticism and, in an increasing number of instances, with meaningful engagement with ranking organisations around the process of collecting the necessary data and responding to the results.

This reaction is wide-spread. There is a strong belief that rankings help maintain and build institutional position and reputation, good students use rankings to shortlist university choice, especially at the postgraduate level, and that governments and stakeholders use rankings to influence their own decisions about funding, sponsorship and employee recruitment.

As such, rankings often form an explicit institutional goal, are incorporated into the strategic objectives implicitly, are used to set actual targets, or are used to measure of achievement or success. In a survey conducted by this author, more than half of international HE leaders said they have taken strategic, organizational, managerial, or faculty action in response to rankings. Only 8% had taken no action. Many universities maintain vigilance about their peer’s performance, nationally and internationally.

In global rankings, the most common approach is to increase the selectivity index – or the proportion of high performing students. It is no coincidence that many of the indicators of quality are associated with student outcomes of employment, salary, etc. – which are in turn are strongly correlated with these characteristics. The effect is to reinforce elite student entry.

The practice of managing student recruitment has received considerable attention in the USA but is not confined to that country. Similar patterns are found elsewhere even where equity and open recruitment is the norm. The practice of urging “supporters” to respond positively to reputational surveys used by many universities.

Given the significance and potential impact of rankings and the fact that high-achieving students – especially international students – are heavily influenced by rankings, these responses are both understandable and rational. However, the really big changes in ranking performance derive from the methodological changes that the ranking organisations themselves engage in. Some of these changes are introduced to improve the rankings. However, one can’t cynically dismiss the view that many of the changes are aimed at generating publicity, and consultancy, for the rankings themselves!

What steps are necessary in order to improve global university rankings?

Rankings are not an appropriate method for assessing or comparing quality, or the basis for making strategic decisions by countries or universities. If a country or institution wishes to improve performance there are alternative methodologies and processes. Beware unintended consequences of simplistic approaches.

These are some suggested Do’s and Don’ts –

Don’t:

• Change your institution’s mission to conform with rankings;
• Use rankings as the only/primary set of indicators to frame goals or assess performance;
• Use rankings to inform policy or resource allocation decisions;
• Manipulate public information and data in order to rise in the rankings.

Do:

• Ensure your university has an appropriate/realistic strategy and performance framework;
• Use rankings only as part of an overall quality assurance, assessment or benchmarking system;
• Be accountable and provide good quality public information about learning outcomes, impact and benefit to students and society;
• Engage in an information campaign to broaden media and public understanding of the limitations of rankings.