Over the past several months, my office [Vice President for Research and Enterprise] has been inundated with requests for data. These requests have come from government and government agencies, and from commercial ranking and profiling organisations. At the same time, since the beginning of 2010, the office hosted five governmental and funder audits totaling 14 days. These sought to scrutinise travel and hospitality costs, student accounts, EU and national research projects. Some have looked to assess expenditure going back over five years. The question is: do we know what we are counting, and whether the data being collected is interpreted appropriately or counting what is important?

The drive for greater transparency and accountability is not new – although it is arguable that globalisation and global rankings combined with the effects of the global financial crisis have re-energised the emphasis on and importance of value-for-money criteria. ‘Transparency instruments’ have become the new policy catchphrase.

Performance outputs are being aligned to resource allocation. Assessment of scientific-scholarly research is a vital tool to help drive up productivity and quality, and assess return-on-investment, especially for publicly funded research. Global rankings – an inevitable response to globalization and global mobility – have helped spur the proliferation of worldwide comparisons of higher education performance and national competitiveness.

Australia’s Kim Carr (2009) opined that ‘[I]t isn’t enough to just go around telling ourselves how good we are – we need to measure ourselves objectively against the world’s best”. I’ve used a similar sound bite: ‘less self-declaration and more external verification’.

But the history of rankings shows that measuring the wrong things can produce distortions and perverse actions. In many instances, governments have directly adopted or ‘folded-in’ the indicators into their own performance measurements or used rankings to set targets for system restructuring. The quantification of performance has become a powerful tool because as Ehrenberg (2001) said, they give the ‘appearance of scientific objectivity’.  Indicators are often chosen and decisions made without fully understanding the methodological shortcomings or the limitations of the data. And, because rankings act like other performance instruments and incentivize behaviour, governments risk perverting public policy imperatives to meet criteria set by the vagaries of ranking or other profiling organisations.

To paraphrase Einstein, are we measuring what counts or counting what can be measured? Three examples:

  1. The education level of entering students is generally considered a good proxy for student ability on the basic assumption that a roughly similar range of performance can be expected throughout their higher education career. This forms the basis by which many HE systems and institutions select students. But do entry scores and standardized testing simply reflect socio-economic advantage? ‘Many colleges recruit great students and then graduate great students [but is] that because of the institution, or the students?’ (Hawkins, 2008).
  2. One of the most noticeable changes in how higher education is funded is the shift from inputs to outputs. Financing the number of students who actually complete and graduate within the determined time-frame is seen as a good measure of quality. But measuring graduation rates may be disadvantageous to lower socio-economic and ethnically disadvantaged groups or mature students whose life or family circumstances disturb ‘normal’ study patterns. It may undermine institutions which are working hard to widen participation, and dis-incentivize access/2-year programmes because students often transfer to other universities which then get the credit for student completion.
  3. Counting peer publications and citations has become the most common way to measure institutional/individual academic research productivity and quality. However, by relying on these methodologies, we undermine that which policy wants to encourage: research beyond the academy, e.g. contributions to technical standards or policy, commercialization and innovation, and other forms of ‘Mode 2’ engagement.

Cross-national comparisons can improve strategic decision making, and help identify and share best practices. But national context resists attempts to make simple and easy comparisons. The new QS and THE rankings and profiling information, for example, each measure academic staff and students but with completely different data sets, which is not just confusing for the institutional researcher but potentially disastrous for those making decisions based upon them.

The drive towards metrics-based tools appears to be a ‘cheap and cheerful’ transparency instrument, but it encourages simplistic solutions, skewing agendas and policies to become what is measured. Ultimately the public policy imperative is lost in the belief that quantification equals quality. Policy-making by numbers is not the solution many governments think it is.

 

Rankings and the Battle for World-Class Excellence: How Rankings are Reshaping Higher Education will be published by Palgrave MacMillan, March 2011.