Death by Metrics

The issue of metrics has come to the fore recently following the refusal by German sociologist, to forward data to the Centre for Higher Education Development (a Think Tank, seeking to, “develop models for the modernisation of higher education systems and institutions”) to be used in the ranking of universities and subjects. Since then other subject areas and universities in Germany have followed suit. I shall return to the significance of this moment of resistance.

Most academics have direct experience of metrics used to assess performance and their use has grown apace since the first introduction of management practices and accountability regimes into universities in the 1980s. Metric data are constructed and gathered on almost all activities conducted by academics - across teaching and research - and they are aggregated at different levels, to assess individuals, departments and whole institutions.

In some contexts, they are an instrument of performance management of individuals and departments, as university governance moves from a collegial to a managerial system. At the same time, however, metric data are used to assess the performance of managers, themselves, and the ‘functions’ under their remit, and also their institutions in national and international rankings. Similarly, editors of journals are concerned about the impact factor of their journal and its standing against comparable journals, which, in turn, re-enters local decision-making in recommendations about individual performance (for example, concerning the standing of their publications) and in terms of journals that should be targeted by a department to maximise its standing in research evaluation. In many places, these judgements affect pay and working conditions as contracts separating teaching from research, or varying the ratio between them, become more frequent.

There are few national systems of higher education that are immune from their effects, though their use is more extensive and systematic in some places, rather than others. They seem to have gone furthest in national systems with a high proportion of public universities, especially in countries with strong neo-liberal public policy regimes – for example, the UK, Australia and New Zealand. They are less extensive where national systems of governance are weaker – for example, the United States and Germany.

My purpose is not to describe the myriad forms of audit-by-metrics, or the national differences in ‘metric regimes’. Rather, I shall draw out two aspects in the development of ‘metric regimes’, which have particular significance for the discipline of sociology. The first is how the form of metric measurement favours particular disciplines over others. The second is the move from ‘co-production’ to ‘commercialisation’ in the construction of metric regimes.

One of the claims made on behalf of metric data is that they provide objective standards. Unsurprisingly, most of their proponents are sophisticated in the claims that they make and most of them – the ‘H-index’, for example - come with qualifications; for example, that cross-subject comparisons are problematic compared with within-subject comparisons; or that rank orders may be problematic when constructed on indicators that show minor differences; or that indicators of student satisfaction may vary by the characteristics of the students, rather than the intrinsic qualities of courses of instruction. All metrics can be reconstructed as data for rank orders, including composite rankings, which compound the errors of the components of which they are constructed. Nonetheless, the first ‘law’ of audit-by-metrics is that claims made about what they indicate always extend the basis for any claims. This is because the function of rank orders is to give reputational advantage to those with a higher position, who then claim that advantage, while decrying other metrics on which they perform less well. This is one way in which academics become complicit in their audit regimes.

However, in any situation where decisions are being made using metric data, how a subject discipline performs in relation to that data will be a critical issue for its potential fate. On the one hand, subject disciplines evolve according to an ‘inner logic’ relating to an audience of subject specialists, but they also develop as an adaptation to external audiences, including policy-makers, etc, some of which are keen to shape universities in the light of metric data.

There have been few studies of disciplines on this basis, but something can be pieced together from the work of Richard Whitley in the 1980s. He developed a typology of disciplinary organisation in terms of high or low task uncertainty (relating to the agreement or otherwise on methods to be used) and strategic uncertainty (relating to agreement or otherwise on problems deemed important for study), and high or low functional and strategic dependence (relating to what we might call the tight, or loose ‘coupling’ among researchers and their problems).

The typology contains an implicit continuum from high technical and strategic uncertainty combined with low functional and strategic dependence, to low technical and strategic task uncertainty combined with high functional and strategic dependence. These two ideal types at either end of the continuum – what Whitley calls ‘fragmented adhocracy’ and ‘conceptually integrated bureaucracy’ - are exemplified by post-1960s sociology (in the US and the UK) and post-1945 physics. Economics, among the social sciences, is closer to physics, as a ‘partitioned bureaucracy’.

What is evident is that the organisation of disciplines maps onto metric performance. So, H-indices follow the continuum set out by Whitley, with STEM subjects performing ‘better’ than social sciences, and internal variation within the social sciences following a similar pattern. Geographers (which includes physical geography) score an average of 5.04, with economists next at 4.83, political scientists at 2.46, and sociologists at 2.38. Journal Impact Factor scores show a similar distribution. Here, the more tightly coupled a subject discipline, the more likely are the citations in its articles to be to other articles published in the last two years. Given that a journal’s IF score is a function of citations to articles within two years, then sociology journals score ‘worse’ than, say, economics journals. Newly developing policies for open access will have different consequences for different disciplines precisely because of these characteristics.

The point of these two examples is to suggest that the ‘objectivity’ of the metric measures favour some disciplines over other disciplines, in terms of how they are represented within the various fora – university-wide, or national and international – in which research policy and funding decisions are made.

I now want to turn to the other issue of the shift from co-production to commercialisation in the construction of metrics. For the most part, commentators on the expansion of audit into the academy have seen it as a neo-liberal policy in which metric measures function as market proxies. As such, the measures are frequently generated as a consequence of consultation and by processes of co-optation at both university and system-wide levels. For example, the REF panels in the UK involve consultation with subject associations about how they should be conducted and subject specialists are nominated to conduct the evaluation and produce the REF metric data.

British sociologist are already heavily implicated in the very ranking system that German sociologists rejected. But, heartening though that rejection is, it is effective in the context of a weak national policy regime. Where a national policy regime is strong, co-produced metric data can be readily supplanted by commercialised data. It was already a feature of the Science (and Social Sciences) Citation Index when set up in the 1970s that it was designed as an income-generating facility that might be used by Deans and College Presidents. Those who wished to use it had to pay.

As the generation of metric data proceeds apace, it is no longer necessary to ‘co-produce’ it, or at least, co-production no longer needs active engagement, because the operation of the internet, data mining, etc provides the information necessary, and commercial organisations are willing to organise the data and make it available. In other words, British research evaluation can, in principle, take place without the willing participation of academics, providing that the will by government is there to enforce it through funding agencies.

Moreover, at the cutting edge of neo-liberal public policy, universities are no longer being subject to governance by market proxies, they are being marketised directly and metric data is part of the process by which the market in higher education is brought into being. Thus, the centralised university application system in Britain requires ‘Key Information Sets’ - metric data on student satisfaction, graduate employment levels, income six months after graduation – on each undergraduate degree programme to facilitate ‘informed’ consumer choice.

In this context, open access to publications and open data are not about the creation of a global commons, but about creating opportunities for a new ‘enclosure’ movement in which publications and data can be exploited for commercial purposes. Moreover, the neo-liberal, marketised university is increasingly becoming a mono-culture, which may threaten the survival of sociology, just in so far as it becomes generalised.

This might be the point to return to Whitley’s characterisation of a ‘fragmented adhocracy’. This is its tendency to produce, “diffuse, discursive knowledge of commonsense objects” (1984: 158). Where this discursive knowledge is aligned with the understandings of elite publics, no particular problem of credibility arises. However, where a discipline has an aspiration to engage with less powerfully placed publics, then a different issue of credibility arises, precisely that of our credibility because we represent a challenge to the certainties of neo-liberal orthodoxies and are witnesses to the consequences of the widening social inequalities with which they are associated.

John Holmwood, University of Nottingham, UK and President of the British Sociological Association
for Global Express, October 30, 2013