German Sociologists Boycott University Rankings

Scientific Evaluation, Yes – CHE Ranking, No

Methodological Problems and Political Implications of the CHE University Ranking

German Sociological Association Statement June 2013 (long version)

The results of the CHE (Centre for Higher Education Development) University Ranking, a subject-level classification covering a range of academic disciplines, have been published each spring since 1998. The ranking has acquired high public visibility by virtue of the fact that it has been published in the weekly newspaper DIE ZEIT and in the annual ZEIT Studienführer (Study Guide) since 2005.

Doubts about the professional quality of the CHE Ranking have been voiced repeatedly within the field of sociology since it was first implemented. However, in view of the informational needs of prospective students of sociology, sociological institutes have participated in the data collection for the ranking. Rather than neglecting to mention it here, we self-critically acknowledge that sociology and the social sciences have been officially represented on the CHE Advisory Board in the past and that they may not have exercised, and availed of, their influence and their supervisory responsibilities – or at least may not have done so effectively enough.

However, since the middle of last year, mounting professional and science-policy-related misgivings on the part of a number of sociological institutes have led to a rethink. In June 2011, the Institute of Sociology at the University of Jena – which had consistently received very good ratings from the CHE – decided that it no longer wished to participate in the CHE Ranking. This prompted the Board of the German Sociological Association (GSA) to undertake a thorough analysis of the CHE Ranking. After studying the available documentation and conducting a lengthy discussion with the representatives of the Centre for Higher Education Development responsible for the ranking, the GSA Board arrived at the appraisal and the recommendations documented below. At its meeting on 20 April 2012, the GSA Council endorsed this appraisal and unanimously adopted the recommendations ensuing therefrom.

Professional and Science-Policy-Related Appraisal of the CHE Ranking

Firstly, the CHE Ranking has a number of serious methodological weaknesses and empirical gaps. Secondly, the summary assessment practice and the specific publication formats of the ranking systematically invite misinterpretations. Both aspects will be discussed in greater detail here.

Professional Appraisal: Research Indicators

For a number of years, at least, the quality of the research conducted at the individual faculties was measured on the basis of publication databases that not only the German Council of Science and Humanities (Wissenschaftsrat), but, meanwhile, also the CHE itself, deems to be an unsuitable, or – in the case of sociology, at least – an insufficiently meaningful indicator. As an alternative, the CHE now measures research performance on the basis of external research funding per (budgeted) academic staff member. When doing so – and without any further differentiation – Higher Education Pact positions, for example, which were created expressly not for research purposes but rather to cope with teaching loads, are also included in the divisor of the external funding values. In effect, this means that – in purely arithmetical terms – as the teaching load of an institute increases (in the area of teacher training, for example), its per capita research performance, which the CHE claims to "measure", deteriorates. It is obvious that the universities particularly affected are those that, because of the region in which they are located, have taken in a large number of students within the framework of the Higher Education Pact. Thus, the "burden of proof" of the quality of research of an individual institute is borne almost entirely by the subjective criterion of that institute's research reputation among fellow academics at other  – in the logic of the ranking, rival – institutions. Anyone who has ever participated in the CHE survey of professors will be aware of its lack of methodological sophistication and the undifferentiated nature of its contents. The informational value of such sweeping faculty-specific judgements for prospective students, as the intended target audience of the ranking, is definitely questionable.

Professional Appraisal: Teaching Indicators

or this specific target audience the central criterion for the choice of a possible study location is obviously the quality of teaching at the various sociological institutes. However, this indicator is measured by the CHE largely on the basis of a student survey characterised by (a) low response rates (19.3% in sociology in the last round), (b) a small number of participants (at every third university, less than 30 students from the subject area in question), and (c) completely unexplained survey selectivity, with the result that the danger of responses biased by careless or inattentive response behaviour is correspondingly high. The CHE is well aware of the fact that by no means all universities draw a genuinely random sample with a calculable probability of selection. Moreover, a self-administered questionnaire survey with no systematic reminders and no nonresponse study can claim practically no validity. By the end of his or her basic training in methodology at the latest, any student of sociology would recognize that the survey is simply absurd. Therefore, massive doubts must be expressed with regard to the results of the CHE student survey – which is often described in discussions about the ranking as an opportunity for student participation qua evaluation.

Moreover, important, if not decisive, parameters for the assessment of the study situation – parameters that cannot be influenced by the teaching staff – are not included in the analysis (or the evaluation) at all. These parameters include, for example, (a) the respective faculty-student ratio (the ratio of the teaching load of faculty employed in budgeted positions to the number of students), (b) the associated arithmetical (and actual) class sizes, and (c) the efficiency of examination offices. Furthermore, the CHE forgoes the collection of qualitative data that are, or would be, extremely relevant for the assessment of the quality of teaching at the individual locations and for prospective students' choice of study programmes, for example, the areas of focus and specialization offered by the various sociological study programmes, and the systematic linking of teaching with the research conducted at the institute in question – irrespective of the external-funding intensity or reputational standing of that research. Such an inadequate, extremely selective, and factually misleading data situation renders absolutely untenable the construction of a ranking of institutes with regard to their teaching performance.

Science-Policy-Related Appraisal: Evaluation Practice and Publication Formats

The basic problem with the university ranking is that the Centre for Higher Education Development aims to construct a ranking of institutes with regard to their teaching performance, and actually "succeeds" in doing so, namely by dividing sociological institutes on the basis of extremely doubtful data into "good" and "bad" – or "better" and "worse" – institutes, and listing them hierarchically with spurious accuracy.  Because of the sweeping evaluation practice and simplistic modes of presentation, the publication formats of the ranking invite systematic misconceptions about the situation in sociology.

The CHE collects data for a total of approximately eighteen indicators of research and teaching quality in the field of sociology, and these indicators are also published in the online version of the ranking. However, for a description of the individual indicators and their derivation, readers are referred to the small print, which most people are unlikely to understand. In the print version published in DIE ZEIT and in the ZEIT Studienführer (Study Guide), however, these eighteen indicators are not combined to form indices. Rather, only five or six indicators are selectively presented. This fact is neither discernible from a cursory reading, nor is any explanation given for the selection that has been made. Moreover, for both the quality of research and the quality of teaching, only the subjective evaluations from what we have shown to be methodologically extremely questionable surveys are presented. In particular, the simplistic ranking by means of traffic-light symbols (recently modified to green, yellow, and blue) obscures the remarkable paucity of the database; in some cases, a single binary-coded response to a questionnaire item can yield a traffic light symbol signalling "good" or "bad" performance. The CHE Ranking – willingly bowing to the presentational demands of the mass media – gives the impression of unequivocal, reliable assessments, which are by no means covered by the available data. Here, systematic differentiations and thick descriptions would clearly be indicated and appropriate.

It is indeed disturbing in itself that the CHE Ranking thus misleads the very group whose interests, according to its authors, it is primarily supposed to serve, namely prospective students of sociology, who could, indeed, benefit from having accurate information about individual study locations when choosing a university and a study programme. It is perhaps a blessing in disguise, therefore, that – as far as teachers of sociology can ascertain – hardly any of the students who are now studying sociology at German universities, at any rate, allowed themselves to be decisively influenced by the CHE Ranking. Obviously, only a small minority of prospective students take serious note of the ranking – and that is a good thing.

On the other hand – and quite apart from its lack of informational value – the CHE Ranking has a very problematic effect on science policy. Therefore, if we are to believe the declared intentions of its authors, the ranking serves de facto a purpose for which it was not "actually" intended. However, in higher-education-policy reality, the CHE Ranking invites – or, indeed, practically demands – extremely simplistic interpretations on the part of faculty- and university management and ministerial bureaucracies. This may lead to structural decisions that have grave consequences for sociology, as an academic discipline, and its study programmes at individual locations – decisions that may well be objectively unfounded.

In view of the danger of such political uses of the CHE Ranking, it appears all the more remarkable that the persons responsible for the ranking at the CHE are unwilling to limit themselves to an informational function – however incomplete and unsatisfactory its implementation may be. They maintain that they cannot do without the construction of a ranking of the sociological institutes in Germany. At the preliminary meeting with those responsible for the ranking at the CHE, the German Sociological Association representatives were told quite openly that it would not be possible for the discipline to satisfy its own informational intentions within the framework of the procedure organized by the CHE, while at the same time avoiding the obligatory assessment and ranking. Thus, it became quite clear to the GSA that the CHE at least accepts the possibility that the university ranking will be politicized. The authors of the ranking claim that it merely depicts existing differences in quality between the sociological faculties. However, in the opinion of the German Sociological Association, there are strong grounds for assuming that the CHE Ranking contributes significantly to the construction of "difference" and, thus, to creating divisions in the university landscape in the field of sociology.

In the worst case, therefore, the ranking will act as a self-fulfilling prophecy in the long term. Faculties labelled on a supposedly sound empirical basis as "good" or "bad" may actually become so in the long run because of the structural policy decisions and – perhaps one day, after all – changing student flows prompted by their rankings. More than any other academic discipline, sociology is aware of the way in which such social definitions of situations influence action. It therefore feels both a scientific obligation to draw attention to the far-reaching consequences of actions based on incorrect definitions of situations, and a scientific responsibility not to contribute to such consequences any longer.

Recommendations Concerning the Handling of the CHE Ranking

Firstly, because the CHE Ranking has serious methodological and empirical deficiencies, secondly, because it withholds vital information from prospective students, as its declared target audience, and, thirdly, because it gives rise to wrong decisions on the part of science-policy decision-makers, sociology must take a stand against this presentation of its teaching and research performance in the public sphere constructed by the media. On the basis of this appraisal and the justifications thereof outlined above, the Board and the Council of the German Sociological Association have arrived at the following recommendations:

1.  Because our analyses and the discussion of the considerable methodological deficiencies with the CHE representatives responsible for the ranking yielded no prospect of significant improvements in the CHE Ranking in the future, we hereby declare that this evaluation does not meet the basic quality requirements of empirical social research. As a professional sociological society, we call on the sociological institutes at German universities not to give the impression any longer that they support an empirical procedure that sociology must reject on professional grounds. In concrete terms, this means that the sociological institutes should defend and explain this resolution and its professional justifications vis-à-vis their faculty- and university managers and their students, and, in particular, that they should not take part in the collection of data for the next CHE Ranking of sociology.

2.  The GSA calls on science-policy decision-makers at university and ministerial level not to rely any longer on appraisals and information derived from the CHE Ranking when deliberating on, and undertaking interventions for, the development of sociology at the discipline's various university locations. More reliable information than that provided by the ranking already exists; in individual cases, occasion-specific evaluations should be conducted, for which both suitable concepts and unbiased institutions are available.

3.  As an empirically oriented social science discipline, sociology claims to be particularly competent in the assessment of all kinds of empirical social research – including evaluations such as the CHE Ranking. In the present case, this competency implies a responsibility to recommend other disciplines, which are perhaps less sensitive in this regard, not to participate in the CHE Ranking any longer. After all, the grave deficiencies and misuses of this ranking that have been observed in the case of sociology are equally characteristic of its application to other disciplines.

4.  Sociology is a discipline that is proficient in evaluation in every sense of the word. For this reason, it made itself available in 2006 for a pilot study on the rating (and precisely not the ranking) of research performance conducted by the Council of Science and Humanities (Wissenschaftsrat). In a process characterised by considerable social and technical complexity, this scientific rating demonstrated in an exemplary way the minimum requirements that a reliable and valid scientific evaluation must fulfil. To further meet the specific and justified desire on the part of prospective students of sociology for assistance in choosing a course of study and a study location, the GSA will develop a publicly accessible information package, which will also feature descriptions of the sociology programmes offered by German universities.

This statement, a summary thereof, and the latest information on the GSA's science-policy initiative launched herewith are available online at www.soziologie.de/che.

for Global Express, June 21, 2013