Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

On 2 June 2009 the European Commission announced the launching of a feasibility study to develop a multidimensional global university ranking.

Its aims were to ‘look into the feasibility of making a multidimensional ranking of universities in Europe, and possibly the rest of the world too’. The Commission believes that accessible, transparent and comparable information would make it easier for students and teaching staff, but also parents and other stakeholders, to make informed choices between different higher education institutions and their programs. It would also help institutions to better position themselves and improve their quality and performance.

The Commission pointed out that existing rankings tend to focus on research in ‘hard sciences’ and ignore the performance of universities in areas like humanities and social sciences, teaching quality and community outreach. While drawing on the experience of existing university rankings and of EU-funded projects on transparency in higher education, the new ranking system should be:

  • multidimensional: covering the various missions of institutions, such as education, research, innovation, internationalization, and community outreach;

  • transparent: it should provide users with a clear understanding of all the factors used to measure performance and offer them the possibility to consult the ranking according to their needs;

  • global: covering institutions inside and outside Europe (in particular those in the US, Asia and Australia).

The project would consist of two consecutive parts:

  • In a first phase the consortium would design a multidimensional ranking system for higher education institutions in consultation with stakeholders.

  • In a second phase the consortium would test the feasibility of the multidimensional ranking system on a sample of no less than 150 higher education and research institutions. The sample would focus on the disciplines of engineering and business studies and should have a sufficient geographical coverage (inside and outside of the EU) and a sufficient coverage of institutions with different missions.

In June 2011 our CHERPA-Network which was awarded the multidimensional ranking project submitted its final report to the European Commission. One of the report’s major conclusions was that an enhanced understanding of the diversity in the profiles and performances of higher education and research institutions at a national, European and global level requires a new ranking tool. The new tool will promote the development of diverse institutional profiles. It will also address most of the major shortcomings of existing ranking instruments. The full report of the project is available on: http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf. We called this new tool U-Multirank as this stresses three fundamental points of departure: it is multidimensional, recognizing that higher education institutions serve multiple purposes and perform a range of different activities; it is a ranking of university performances (although not in the sense of an aggregated league table like other global rankings); and it is user-driven (as a stakeholder with particular interests, you are enabled to rank institutions with comparable profiles according to the criteria important to you).

This chapter addresses the basic design aspects of the new, multidimensional global ranking tool. First, we present the general design principles that to a large extent have guided the design process. Secondly, we describe the conceptual framework from which we deduce the five dimensions of the new ranking tool. Finally, we outline a number of methodological choices that have a major impact on the operational design of U-Multirank.

2 Design Principles

U-Multirank aims to address the challenges identified as arising from the various currently existing ranking tools. Using modern theories and methodologies of design processes as our base (Bucciarelli, 1994; Oudshoorn & Pinch, 2003) and trying to be as explicit as possible about our approach, we formulated a number of design principles that guided the development of the new ranking tool. The following list contains the basic principles applied when designing and constructing U-Multirank.

  • Our fundamental epistemological argument is that as all observations of reality are theory-driven (formed by conceptual systems) an ‘objective ranking’ cannot be developed (see Chap. 1). Every ranking will reflect the normative design and selection criteria of its constructors.

  • Given this epistemological argument, our position is that rankings should be based on the interests and priorities of their users: rankings should be user-driven . This principle ‘democratizes’ the world of rankings by empowering potential users (or categories of users) to be the dominant actors in the design and application of rankings rather than rankings being restricted to the normative positions of a small group of constructors. Different users and stakeholders should be able to construct different sorts of rankings. (This is one of the Berlin Principles).

  • Our second principle is multidimensionality. Higher education and research institutions are predominantly multipurpose, multiple-mission organizations undertaking different mixes of activities (teaching and learning, research, knowledge transfer, regional engagement, and internationalization are five major ­categories that we have identified; see the following section). Rankings should reflect this multiplicity of functions and not focus on one function (research) to the virtual exclusion of all else. An obvious corollary to this principle is that institutional performance on these different dimensions should never be aggregated into a composite overall ranking.

  • The next design principle is comparability. In rankings, institutions and programs should only be compared when their purposes and activity profiles are sufficiently similar. Comparing institutions and programs that have very different purposes is worthless. It makes no sense to compare the research performance of a major metropolitan research university with that of a remotely located University of Applied Science; or the internationalization achievements of a national humanities college whose major purpose is to develop and preserve its unique national language with an internationally orientated European university with branch campuses in Asia.

  • The fourth principle is that higher education rankings should reflect the multilevel nature of higher education. With very few exceptions, higher education institutions are combinations of faculties, departments and programs of varying strength. Producing only aggregated institutional rankings disguises this reality and does not produce the information most valued by major groups of stakeholders: students, potential students, their families, employers, academic staff and professional organizations. These stakeholders are mainly interested in information about a particular field. This does not mean that institutional-level rankings are not valuable to other stakeholders and for particular purposes. The new instrument should allow for the comparisons of comparable institutions at the level of the organization as a whole and also at the level of the disciplinary fields and multidisciplinary in which they are active.

  • Finally we include the principle of methodological soundness. The new instrument should refrain from methodological mistakes such as the use of composite indicators, the production of league tables and the denial of contextuality. In addition it should minimize the incentives for strategic behavior on the part of institutions to ‘game the results’.

These principles underpin the design of U-Multirank, resulting in a user-driven, multidimensional and methodologically robust ranking instrument. In addition, U-Multirank aims to enable its users to identify institutions and programs that are sufficiently comparable to be ranked, and to undertake both institutional and field level analyses.

A fundamental question regarding the design of any transparency tool has to do with the choice of the ‘dimensions’: on which subject(s) will the provision of information focus? What will be the topics of the new ranking tool?

We take the position that any process of collecting information is driven by a – more or less explicit – conceptual framework. Transparency tools should clearly show what these conceptual frameworks are and how they have played a role in the selection of the broader categories of information on which these tools are focused.

For the design of U-Multirank we specify our own conceptual framework in the following section.

3 Conceptual Framework

A meaningful ranking requires a conceptual framework in order to decide on its content categories. We call these categories the ‘dimensions’ of the new ranking tool. We found a number of points of departure for a general framework for studying higher education and research institutions in the higher education literature. Four different conceptual perspectives have been combined in this approach.

First, a common point of departure is that processing knowledge is the general characteristic of higher education and research institutions (Becher & Kogan, 1992; Clark, 1983). ‘Processing’ can be the discovery of new knowledge as in research, or its transfer to stakeholders outside the higher education and research institutions (knowledge transfer) or to various groups of ‘learners’ (education). Of course, a focus on the overall objectives of higher education and research institutions in the three well-known primary processes or functions of ‘teaching and learning, research, and knowledge transfer’ is a simplification of the complex world of higher education and research institutions. These institutions are, in varying combinations of focus, committed to the efforts to discover, conserve, refine, transmit and apply knowledge (Clark). But the simplification helps to encompass the wide range of activities in which higher education and research institutions are involved. The three functions are a useful way to describe conceptually the general purposes of these institutions and therefore are the underlying three dimensions of our new ranking tool.

The second conceptual assumption is that the performance of higher education and research institutions may be directed at different ‘audiences’. In the current higher education and research policy area, two main general audiences have been prioritized, the first through the international orientation of higher education and research institutions. This emphasizes how these institutions are seen as society’s portals to the globalized world (both ‘incoming’ influences and ‘outgoing’ contributions to the international discourse). At the same time, the institutions’ engagement with the region can be distinguished. Here the emphasis is on the involvement with and impact on the region in which a higher education institution operates. In reality these ‘audiences’ are of course often combined in the various activities of higher education and research institutions.

It is understood that the functions higher education and research institutions fulfill for international and regional audiences are manifestations of their primary processes, i.e. the three functions of education, research and knowledge transfer mentioned before. What we mean by this is that there may be educational elements, research elements and knowledge transfer elements to the international orientation. Similarly, regional engagement may be evident in an institution’s education, research and knowledge transfer activities. International and regional orientations are two further dimensions of the multidimensional ranking.

The term ‘processing’ used above points to the third main conceptual assumption, namely the major stages in any process of creation or production: input, throughput (or the process in a narrow sense) and its results, which can be subdivided into immediate outputs and further‐reaching impacts. A major issue in higher education and research institutions, as in many social systems, has been that the transformation from inputs to performances is not self‐evident. One of the reasons why there is so much criticism of league tables is exactly the point that from similar sets of inputs, different higher education and research institutions may reach quite different types and levels of performance.

We make a general distinction between the ‘enabling’ stages of the overall creation stages on the one hand and the ‘performance’ stages on the other. The enabling stages consist of the inputs and processes of creation/production processes while the performance stages include their outputs and impacts. We have used the distinction of the various stages of a creation/production process to further elaborate the conceptual framework for the selection of indicators in the new ranking instrument.

A fourth assumption refers to the different stakeholders or users of rankings. Ranking information is produced to inform users about the value of higher education and research, which is necessary as it is not obvious that they are easily able to take effective decisions without such information. (Higher) education is not an ordinary ‘good’ for which the users themselves may assess the value a priori (using, e.g., price information). Higher education is to be seen as an experience good (Nelson, 1970): the users may assess the quality of the good only while or after ‘experiencing’ it (i.e. the higher education program), but such ‘experience’ is ex post knowledge. It is not possible for users to know beforehand whether the educational program meets their standards or criteria. Ex ante they only can refer to the perceptions of previous users. Some even say that higher education is a credence good (Dulleck & Kerschbamer, 2006): the value of the good cannot be assessed while experiencing it, but only (long) after. If users are interested in the value added of a degree program on the labor market, information on how well a class is taught is not relevant. They need information on how the competences acquired during higher education will improve their position on the career or social ladder. So stakeholders and users have to rely on information that is provided by a variety of transparency tools and quality assessment outcomes. However, different users require different types of information.

Some users are interested in the overall performance of higher education and research institutions (e.g. policy-makers) and for them the internal processes contributing to performance are of less interest. The institution may well remain a ‘black box’ for these users. Other stakeholders (students and institutional leaders are prime examples) are interested precisely in what happens inside the box. For instance, students may want to know the quality of teaching in the field in which they are interested. They may want to know how the program is delivered, as they may consider this as an important aspect of their learning experience and their time in higher education (consumption motives). Students might also be interested in the long‐term impact of taking the program as they may see higher education as an investment and are therefore interested in its future returns (investment motives).

Users engage with higher education for a variety of reasons and therefore will be interested in different dimensions and performance indicators of higher education institutions and the programs they offer. Rankings must be designed in a balanced way and include relevant information on the various stages of knowledge processing which are relevant to the different stakeholders and their motives for using rankings.

The conceptual grid shown below must be applied twice: once to the institution as a whole and once at the field level, and it has to accommodate interest in both performance and (to a lesser extent) process. For different dimensions (research, teaching & learning, knowledge transfer) and different stakeholders/users the relevance of information about different aspects of performance may vary.

The result of this elementary conceptual framework is a matrix showing the types of indicators that could be used in rankings and applied at both institutional and field levels (Table 6.1). Filtering higher education and research institutions into homogeneous groups requires contextual information rather than only the input and process information that is directly connected with enabling the knowledge processes. Contextual information for higher education and research institutions relates to their positioning in society and specific institutional appearances. It describes the conditions in which the primary processes of education, research and knowledge transfer operate. A substantial part of the relevant context is captured by applying another multidimensional transparency tool (U-Map) in pre-selecting higher education and research institutions (see below). Additional context information may be needed to allow for the valid interpretation of specific indicators by different stakeholders.

Table 6.1 Conceptual grid U-Multirank
Full size table

Using this conceptual framework we selected the following five dimensions as the major content categories of U-Multirank:

  • Teaching & Learning

  • Research

  • Knowledge Transfer

  • International Orientation

  • Regional Engagement

In the next chapter we will discuss the various indicators to be used in these five dimensions.

An important factor in the criticism of rankings and league tables is the fact that often their selection of indicators is guided primarily by the (easy) availability of data rather than by relevance. This often leads to an emphasis on indicators of the enabling stages of the higher education production process, rather than on the area of performance, largely because governance of higher education and research institutions has concentrated traditionally on the bureaucratic (in Weber’s neutral sense of the word) control of inputs: budgets, personnel, students, facilities, etc. Then too, inputs and processes can be influenced by managers of higher education and research institutions. They can deploy their facilities for teaching, but in the end it rests with the students to learn and, after graduation, work successfully with the competencies they have acquired. Similarly, higher education and research institution managers may make facilities and resources available for research, but they cannot guarantee that scientific breakthroughs are ‘created’. Inputs and processes are the parts of a higher education and research institution’s system that are best documented. But assessing the performance of these institutions implies a more comprehensive approach than a narrow focus on inputs and processes and the dissatisfaction among users of most current league tables and rankings is because they often are more interested in institutional performance while the information they get is largely about inputs. In our design of U-Multirank we focused on the selection of output and impact indicators. U-Multirank is a multidimensional performance assessment tool and thus includes indicators that relate to the performances of higher education and research institutions.

4 Methodological Aspects

There are a number of methodological aspects that have a clear impact on the way a new, multidimensional ranking tool like U-Multirank can be developed. In this section we explain the various methodological choices made when designing U-Multirank.

4.1 Methodological Standards

In addition to the content-related conceptual framework, the new ranking tool and its underlying indicators must be based also on methodological standards of empirical research, validity and reliability in the first instance. In addition, because U-Multirank is intended to be an international comparative transparency tool, it must deal with the issue of comparability across cultures and countries and finally, in order to become sufficiently operational, U-Multirank has to address the issue of feasibility.

4.1.1 Validity

(Construct) validity refers to the evidence about whether a particular operationalization of a construct adequately represents what is intended by the theoretical account of the construct being measured. When characterizing, e.g. the internationality of a higher education institution, the percentage of international students is a valid indicator only if scores are not heavily influenced by citizenship laws. Using the nationality of the qualifying diploma on entry has therefore a higher validity than using citizenship of the student.

4.1.2 Reliability

Reliability refers to the consistency of a set of measurements or measuring instrument. A measure is considered reliable if, repeatedly applied in the same population, it would always arrive at the same result. This is particularly an issue with survey data (e.g. among students, alumni, staff) used in rankings. In surveys and with regard to self-reported institutional data, the operationalizing of indicators and formulation of questions requires close attention – in particular in international rankings, where cross-cultural understanding of the questions will be an issue.

4.1.3 Comparability

A ranking is the comparison of institutions and programs using numerical indicators. Hence the indicators and underlying data/measure must be comparable between institutions; they have to measure the same quality in different institutions. In addition to the general issue of comparability of data across institutions, international rankings have to deal with issues of international comparability. National higher education systems are based on national legislation setting specific legal frameworks, including legal definitions (e.g. what/who is a professor). Additional problems arise from differing national academic cultures. Indicators, data elements and underlying questions have to be defined and formulated in a way that takes such contextual variations into account. For example, if we know that doctoral students are counted as academic staff in some countries and as students in others, we need to ask for the number of doctoral students counted as academic staff in order to harmonize data on academic staff (excluding doctoral students).

4.1.4 Feasibility

The objective of our project was to design a multidimensional global ranking tool that is feasible in practice. The ultimate test of the feasibility of our ranking tool has to be empirical: can U-Multirank be applied in reality and can it be applied with a favorable relation between benefits and costs in terms of financial and human resources? We report on the empirical assessment of the feasibility of U-Multirank in Chap. 9.

4.2 User-Driven Approach

To guide the readers’ understanding of U-Multirank, we now briefly describe the way we have methodologically worked out the principle of being user-driven. We propose an interactive web‐based approach, where users will be able to declare their interests in a three step, user‐driven process:

  1. 1.

    select a set of institutions or fields in institutions (‘units’) that are homogeneous on descriptive issues judged by the users to be relevant given their declared interests;

  2. 2.

    choose whether to focus the ranking on higher education and research institutions as a whole (focused institutional rankings) or on fields within these institutions (field‐based rankings);

  3. 3.

    select a set of indicators to rank the chosen units. This will result in users creating their own specific and different rankings, according to their needs and wishes, from the entire database.

The first step can be based on the existing U-Map classification tool (see the following Sect. 6.4.3). We argue that it does not make sense to compare all institutions irrespective of their missions, profiles and characteristics, so a selection of comparable institutions based on U-Map should be the basis for any ranking.

In the second step, the users make their choices regarding the ranking level, i.e. whether a ranking will be created at the institutional level, creating a focused institutional ranking, or at the field level, creating a field-based ranking.

The final step is the selection of the indicators to be used in the ranking. There are two ways to organize this choice process. In the first option, users have complete freedom to select from the overall set of indicators, choosing any indicator, addressing any cell in the conceptual grid. We call this the ‘personalized rankings’. Through this personalized approach the users may find information on those aspects in which they are particularly interested. Compared to existing league tables we see this as one of the advantages of our approach. However this kind of individualized, one‐off ranking (which may be different even if the same user applies different indicators) may not be attractive to all types of users, as there is no clear non-relative result for a particular institution or program. In the second option the indicators can be pre-selected. Such a selection can be undertaken from the perspective of a specific organization or institution, and will be called an ‘authoritative ranking’. It is important that the selection of the indicators is made as transparent as possible.

4.3 U-Map and U-Multirank

The principle of comparability calls for a method that helps us in finding institutions the purposes and activity patterns of which are sufficiently similar in order to enable useful and effective rankings. Such a method, we suggest, can be found in the connection of U-Multirank with U-Map (see www.u-map.eu).

U-Map, being a classification tool, describes (‘maps’) higher education institutions on a number of dimensions, each representing an aspect of their activities. This mapping produces activity profiles of the institutions, displaying what the institutions do and how that compares to other institutions. U-Map prepares the ground for U-Multirank in the sense that it helps identify those higher education institutions that are comparable and for which, therefore, performance can be compared by means of the U-Multirank ranking tool.

Where U-Map is describing what the institutions do (and thus offers descriptive profiles), U-Multirank focuses on the performance aspects of higher education and research institutions. U-Multirank shows how well the higher education institutions are performing in the context of their institutional profile. Thus, the emphasis is on indicators of performance, whereas in U-Map it lies on the enablers of that performance – the inputs and activities. Despite the difference in emphasis, U-Map and U-Multirank share the same conceptual model. The conceptual model provides the rationale for the selection of the indicators in both U-Map and U-Multirank, both of which are complementary instruments for mapping diversity; horizontal diversity in classification and vertical diversity in ranking.

4.4 Grouping

U-Multirank does not calculate league tables. As has been argued in Chap. 4, league table rankings have severe flaws which make them, methodologically speaking, unreliable as transparency tools. As an alternative U-Multirank uses a grouping method. Instead of calculating ‘exact’ league table positions we will assign institutions to a limited number of groups.

Within groups there will be no further differentiation. Between the groups ­statistical methods guarantee that there is a clear difference between performance levels of different groups. The number of groups should be related to the number of institutions ranked. On the one hand the number of groups should express clear differences of performance; on the other hand the number should not be so low as to be restrictive, with the end result that many institutions end up clustered in one group. Last but not least, the number of groups and the methods for calculating the groups must be clear and comprehensible to users.

4.5 Design Context

In this chapter we have described the general aspects of the design process regarding U-Multirank. We have indicated our general design principles; we have described the conceptual framework from which the five dimensions of U-Multirank are deduced, and we have outlined a number of methodological approaches to be applied in U-Multirank. Together these elements form the design context from which we have constructed U-Multirank.

The design choices made here are in accordance with both the Berlin Principles and the recommendations by the Expert Group on the Assessment of University‐based Research. The Berlin PrinciplesFootnote 1 emphasize (a. o.) the importance of being clear about the purpose of rankings and their target groups, of recognizing the diversity of institutional profiles, providing users the option to create tailor-made approaches, and of the need to focus on performance rather than on input factors. The AUBR Expert GroupFootnote 2 (a. o.) underlines the importance of stakeholders’ needs and ­involvement, as well as the principles of purposefulness, contextuality, and multidimensionality of rankings.

Based on our design context, in the following chapters we report on the construction of U-Multirank.