Abstract
In this chapter, we look closely at the fragmentation of rankings and indicators relevant to knowledge governance in higher education, economic competitiveness, innovation, and good governance that has challenged the established producers of numeric knowledge. Not only have the amount of international datasets multiplied, but the varieties of measurement—concerning conceptual and methodological decisions—have also increased. We find that the process of fragmentation has not effectively challenged the ideas behind the figures. Instead, the emerging indicator sets are woven into the fabric of the existing measurements as the figures that enter the field largely build on the existing ones without fundamentally challenging their ideational premises. This further embeds the use of numerical assessment in transnational governance.
This is a preview of subscription content, log in via an institution to check access.
Notes
- 1.
The World Bank has used Country Policy and Institutional Assessment (CPIA) tool since mid-1970s for assessing the eligibility for funding.
- 2.
Also the use of the good governance indicators has drawn interest, most notably with regards to development funding (Hammergren 2011; Knoll and Zloczysti 2012; Saisana and Saltelli 2011). Here, the indicators such as the Worldwide Governance Indicators are seen instrumental for development aid, while also attracting attention on the local level (Morgan 2011; Stubbs 2009). While the World Bank has not used the WGI in its allocation of funding, the index has obtained such uses. The most prominent user of the governance indices in development funding has been the US government through its Millennium Challenge Corporation (MCC) that was established in 2004.
- 3.
Most notably, the Millennium Challenge Corporation uses Fringe Special (and Open Net Initiative) for its financing criteria since 2012, having previously used WGI Voice and Accountability data.
- 4.
The indices explicitly criticized were the World Bank Governance Indicators, the European Central Bank’s Public Sector Efficiency Study, the World Economic Forum’s Public Institutions Index in the Global Competitiveness Report, and the “Government Efficiency” Indicator developed by the International Institute for Management Development in the World Competitiveness Yearbook.
- 5.
“There is a significant growth in broad measures of “governance”, including some comparative data concerning public sector bureaucratic quality. However, most of these data are based on subjective assessments, and were not initially collected with comparative analysis of public management as a principal aim. […] Reviews of these data note that these indicators incorporate significant methodological problems. The data often do not adequately measure what they claim to measure, and can aggregate many diverse indicators, achieving statistical quality at the price of significant loss of conceptual precision. Often data amount to broad subjective evaluations combined with service-specific performance indicators. The former can be excessively impressionistic and the latter cannot be aggregated in any meaningful way” (OECD 2005, 6).
- 6.
“The absence of a well-accepted theoretical framework for governance ensures that any composite indicators are largely devices for communication—for crystallizing concerns about corruption etc. into a single short and pithy summary” (OECD 2006c, 60).
- 7.
“More generally, recognizing the importance of margins of error and the imprecision of country rankings, we do not follow the popular practice of producing precisely ranked ‘top ten’ or ‘bottom ten’ lists of countries according to their performance on the WGI, recognizing that such seemingly precise ‘horse races’ are of dubious relevance and reliability” (Kaufmannn et al. 2008, 5).
- 8.
“In 2003 after the publication of the Shanghai Jiatong University breakthrough ranking, the Academic Ranking of World Universities (ARWU), we decided to adopt the main innovations proposed by Liu and his team. The ranking will be built from publicly available web data, combining the variables into a composite indicator, and with a true global coverage. The first edition was published in 2004, it appears twice per year since 2006 and after 2008 the portal also includes webometrics rankings for research centers, hospitals, repositories and business schools.” Webometrics Methodology, [http://www.webometrics.info/en/Methodology].
- 9.
2017. Ranking Web of Universities. January New Edition, [http://www.webometrics.info/en/node/178], date accessed 30 June 2017.
- 10.
http://www.educationalpolicy.org/pdf/global2005.pdf, date accessed 28 February 2013.
- 11.
CWTS Leiden Ranking 2017 Indicators, [http://www.leidenranking.com/information/indicators], date accessed 30 June 2017.
- 12.
SIR Methodology, http://www.scimagoir.com/methodology.php, date accessed 30 June 2017.
- 13.
Transparent Ranking: Top Universities by Google Scholar Citations [http://www.webometrics.info/en/node/169], date accessed 30 June 2017.
- 14.
https://www.topuniversities.com/qs-world-university-rankings/methodology, date accessed 30 June 2017.
- 15.
https://www.timeshighereducation.com/world-university-rankings/methodology-world-university-rankings-2016-2017, date accessed 30 June 2017.
- 16.
Webometrics Methodology, [http://www.webometrics.info/en/Methodology].
- 17.
THE World University Rankings 2016–2017 methodology [https://www.timeshighereducation.com/world-university-rankings/methodology-world-university-rankings-2016-2017].
- 18.
SCImago Institutions Ranking methodology, http://www.scimagoir.com/methodology.php.
- 19.
Centrum für Hochschulentwicklung, Berlin Principles on Ranking of Higher Education Institutions [https://www.che.de/downloads/Berlin_Principles_IREG_534.pdf].
- 20.
Testing Student and University Performance Globally: OECD’s AHELO—OECD, [http://www.oecd.org/edu/skills-beyond-school/testingstudentanduniversityperformancegloballyoecdsahelo.htm].
- 21.
- 22.
- 23.
The CHERPA consisted of five partners: Centrum für Hochschulentwicklung (CHE, Germany), the Center for Higher Education Policy Studies at the University of Twente (Netherlands), the Centre for Science and Technology Studies (CWTS) at Leiden University (Netherlands), INCENTIM research division at the Catholic University of Leuven (Belgium), and the Observatoire des Sciences et des Techniques in Paris.
- 24.
“U-Multirank—Education and Training—European Commission”. Education and Training. [https://ec.europa.eu/education/initiatives/u-multirank_en].
- 25.
The WEF report names few concrete examples of perceived problems such as the methods for calculating physical sales of goods and services that do not consider virtual platforms and nonmonetary exchanges of services as well as measurement issues in GDP as an indicator of economic progress (World Economic Forum 2016, 51–52).
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
The seven factors are R&D intensity, manufacturing value added, productivity, high-tech density, tertiary efficiency, researcher concentration, and patent activity.
- 33.
- 34.
- 35.
Haas further identifies joint policy enterprise as criteria for epistemic community (Haas 1992). This might apply in the field of good governance, where the actors are often explicitly committed to governance reform. However, in the domain of university rankings the motivations for creating the figures are less clear.
References
Aguillo, Isidro F., Judit Bar-Ilan, Mark Levene, and José Luis Ortega. 2010. Comparing University Rankings. Scientometrics 85 (1): 243–256.
Andrews, Matt. 2008. The Good Governance Agenda: Beyond Indicators without Theory. Oxford Development Studies 36 (4): 379–407. https://doi.org/10.1080/13600810802455120.
Andrews, Matt, Roger Hay, and Jerrett Myers. 2010. Can Governance Indicators Make Sense? Towards a New Approach to Sector-Specific Measures of Governance. Oxford Development Studies 38 (4): 391–410. https://doi.org/10.1080/13600818.2010.524696.
Bacchi, Carol Lee. 1999. Women, Policy and Politics: The Construction of Policy Problems. London: Sage Publications.
Baert, Patrick. 1991. Unintended Consequences: A Typology and Examples. International Sociology 6 (2): 201–210. https://doi.org/10.1177/026858091006002006.
Barendrecht, Maurits. 2011. Rule of Law, Measuring and Accountability: Problems to Be Solved Bottom Up. Hague Journal on the Rule of Law 3 (02): 281–304. https://doi.org/10.1017/S1876404511200071.
CHE. 2009. News: The CHERPA-Network Wins a European Tender to Develop a Multi-Dimensional Global Ranking of Universities. http://www.che.de/cms/?getObject=302&getNewsID=983&getCB=309&getLang=en
Cho, Dong-Sung, and Hwy-Chang Moon. 2000. From Adam Smith to Michael Porter: Evolution of Competitiveness Theory. Singapore: World Scientific Publishing Company.
Dakowska, Dorota. 2013. Polish Higher Education and the Global Academic Competition: University Rankings in the Reform Debates. In Global University Rankings. Challenges for European Higher Education, ed. Tero Erkkilä, 107–123. Basingstoke: Palgrave Macmillan.
Desrosières, Alain. 1998. The Politics of Large Numbers: A History of Statistical Reasoning. Cambridge: Harvard University Press.
Drechsler, Wolfgang. 2004. Governance, Good Governance, and Government: The Case for Estonian Administrative Capacity. TRAMES 4: 388–396.
Dubouloz, Catherine. 2008. L’Europe Veut Lancer Son Propre Classement Des Universités. Le Temps (CH), December 5.
Erkkilä, Tero. 2016. Global Governance Indices as Policy Instruments: Actionability, Transparency and Comparative Policy Analysis. Journal of Comparative Policy Analysis: Research and Practice 18 (4): 382–402. https://doi.org/10.1080/13876988.2015.1023052.
Erkkilä, Tero, and Ossi Piironen. 2013. Reforming Higher Education Institutions in Finland: Competitiveness and Global University Rankings. In Global University Rankings. Challenges for European Higher Education, ed. Tero Erkkilä, 124–143. Basingstoke: Palgrave Macmillan.
———. 2014. (De)Politicizing Good Governance: The World Bank Institute, the OECD and the Politics of Governance Indicators. Innovation: The European Journal of Social Science Research 27 (4): 344–360. https://doi.org/10.1080/13511610.2013.850020.
European Commission. 2006. Creating an Innovative Europe. Report of the Independent Expert Group on R&D and Innovation Appointed Following the Hampton Court Summit and Chaired by Mr. Esko Aho. European Commission. http://ec.europa.eu/invest-in-research/pdf/download_en/aho_report.pdf
———. 2011. Supporting Growth and Jobs—An Agenda for the Modernisation of Europe’s Higher Education System, COM(2011) 567 Final. Brussels: European Commission.
Freistein, Katja. 2016. Effects of Indicator Use: A Comparison of Poverty Measuring Instruments at the World Bank. Journal of Comparative Policy Analysis: Research and Practice 18 (4): 366–381. https://doi.org/10.1080/13876988.2015.1023053.
Giddens, Anthony. 1984. The Constitution of Society: Outline of the Theory of Structuration. Cambridge: Polity Press.
Gieryn, Thomas F. 1983. Boundary-Work and the Demarcation of Science from Non-Science: Strains and Interests in Professional Ideologies of Scientists. American Sociological Review 48 (6): 781–795. https://doi.org/10.2307/2095325.
Ginsburg, Tom. 2011. Pitfalls of Measuring the Rule of Law. Hague Journal on the Rule of Law 3 (02): 269–280. https://doi.org/10.1017/S187640451120006X.
Global Integrity. 2011. The Global Integrity Report: 2011. Methodology White Paper. https://www.globalintegrity.org/wp-content/uploads/2013/08/2011_GIR_Meth_Whitepaper.pdf
Gramatikov, Martin, Maurits Barendrecht, and Jin Ho Verdonschot. 2011. Measuring the Costs and Quality of Paths to Justice: Contours of a Methodology. Hague Journal on the Rule of Law 3 (02): 349–379. https://doi.org/10.1017/S1876404511200101.
Guha, Krishna, and Richard McGregor. 2007. World Bank Directors Test Zoellick. Financial Times, July 13. http://www.ft.com/intl/cms/s/0/fe1d7ece-30d8-11dc-9a81-0000779fd2ac.html#axzz2883nwxuM
Haas, Peter M. 1992. Introduction: Epistemic Communities and International Policy Coordination. International Organization 46 (1): 1–35.
Hammergren, Linn. 2011. Indices, Indicators and Statistics: A View from the Project Side as to Their Utility and Pitfalls. Hague Journal on the Rule of Law 3 (02): 305–316. https://doi.org/10.1017/S1876404511200083.
Hinthorne, Lauren Leigh. 2011. Democratic Crisis or Crisis of Confidence? What Local Perceptual Lenses Tell Us about Madagascar’s 2009 Political Crisis. Democratization 18 (2): 535–561. https://doi.org/10.1080/13510347.2011.553371.
Hobsbawm, Eric. 1987. Introduction: Inventing Traditions. In The Invention of Tradition, ed. Eric Hobsbawm and Terence Ranger. Cambridge: Cambridge University Press.
IMD. 2017. Competitiveness History Garelli and Bris—IMD Executive Education. IMD Business School. Accessed June 3. https://www.imd.org/wcc/world-competitiveness-center-mission/center-history-bris-garelli/
Information Technology and Innovation Foundation. 2016. Contributors and Detractors: Ranking Countries’ Impact on Global Innovation. Executive Summary. Information Technology and Innovation Foundation. http://www2.itif.org/2016-contributors-and-detractors-executive-summary.pdf?_ga=1.249958406.127216268.1464961189
INSEAD. 2007. Global Innovation Index 2007: The Power of Innovation. World Business. https://www.globalinnovationindex.org/userfiles/file/GII-2007-Report.pdf
———. 2013. The Global Talent Competitiveness Index 2013. INSEAD. https://www.insead.edu/sites/default/files/assets/dept/globalindices/docs/GTCI-2013-report.pdf
———. 2016. The Global Talent Competitiveness Index 2017. INSEAD. http://www.gtci2017.com/documents/GTCI_2017_web_r5.pdf
Ivanov, Kalin. 2009. Fighting Corruption Globally and Locally. In Ethics and Integrity in Public Administration: Concepts and Cases, ed. Raymond W. Cox III, 146–154. Armonk: M.E. Sharpe.
Jobbins, David. 2005. Moving to a Global Stage: A Media View. Higher Education in Europe 30 (2): 137–145. https://doi.org/10.1080/03797720500260009.
Joshi, Devin. 2011. Good Governance, State Capacity, and the Millennium Development Goals. Perspectives on Global Development & Technology 10 (2): 339–360. https://doi.org/10.1163/156914911X582468.
Kaufmann, Daniel, Aart Kraay, and Pablo Zoido-Lobatón 1999a. Aggregating Governance Indicators. World Bank Policy Research working Paper No. 2195.
———. 1999b. Governance Matters. World Bank Policy Research Working Paper No. 2196.
Kaufmann, Daniel, Aart Kraay, and Massimo Mastruzzi. 2003. Governance Matters III: Governance Indicators for 1996–2002. World Bank Policy Research Working Paper No. 3106.
———. 2007. Governance Matters VI: Governance Indicators for 1996–2006. World Bank Policy Research Working Paper No. 4149.
———. 2008. Governance Matters VII: Aggregate and Individual Governance Indicators, 1996–2007. World Bank Policy Research Working Paper No. 4654.
———. 2010. Response to ‘What Do the Worldwide Governance Indicators Measure?’. European Journal of Development Research 22 (1): 55–58. https://doi.org/10.1057/ejdr.2009.49.
———. 2011. The Worldwide Governance Indicators: Methodology and Analytical Issues. Hague Journal on the Rule of Law 3 (02): 220–246. https://doi.org/10.1017/S1876404511200046.
Kauppi, Niilo, and Tero Erkkilä. 2011. The Struggle Over Global Higher Education: Actors, Institutions, and Practices. International Political Sociology 5 (3): 314–326.
Knack, Stephen, Mark Kugler, and Nick Manning. 2003. Second-Generation Governance Indicators. International Review of Administrative Sciences 69 (3): 345–364. https://doi.org/10.1177/0020852303693004.
Knoll, Martin, and Petra Zloczysti. 2012. The Good Governance Indicators of the Millennium Challenge Account: How Many Dimensions Are Really Being Measured? World Development 40 (5): 900–915. https://doi.org/10.1016/j.worlddev.2011.11.010.
Langbein, Laura, and Stephen Knack. 2010. The Worldwide Governance Indicators: Six, One, or None? Journal of Development Studies 46 (2): 350–370. https://doi.org/10.1080/00220380902952399.
Liu, Nian Cai, and Ying Cheng. 2005. The Academic Ranking of World Universities. Higher Education in Europe 30 (2): 127–136. https://doi.org/10.1080/03797720500260116.
Liu, Nian Cai, Ying Cheng, and Liu Li. 2005. Academic Ranking of World Universities Using Scientometrics—A Comment to the ‘Fatal Attraction’. Scientometrics 64 (1): 101–109. https://doi.org/10.1007/s11192-005-0241-z.
McFerson, Hazel M. 2009. Measuring African Governance. Journal of Developing Societies (Sage Publications Inc.) 25 (2): 253–274. https://doi.org/10.1177/0169796X0902500206.
Morgan, Grace. 2011. Traction on the Ground: From Better Data to Better Policy. Hague Journal on the Rule of Law 3 (02): 380–396. https://doi.org/10.1017/S1876404511200113.
Mustajoki, Arto. 2013. Measuring Excellence in Social Sciences and Humanities: Limitations and Opportunities. In Global University Rankings. Challenges for European Higher Education, ed. Tero Erkkilä, 147–165. Basingstoke: Palgrave Macmillan.
Neumann, Robert, and Peter Graeff. 2010. A Multitrait-Multimethod Approach to Pinpoint the Validity of Aggregated Governance Indicators. Quality & Quantity 44 (5): 849–864. https://doi.org/10.1007/s11135-009-9238-7.
OECD. 2001. Citizens as Partners: Information, Consultation and Public Participation in Policy-Making. Paris: OECD.
———. 2003. Open Government. Fostering Dialogue with Civil Society. Paris: OECD.
———. 2005. Management in Government: Feasibility Report on the Development of Comparative Data. GOV/PGC(2005)10/ANN. Paris: OECD.
———. 2006a. Issues in Outcome Measurement for “Government at a Glance”. OECD GOV Technical Paper 3. GOV/PGC(2006)10/ANN3. Paris: OECD.
———. 2006b. Issues in Output Measurement for “Government at a Glance”. OECD GOV Technical Paper 2. GOV/PGC(2006)10/ANN2. Paris: OECD.
———. 2006c. How and Why Should Government Activity Be Measured in “Government at a Glance”? OECD GOV Technical Paper 1. GOV/PGC(2006)10/ANN1. Paris: OECD.
———. 2007. Towards Better Measurement of Government. OECD Working Papers on Public Governance, 2007/1, OECD Publishing.
———. 2009. Government at a Glance 2009. Paris: OECD.
Palonen, Kari. 2003. Four Times of Politics: Policy, Polity, Politicking, and Politicization. Alternatives 28 (2): 171–186.
Porter, Theodore M. 1996. Trust in Numbers. Princeton: Princeton University Press.
Raan, Anthony F.J.van. 2005a. Fatal Attraction: Conceptual and Methodological Problems in the Ranking of Universities by Bibliometric Methods. Scientometrics 62 (1): 133–143. https://doi.org/10.1007/s11192-005-0008-6.
———. 2005b. Reply to the Comments of Liu et al. Scientometrics 64 (1): 111–112. https://doi.org/10.1007/s11192-005-0242-y.
Reinalda, Bob. 2013. Global, Asian and European Backgrounds of Global University Rankings. In Global University Rankings. Challenges for European Higher Education, ed. Tero Erkkilä, 36–50. Basingstoke: Palgrave Macmillan.
Saisana, Michaela, and Andrea Saltelli. 2011. Rankings and Ratings: Instructions for Use. Hague Journal on the Rule of Law 3 (02): 247–268. https://doi.org/10.1017/S1876404511200058.
Saisana, Michaela, Béatrice d’Hombres, and Andrea Saltelli. 2011. Rickety Numbers: Volatility of University Rankings and Policy Implications. Research Policy, Special Section on Heterogeneity and University-Industry Relations, 40 (1): 165–177. https://doi.org/10.1016/j.respol.2010.09.003.
Skinner, Quentin. 1969. Meaning and Understanding in the History of Ideas. History and Theory 8 (1): 3–53.
Stubbs, Rebecca. 2009. The Millennium Challenge Account: Influencing Governance in Developing Countries Through Performance-Based Foreign Aid. Vanderbilt Journal of Transnational Law 42 (2): 621–682.
Sum, Ngai-Ling. 2009. The Production of Hegemonic Policy Discourses: ‘Competitiveness’ as a Knowledge Brand and Its (Re-)Contextualizations. Critical Policy Studies 3 (2): 184–203. https://doi.org/10.1080/19460170903385668.
Thomas, M.A. 2010. What Do the Worldwide Governance Indicators Measure? European Journal of Development Research 22 (1): 31–54. https://doi.org/10.1057/ejdr.2009.32.
Trapnell, Stephanie E. 2011. Actionable Governance Indicators: Turning Measurement into Reform. Hague Journal of the Rule of Law 3 (2): 317–348. https://doi.org/10.1017/S1876404511200095.
U-Multirank. 2017. Our Approach to Ranking/U-Multirank/Universities Compared. Your Way. Accessed June 2. http://www.umultirank.org/#!/about/methodology/approach-to-ranking?trackType=about&sightMode=undefined
Waltman, Ludo, Clara Calero-Medina, Joost Kosten, Ed C.M. Noyons, Robert J.W. Tijssen, Nees Jan van Eck, Thed N. van Leeuwen, Anthony F.J. van Raan, Martijn S. Visser, and Paul Wouters. 2012. The Leiden Ranking 2011/2012: Data Collection, Indicators, and Interpretation. Journal of the American Society for Information Science and Technology 63 (12): 2419–2432. https://doi.org/10.1002/asi.22708.
World Bank. 2007. Strengthening World Bank Group Engagement on Governance and Anticorruption. World Bank.
World Economic Forum. 2011. The Global Competitiveness Report 2011–2012. World Economic Forum. http://www3.weforum.org/docs/WEF_GCR_Report_2011-12.pdf
———. 2014. Global Competitiveness Report 2014–2015. World Economic Forum. http://www3.weforum.org/docs/WEF_GlobalCompetitivenessReport_2014-15.pdf
———. 2015. Global Competitiveness Report 2015–2016. World Economic Forum. http://www3.weforum.org/docs/gcr/2015-2016/Global_Competitiveness_Report_2015-2016.pdf
———. 2016. The Global Competitiveness Report 2016–2017. World Economic Forum. http://www3.weforum.org/docs/GCR2016-2017/05FullReport/TheGlobalCompetitivenessReport2016-2017_FINAL.pdf
Zanotti, Laura. 2005. Governmentalizing the Post—Cold War International Regime: The UN Debate on Democratization and Good Governance. Alternatives 30 (4): 461–487.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2018 The Author(s)
About this chapter
Cite this chapter
Erkkilä, T., Piironen, O. (2018). Field Structuration and Fragmentation of Global Rankings. In: Rankings and Global Knowledge Governance. Palgrave Studies in Global Higher Education. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-68941-8_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-68941-8_5
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-319-68940-1
Online ISBN: 978-3-319-68941-8
eBook Packages: EducationEducation (R0)