Advertisement

When do performance assessments influence policy behavior? Micro-evidence from the 2014 Reform Efforts Survey

  • Takaaki MasakiEmail author
  • Bradley C. Parks
Article

Abstract

Sovereign governments, multilateral institutions, and non-governmental organizations have created a wide array of tools to assess the policy performance of developing countries, including conditional aid and debt relief programs, blacklists, watch lists, performance-based organizational accession procedures, cross-country benchmarking exercises, and country-specific diagnostics. Scholars and policymakers generally agree that these types of performance assessments can influence the policy priorities and actions of public sector decision-makers in low-income and middle-income countries. However, there is little systematic evidence about the specific conditions under which performance assessments instigate changes in policy behavior. There is also a lack of understanding about the causal mechanisms through which different types of performance assessments facilitate policy changes. We seek to close this evidence gap by leveraging a survey of 1784 government officials that provides comparative data on the agenda-setting influence and reform design influence of more than 100 performance assessments in 123 low-income and middle-income countries. Using a multilevel linear model that accounts for the hierarchical structure of the survey data, we find robust evidence that performance assessments yield greater policy influence when they make an explicit comparison of government performance across countries and allow assessed governments to participate in the assessment process. This pattern is consistent with the theoretical argument that policymakers carefully weigh the credibility signaling benefits and policy autonomy costs of addressing performance assessment requirements.

Keywords

Performance assessment Sovereignty Credibility Reform FDI Elite survey 

Notes

Acknowledgments

We thank Dan Honig, Eddy Malesky, Elise Le, Steven Bernstein, Anton Strezhnev, Judith Kelley, and Beth Simmons for providing helpful comments on previous drafts of the manuscript. We also thank NORC at the University of Chicago—in particular, Renee Hendley, Kate Hobson, David Kordus, Elise Le, Aparna Ramakrishnan, Alex Rigaux, Bhanuj Soni, and Stacy Stonich—for their survey design and data collection assistance. Additionally, we owe a debt of gratitude to the nearly 6,750 development policymakers and practitioners from 126 countries who participated in the 2014 Reform Efforts Survey. Were it not for their willingness to graciously share of their experiences and insights, this study would not have been possible. This study was also made possible through generous financial support from the John Templeton Foundation and the Smith Richardson Foundation. The findings, interpretations, and conclusions expressed in this publication are those of the authors and should not be attributed in any manner to the College of William and Mary, AidData, The World Bank, its Board of Executive Directors, or the governments they represent.

Supplementary material

11558_2018_9342_MOESM1_ESM.docx (864 kb)
ESM 1 (DOCX 863 kb)

References

  1. Abiad, A., & Mody, A. (2005). Financial reform: What shakes it? What shapes it? American Economic Review, 95(1), 66–88.Google Scholar
  2. Acemoglu, D., Collier, P., Johnson, S., Klein, M., Wheeler, G. (2013). A review of doing business. Mimeo.Google Scholar
  3. Alesina, A., & Dollar, D. (2000). Who gives foreign aid to whom and why? Journal of Economic Growth, 5(1), 33–63.Google Scholar
  4. Andrews, M. (2011). Which organizational attributes are amenable to external reform? An empirical study of African public financial management. International Public Management Journal, 14(2), 131–156.Google Scholar
  5. Andrews, M. (2013). The limits of institutional reform in development: Changing rules for realistic solutions. Cambridge: Cambridge University Press.Google Scholar
  6. Arndt, C., & Oman, C. (2006). Uses and abuses of governance indicators. Paris: OECD.Google Scholar
  7. Arruñada, B. (2007). Pitfalls to avoid when measuring institutions: Is doing business damaging business? Journal of Comparative Economics, 35(4), 729–747.Google Scholar
  8. Bagaudinova, S., Omran, D., Shavurov, U. (2007). Georgia: Licensing 159 activities—Not 909. In Celebrating Reform 2007. Washington DC: World Bank.Google Scholar
  9. Bendukidze, K., Saakhashvili, M. (2014). Georgia The Most Radical Catch-Up Reforms, In Anders Aslund and Simeon Djankov, eds. The Great Rebirth: The Victory of Capitalism over Communism, Peterson Institute for International Economics, November.Google Scholar
  10. Besley, T. (2015). Law, regulation, and the business climate: The nature and influence of the World Bank doing business project. Journal of Economic Perspectives, 29(3), 99–120.Google Scholar
  11. Besley, T., & Burgess, R. (2002). The political economy of government responsiveness: Theory and evidence from India. The Quarterly Journal of Economics, 117(4), 1415–1451.Google Scholar
  12. Biglaiser, G., & DeRouen, K. (2010). The effects of IMF programs on U.S. foreign direct Investment in the Developing World. Review of International Organizations, 5(1), 73–95.Google Scholar
  13. Bisbee, J.H., Hollyer, J.R., Rosendorff, B.P., Vreeland, J.R. (2017). The millennium development goals and education: Accountability and substitution in global assessment. Mimeo.Google Scholar
  14. Brooks, S., Cunha, R., & Mosley, L. (2015). Categories, creditworthiness and contagion: How investors’ shortcuts affect sovereign debt markets. International Studies Quarterly, 59(3), 587–601.Google Scholar
  15. Bueno de Mesquita, B., & Smith, A. (2010). Leader survival, revolutions and the nature of government finance. American Journal of Political Science, 54, 936–950.Google Scholar
  16. Buntaine, M., Parks, B., & Buch, B. (2017). Aiming at the wrong targets: The domestic consequences of international efforts to build institutions. International Studies Quarterly, 61(2), 471–488.Google Scholar
  17. Busia, K. (2014). The Search for Domestic Accountability in Development Aid: What Role for the APRM in Reshaping Governance in Africa? In M. Ndulo & N. van de Walle (Eds.), Problems, Promises, and Paradoxes of Aid: Africa’s Experience (pp. 171–210). Newcastle upon Tyne: Cambridge Scholars Publishing/Institute for African Development.Google Scholar
  18. Büthe, T., & Milner, H. V. (2008). The politics of foreign direct investment into developing countries: Increasing FDI through international trade agreements. American Journal of Political Science, 52(October), 741–762.Google Scholar
  19. Chassy, A. (2014). Civil Society and Development Effectiveness in Africa. In Problems, Promises, and Paradoxes of Aid: Africa’s Experience. Newcastle upon Tyne: Cambridge Scholars Publishing.Google Scholar
  20. Chhotray, V., & Hulme, D. (2009). Contrasting visions for aid and governance in the 21st century: The white house millennium challenge account and DFID’s drivers of change. World Development, 37(1), 36–49.Google Scholar
  21. Chwieroth, J. (2007). Neoliberal economists and capital account liberalization in emerging markets. International Organization, 61(2), 443–463.Google Scholar
  22. Chwieroth, J. M. (2013). The silent revolution: How the staff exercise informal governance over IMF lending. Review of International Organizations, 8(2), 265–290.Google Scholar
  23. Corcoran, A., & Gillanders, R. (2015). Foreign direct investment and the ease of doing business. Review of World Economics, 151(1), 103–126.Google Scholar
  24. Crescenzi, M. (2007). Reputation and international conflict. American Journal of Political Science, 51, 382–396.Google Scholar
  25. Custer, S., Masaki, T., Sethi, T., Latourell, R., Rice, Z., & Parks, B. (2016). Governance Data: Who Uses It and Why? Williamsburg: AidData and the Governance Data Alliance.Google Scholar
  26. David-Barrett, E., & Okamura, K. (2016). Norm diffusion and reputation: The rise of the extractive industries transparency initiative. Governance, 29(2), 227–246.Google Scholar
  27. De Waal, A. (2011). Georgia’s choices: Charting a future in uncertain times. Washington DC: Carnegie Endowment for International Peace.Google Scholar
  28. Di Puppo, L. (2014). The construction of success in anti-corruption activity in Georgia. East European Politics, 30(1), 105–122.Google Scholar
  29. Dreher, A., & Voigt, S. (2011). Does membership in international organizations increase governments’ credibility? Testing the effects of delegating powers. Journal of Comparative Economics, 39(3), 326–348.Google Scholar
  30. Dreher, A., Nunnenkamp, P., & Öhler, H. (2012). Why it pays for aid recipients to take note of the Millennium Challenge Corporation: Other donors do! Economics Letters, 115, 373–375.Google Scholar
  31. Easterly, William. And Williamson, Claudia R. 2011. Rhetoric vs. reality: The best and worst of aid agency practices. World Development 39: 1930–1949.Google Scholar
  32. Elkins, Z., Guzman, A. T., & Simmons, B. A. (2006). Competing for capital: The diffusion of bilateral investment treaties, 1960-2000. International Organization 60(4), 811–846.Google Scholar
  33. Elkins, Z., Guzman, A. T., & Simmons, B. A. (2008). Competing for capital: The diffusion of bilateral investment treaties, 1960­2000. In The Global Diffusion of Markets and Democracy. Cambridge: Cambridge University Press.Google Scholar
  34. Etter, L. (2014). Can transparency reduce corruption? Evidence from firms in Peru and Mali on the impact of the extractive industries transparency initiative (EITI) on corruption. Paper presented at The Doing Business Conference at Georgetown University in Washington DC on 2014 February 20–21.Google Scholar
  35. Extractive Industries Transparency Initiative (EITI) International Secretariat. (2015). Quantifying intangibles using the EITI in credit rating assessments. Oslo: EITI International Secretariat.Google Scholar
  36. Fariaa, A., & Mauro, P. (2009). Institutions and the external capital structure of countries. Journal of International Money and Finance, 28(3), 367–391.Google Scholar
  37. Fenton Villar, P., & Papyrakis, E. (2017). Evaluating the impact of the extractive industries transparency initiative (EITI) on corruption in Zambia. The Extractive Industries and Society, 4(4), 795–805.Google Scholar
  38. Flores, T., Lloyd, G., Nooruddin, I. (2015). The Technocratic Advantage: Does Technocratic Leadership Attract International Financing? Working Paper.Google Scholar
  39. Gainer, M., Chan, S., & Skoet, L. (2016). Faster together: A one-stop shop for business registration in Senegal, 2006–2015. Princeton: Innovations for Successful Societies.Google Scholar
  40. Garriga, A. C., & Phillips, B. J. (2013). Foreign aid as a signal to investors predicting FDI in post-conflict countries. Journal of Conflict Resolution, 58(2), 280–306.Google Scholar
  41. Girod, D. M., & Tobin, J. (2016). Take the money and run: The determinants of compliance with aid agreements. International Organization, 70(1), 209–239.Google Scholar
  42. Goldsmith, A. A. (2011). No country left behind? Performance standards and accountability in US foreign assistance. Development Policy Review, 29, s157–s176.Google Scholar
  43. Gray, J. (2009). International organization as a seal of approval: European Union accession and investor risk. American Journal of Political Science, 53(4), 931–949.Google Scholar
  44. Gray, J. (2013). The company states keep. International economic organization and sovereign risk in emerging markets. Cambridge: Cambridge University Press.Google Scholar
  45. Gray, J., & Slapin, J. (2012). How effective are preferential trade agreements? Ask the experts. Review of International Organizations, 7(3), 309–333.Google Scholar
  46. Gregory, N. (2012). Doing business and doing development. World Economics, 13(3), 13–26.Google Scholar
  47. Grindle, M. S., & Thomas, J. W. (1991). Public choices and policy change: The political economy of reform in developing countries. Baltimore: John Hopkins University Press.Google Scholar
  48. Hafner-Burton, E. M. (2008). Sticks and stones: Naming and shaming the human rights enforcement problem. International Organization, 62(4), 689–716.Google Scholar
  49. Hallward-Driemeier, M., & Pritchett, L. (2015). How business is done in the developing world: Deals versus rules. Journal of Economic Perspectives, 29(3), 121–140.Google Scholar
  50. Harding, R. (2013). China seeks to water down key World Bank report. Financial Times May 6, 2013.Google Scholar
  51. Hoffmann-Lange, U. (2007). Methods of elite research. In R. J. Dalton & H.-D. Klingemann (Eds.), Oxford handbook of political behavior (pp. 910–927). Oxford: Oxford University Press.Google Scholar
  52. Hollyer, J. R., & Rosendorff, B. P. (2012). Leadership survival, regime type, policy uncertainty and PTA accession. International Studies Quarterly, 56(4), 748–764.Google Scholar
  53. Hollyer, J. R., Rosendorff, B. P., & Vreeland, J. R. (2011). Democracy and transparency. Journal of Politics, 73(4), 1191–1205.Google Scholar
  54. Independent Evaluation Group of The World Bank (IEG). (2008). Doing Business: An Independent Evaluation. Taking the Measure of The World Bank-IFC Doing Business Indicators. Washington D.C.: World Bank.Google Scholar
  55. Independent Evaluation Group of The World Bank (IEG). (2009). The World Bank in Georgia 1993–2007: Country Assistance Evaluation. Washington D.C.: World Bank.Google Scholar
  56. International Monetary Fund. (2009). Georgia: 2009 article IV consultation and second review under the stand-by arrangement—Staff report; press release and public information notice on the executive board discussion; and statement by the executive director for Georgia. IMF Country Report No. 09/127.Google Scholar
  57. Jacoby, W. (2006). Inspiration, coalition, and substitution: External influences on Postcommunist transformations. World Politics, 58(4), 623–651.Google Scholar
  58. Jensen, N. M. (2003). Democratic governance and multinational corporations: Political regimes and inflows of foreign direct investment. International Organization, 57(3), 587–616.Google Scholar
  59. Johnson, K., Greenseid, L. O., Toal, S. A., King, J. A., Lawrenz, F., & Volkov, B. (2009). Research on Evaluation use: A review of the empirical literature from 1986 to 2005. American Journal of Evaluation, 30(3), 377–410.Google Scholar
  60. Kapstein, E. B., & Converse, N. (2008). The fate of young democracies. Cambridge: Cambridge University Press.Google Scholar
  61. Kelley, J. G. (2017). Scorecard diplomacy: Grading states to influence their reputation and behavior. Cambridge: Cambridge University Press.Google Scholar
  62. Kelley, J. G., & Simmons, B. A. (2015). Politics by number: Indicators as social pressure in international relations. American Journal of Political Science, 59(1), 55–70.Google Scholar
  63. Kelley, J.G., Simmons B.A. (2016). Introduction: Global assessment power in the twenty-first century. Paper prepared for the annual meeting of the American Political Science Association. Philadelphia, PA. September 1–4, 2016.Google Scholar
  64. Kelley, J.G., Simmons, B.A., Doshi, R. (2017). The power of ranking: The ease of doing business Indicator as a form of social pressure. Mimeo.Google Scholar
  65. Kerner, A., Jerven, M., & Beatty, A. (2017). Does it pay to be poor? Testing for systematically underreported GNI estimates. The Review of International Organizations, 12(1), 1–38.Google Scholar
  66. Kijima, R., Lipscy, P.Y. (2017). The politics of international testing. Mimeo.Google Scholar
  67. Knack, S. (2013). Aid and donor Trust in Recipient Country Systems. Journal of Development Economics, 101, 316–329.Google Scholar
  68. Knecht, T. (2010). Paying Attention to Foreign Affairs: How Public Opinion Affects Presidential Decision Making. University Park: Penn State University Press.Google Scholar
  69. Koliev, F., Sommerer, T., Tallberg, J. (2017). Reporting matters: Performance assessment and compliance in the ILO. Mimeo.Google Scholar
  70. Krasner, S. D. (2011). Foreign aid: Competing paradigms. Journal of Intervention and Statebuilding, 5(2), 123–149.Google Scholar
  71. Kydd, A. (2000). Trust, reassurance, and cooperation. International Organization, 54(2), 325–357.Google Scholar
  72. Lake, D., & Farris, C. J. (2014). Why international trusteeship fails: The politics of external Authority in Areas of limited statehood. Governance, 27(4), 569–587.Google Scholar
  73. Lebovic, J. H., & Voeten, E. (2009). The cost of shame: International organizations and foreign aid in the punishing of human rights violators. Journal of Peace Research, 46(1), 79–97.Google Scholar
  74. Lee, C.-y. (2017). Terrorism, counterterrorism aid, and Foreign Direct Investment. Foreign Policy Analysis, 13(1), 168–187.Google Scholar
  75. Licht, A. (2010). Coming into money: The impact of foreign aid on leader survival. Journal of Conflict Resolution, 54(1), 58­87.Google Scholar
  76. Light, M. (2014). Police reforms in the republic of Georgia: The convergence of domestic and foreign policy in an anti-corruption drive. Policing and Society, 24(3), 318–345.Google Scholar
  77. Loayza, N., Rigolini, J., & Llorente, G. (2012). Do middle classes bring institutional reforms? World Bank policy research working paper 6015. Washington DC: World Bank.Google Scholar
  78. Lombardi, D., & Woods, N. (2008). The politics of influence: An analysis of IMF surveillance. Review of International Political Economy, 15(5), 709–737.Google Scholar
  79. Luongo, M. (2012). A rush to do business in the Middle East. July 16, 2012. New York Times. Google Scholar
  80. Martin, F., Mondragón-Vélez, C, Vila, L. (2014). Investing on Business Climate and Reform in Emerging Markets: An equity portfolio performance Evaluation. Paper presented at the doing business: Past, present and future of business regulation conference at Georgetown University’s McDonough School of Business. Washington, DC. February 20–21, 2014.Google Scholar
  81. Mcmillan, J., & Zoido, P. (2004). How to subvert democracy: Montesinos in Peru. Journal of Economic Perspectives, 18(4), 69–92.Google Scholar
  82. Merchant-Vega, N., & Malesky, E. (2011). A peek under the engine Hood: The methodology of subnational economic governance indices. Hague Journal on the Rule of Law, 3(2), 186–219.Google Scholar
  83. Mitchell, L. A. (2009). Uncertain democracy: U.S. foreign policy and Georgia's rose revolution. Philadelphia: University of Pennsylvania Press.Google Scholar
  84. Morse, J.C. (2017). Institutional monitoring and market enforcement: Generating compliance in the regime to combat terrorist financing. Mimeo.Google Scholar
  85. Murdie, A., & Peksen, D. (2015). Women’s rights INGO shaming and the government respect for Women’s rights. Review of International Organizations, 10(1), 1–22.Google Scholar
  86. Nelson, S.C. (2009). Endangered species: Exploring the political survival of economic policymakers. Paper presented at the Cornell PSAC seminar. April 2009.Google Scholar
  87. Nelson, S. C. (2017). The currency of confidence: How economic beliefs shape the IMF’s relationship with its borrowers. Ithaca: Cornell University Press.Google Scholar
  88. Noland, M. (1997). Chasing phantoms: The political economy of USTR. International Organization, 51, 365–387.Google Scholar
  89. Öhler, H., Nunnenkamp, P., & Dreher, A. (2012). Does conditionality work? A test for an innovative US aid scheme. European Economic Review, 56, 138–153.Google Scholar
  90. Parks, B., Davis, C. (forthcoming). When do governments trade domestic reforms for external rewards? Explaining policy responses to the Millennium Challenge Corporation’s eligibility standards. Governance..Google Scholar
  91. Parks, B., Rice, Z., & Custer, S. (2015). Marketplace of ideas for policy change: Who do developing world leaders listen to and why? Williamsburg: AidData and the College of William and Mary.Google Scholar
  92. Patton, M. Q. 2008. Utilization focused evaluation. 4th Edition. Saint Paul, MN: Sage Publishing.Google Scholar
  93. Pevehouse, J. C. (2002). With a little help from my friends? Regional organizations and the consolidation of democracy. American Journal of Political Science, 46(3), 611–626.Google Scholar
  94. Pitlik, H., Frank, B., & Firchow, M. (2010). The demand for transparency: An empirical note. Review of International Organizations, 5, 177–195.Google Scholar
  95. Pop-Eleches, G. (2009). From Economic Crisis to Reform: IMF Programs in Latin American and Eastern Europe. Princeton: Princeton University Press.Google Scholar
  96. Quiroz Flores, A., & Smith, A. (2013). Leader survival and natural disasters. British Journal of Political Science, 43(4), 821–843.Google Scholar
  97. Roberts, J., Tellez, J. (2017). Freedom House’s scarlet letter: Assessment power through transnational pressure. Mimeo.Google Scholar
  98. Rodrik, D. (1996). Why is there multilateral lending? In B. M. Arvin (Ed.), Annual World Bank conference on development economics. Westport: Praeger.Google Scholar
  99. Rodrik, D. (2014). When ideas trump interests: Preferences, worldviews, and policy innovations. Journal of Economic Perspectives, 28(1), 189–208.Google Scholar
  100. Russ-Eft, D., & Preskill, H. (2009). Evaluation in organizations: A systematic approach to enhancing learning, performance, and change (Second ed.). Cambridge: Perseus.Google Scholar
  101. Sandefur, J., & Glassman, A. (2015). The political economy of bad data: Evidence from African survey and administrative statistics. Journal of Development Studies, 51(2), 116–132.Google Scholar
  102. Savedoff, W. (2011). Incentive Proliferation? Making Sense of a New Wave of Development Programs. Washington D.C.: Center for Global Development.Google Scholar
  103. Schueth, S. J. (2011). Assembling international competitiveness: The republic of Georgia, USAID, and the Doing Business Project. Economic Geography, 37(1), 51–77.Google Scholar
  104. Solt, F. (2008). Economic inequality and democratic political engagement. American Journal of Political Science, 52(1), 48–60.Google Scholar
  105. Svensson, J. (1998). Investment, property rights and political instability: Theory and evidence. European Economic Review, 42, 1317–1341.Google Scholar
  106. Swedlund, H. (2013). The domestication of governance assessments: Evidence from the Rwandan joint governance assessment. Conflict, Security & Development, 13(4), 449–470.Google Scholar
  107. Tansey, O. (2007). Process tracing and elite interviewing: A case for non-probability sampling. PS: Political Science and Politics, 40(4), 765–772.Google Scholar
  108. Tomz, M. (2007). Reputation and international cooperation: Sovereign debt across three centuries. Princeton: Princeton University Press.Google Scholar
  109. Ubaldi, B. (2013). Open government data: Towards empirical analysis of open government data initiatives. OECD Working Papers on Public Governance, No. 22.Google Scholar
  110. UNDP. (2008). A users’ guide to measuring corruption. UNDP Oslo Governance Centre.Google Scholar
  111. USAID. (2009). Georgia: Opened for business. Georgia business climate reform final report, Washington DC: USAID. Accessed at http://pdf.usaid.gov/pdf_docs/Pdacn591.pdf
  112. Vreeland, J. R. (2003). The IMF and economic development. Cambridge: Cambridge University Press.Google Scholar
  113. Weyland, K. G. (2005). Theories of policy diffusion: Lessons from Latin American pension reform. World Politics, 57(2), 262–295.Google Scholar
  114. World Bank. (2007). Celebrating Reform 2007: Doing Business Case Studies. Washington D.C.: World Bank.Google Scholar
  115. World Bank. (2012). Fighting corruption in public services: Chronicling Georgia’s reforms. Washington DC: World Bank.Google Scholar
  116. World Bank Independent Review Panel. (2013). Independent panel review of the doing business report. Washington, DC: World Bank.Google Scholar
  117. Zabanova, Y. (2011). Georgia's post-Soviet Libertarian elite: Choosing a development model and marketing the strategy. Working paper at the conference “The Caucasus and Central Asia, twenty years after independences: Questioning the notion of 'South countries', August 25–27, 2011, Almaty, Kazakhstan.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.The World BankWashingtonUSA
  2. 2.AidData, Global Research InstituteThe College of William and MaryWilliamsburgUSA

Personalised recommendations