Do students make greater achievement gains in some higher education institutions’ programs than others? Insights from Brazil

  • Danilo Leite DalmonEmail author
  • Izabel Fonseca
  • Cláudio Pondé Avena
  • Martin Carnoy
  • Tatiana Khavenson


How much university students learn in their studies is highly debated and important to understanding the value of higher education. Yet, information on learning gains at this level are scarce. Our paper contributes to the debate by using unique data for Brazil to estimate absolute test score gains across various fields of study in higher education and to assess whether students who attend certain categories of programs (public/private, research/non-research, highly selective/less selective) make greater relative gains than in others. Our results suggest that students in STEM fields tend to have higher absolute achievement gains compared to students in humanities and pedagogical programs, and that in a few fields, such as civil engineering and history, the relative gains for students in highly selective programs in that field of study are significantly higher than if they had attended somewhat less selective programs. However, students attending lowest quintile selective programs in a field of study have consistently lower gains across a range of study fields than similar students attending programs just one quintile higher. The results have important implications for the equity effects of higher education.


Achievement gains Higher education accountability Propensity score matching Institutional selectivity 



The authors would like to thank the Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixera (INEP), the Stanford Lemann Center for Entrepreneurship and Innovation in Brazilian Education, and the Basic Research Program at the National Research University Higher School of Economics (HSE), which supported this project within the framework of a subsidy by the Russian Academic Excellence Project ”5-100.”

Supplementary material

10734_2019_376_MOESM1_ESM.docx (32 kb)
ESM 1 (DOCX 31 kb)


  1. Agasisti, T., & Johnes, G. (2009). Beyond frontiers: comparing the efficiency of higher education decision-making units across more than one country. Education Economics, 17(1), 59–79.CrossRefGoogle Scholar
  2. Altbach, P. G. (2015). AHELO: the myth of measurement and comparability. Internationall Higher Education, 82(Fall), 2–3.Google Scholar
  3. Anaya, G. (1999). College impact on student learning. Research in Higher Education, 40(5), 499–526.CrossRefGoogle Scholar
  4. Andrade, E., Moita, R., & Silva, C. (2009). A escolha da faculdade pelo aluno: estimação da demanda e precificação dos atributos. [The choice of college by the student: estimating demand and pricing of attributes] Working Paper, 182, 2009.Google Scholar
  5. Arum, R., & Roksa, J. (2011). Academically adrift: limited learning on college campuses. Chicago: University of Chicago Press.Google Scholar
  6. Astin, A. W. (1977). Four critical years. San Francisco: Joseey-Bass.Google Scholar
  7. Astin, A. W. (1993). What matters in college: four critical years revisited. San Francisco: Josssey-Bass.Google Scholar
  8. Berbegal, J., & Sole, F. (2012). What are we measuring when evaluating universities’ efficiency? Regional and Sectoral Economic Studies, 12(3), 31–46.Google Scholar
  9. Bittencourt, H. R., Viali, L., de Oliveira Casartelli, A., & Rodrigues, A. (2013). Uma análise da relação entre os conceitos Enade e IDD [An analysis of the relationship between ENADE scores and IDD]. Estudos em Avaliação Educacional, 19(40), 247–262.CrossRefGoogle Scholar
  10. Bok, D. (2013). Higher education in America. Princeton: Princeton University Press.Google Scholar
  11. Bowen, H. (2018). Investment in learning: the individual and social value of American higher education. Routledge.Google Scholar
  12. Carnoy, M., & Rothstein, R. (2013). What do international tests really show about US student performance. Washington, D.C.: Economic Policy Institute.Google Scholar
  13. Carnoy, M., Loyalka, P., Froumin, I., Dobryakova, M., Dossani, R., Kuhns, K., & Wang, R. (2013). University expansion in a changing global economy: triumph of the BRICs? Stanford: Stanford University Press.CrossRefGoogle Scholar
  14. Carrasco, R. & Carnoy, M. (2012). Achievement gains in Brazilian universities: the case of engineering and computer science programs. Stanford University (mimeo).Google Scholar
  15. Cunha, J. M., & Miller, T. (2012). Measuring value-added in higher education: possibilities and limitations in the use of administrative data. Economics of Education Review, 42, 64–77.CrossRefGoogle Scholar
  16. Dale, S. B., & Krueger, A. B. (2002). Estimating the payoff to attending a more selective college: an application of selection on observables and unobservables. The Quarterly Journal of Economics, 117(4), 1491–1527.CrossRefGoogle Scholar
  17. Deem, R., & Brehony, K. J. (2005). Management as ideology: the case of ‘new managerialism’in higher education. Oxford Review of Education, 31(2), 217–235.CrossRefGoogle Scholar
  18. Feldman, K. A. (1994). Review essay: what matters in college: four critical years revisited. Journal of Higher Education, 65(5), 615–622.Google Scholar
  19. Hanushek, E., Peterson, P., & Woessmann, L. (2013). Endangering prosperity: A global view of the American school. Washington, DC: Brookings Institution Press.Google Scholar
  20. Hastings, J. S., Neilson, C. A., & Zimmerman, S. D. (2013). Are some degrees worth more than others? Evidence from college admission cutoffs in Chile (No. w19241). National Bureau of Economic Research.Google Scholar
  21. Hénard, F. (2010). OECD feasibility study for AHELO. Paris: OECD.Google Scholar
  22. INEP, National Institute of Educational Research Anísio Teixeira (2007). Nota Técnica: Cálculo do Conceito Preliminar de Cursos de Graduação, [Technical note: calculation of the program preliminary concept].Google Scholar
  23. INEP, National Institute of Educational Research Anísio Teixeira. (2018). Sinopse Estadistica da Educação Superior 2017. Brasilia: INEP.Google Scholar
  24. Jencks, C., & Riesman, D. (1968). The academic revolution. Oxford: Doubleday.Google Scholar
  25. Ladd, H. (2008). Teacher effects: what do we know? In G. Duncan & J. Spillane (Eds.), Teacher quality: broadening the debate. Evanston: Northwestern University.Google Scholar
  26. Leitão, T., Moriconi, G., Abrão, M., & Silva, D. (2010). Uma análise acerca do boicote dos estudantes aos exames de avaliação da educação superior. [An analysis on the boycott of students to the higher education evaluation exams]. Revista Brasileira de Educação, Rio de Janeiro, 15(43), 21–44.CrossRefGoogle Scholar
  27. Lemos, K. C. S., & Miranda, G. J. (2014). Alto e Baixo Desempenho no ENADE - que variaveis explicam? [High and low performance at ENADE: which variables explain it?] In Proceedings of V Congresso UFSC. (pp. 1–19).Google Scholar
  28. Liu, O. L. (2011). Measuring value-added in higher education: conditions and caveats. Assessment & Evaluation in Higher Education, 36(1), 81–94.CrossRefGoogle Scholar
  29. Liu, O. L., Frankel, L., & Roohr, K. C. (2014). Assessing critical thinking in higher education: current state and directions for next-generation assessment. ETS Research Report Series, 2014(1), 1–23.CrossRefGoogle Scholar
  30. Loyalka, P. Liu, O.L, Li, G.R., Kardanova, E., Chirikov, I., Hu, S., Yu, N. et al (2019). Skills in college: evidence from China, India, Russia, and the United States. Stanford University, Working Paper.Google Scholar
  31. Lynch, K. (2006). Neo-liberalism and marketisation: the implications for higher education. European Educational Research Journal, 5(1), 2006.CrossRefGoogle Scholar
  32. Mayhew, M., Rockenbach, A. M., Bowman, N. A., Seifert, T. A., & Wolniak, G. C. (2016). How college affects students (Vol. 3). San Francisco: Jossey-Bass.Google Scholar
  33. McMahon, W. (2009). Higher learning, greater good. Baltimore: Johns Hopkins Press.Google Scholar
  34. Melguizo, T., & Wainer, J. (2016). Toward a set of measures of student learning outcomes in higher education: evidence from Brazil. Higher Education, 72(3), 381–401.CrossRefGoogle Scholar
  35. Meyer, J. W., Ramirez, F. O., Frank, D. J., & Schofer, E. (2007). Higher education as an institution. In P. Gumport (Ed.), Sociology of higher education: contributions and their contexts. Baltimore: Johns Hopkins Press.Google Scholar
  36. Moreira, A. M. de A (2010). Fatores Institucionais e Desempenho Academico no ENADE - biologia, engenharia civil, história e pedagogia (Ph.D. dissertation, Universidade de Brasília, Brasília, Brazil).Google Scholar
  37. Moriconi, G. M., Nascimento, P. A. M. M., (2014). Fatores Associados ao Desempenho dos Concluintes de Engenharia no ENADE 2011 [Factors associated to performance of graduating students of engineering in ENADE 2011]. Estudos de Avaliacao da Educacao, 25(57).Google Scholar
  38. Neave, G. (2012). The evaluative state, institutional autonomy and re-engineering higher education in Western Europe. London: Palgreave McMillan.CrossRefGoogle Scholar
  39. Organization for Economic Cooperation and Development (OECD). (2017). Benchmarking higher education system performance: Conceptual framework and data. Enhancing Higher Education System Performance. Paris: OECD.Google Scholar
  40. Paiva, G. S. (2009). Avaliação do desempenho dos estudantes da educação superior: a questão da equidade e obrigatoriedade no Provão e ENADE [Evaluation and performance of higher education students: equity and mandatory issues in Provao and ENADE]. Ensaio: Avaliação e Políticas Públicas em Educação, 16(58), 31–46.Google Scholar
  41. Pascarella, E. T., & Terenzini, P. T. (1998). Studying college students in the 21st century: meeting new challenges. The Review of Higher Education, 21(2), 151–165.Google Scholar
  42. Pusser, B., & Marginson, S. (2013). University rankings in critical perspective. The Journal of Higher Education, 84(4), 544–568.CrossRefGoogle Scholar
  43. Rezende, M. (2010). The effects of accountability on higher education. Economics of Education Review, 29(5), 842–856.CrossRefGoogle Scholar
  44. Saavedra, A. R., & Saavedra, J. E. (2011). Do colleges cultivate critical thinking, problem solving, writing and interpersonal skills? Economics of Education Review, 30(6), 1516–1526.CrossRefGoogle Scholar
  45. Schwartzman, S. (2008). O “conceito preliminar” e as boas práticas de avaliação do ensino superior. [The “preliminary concept” and the good practices of higher education evaluation]. Estudos: Revista da Associação Brasileira de Mantenedoras de Ensino Superior, 26(38), 9–22.Google Scholar
  46. Schwartzman, J., & Schwartzman, S. (2002). O ensino superior privado como setor econômico. [Private higher education as an economic sector]. Ensaio–Avaliação e Políticas Públicas em Educação, 10(37), 411–440.Google Scholar
  47. Spellings, M. (2006). A test of leadership. Washington, D.C.: US Department of Education, Report of the Commission on the Future of Higher Education.Google Scholar
  48. Times Higher Education (2018). World University Rankings 2018 methodology. Accessed 1 March 2019.
  49. Verhine, R., Dantas, L. M. V., & Soares, J. F. (2006). Do Provão ao ENADE: uma análise comparativa dos exames nacionais utilizados no Ensino Superior Brasileiro [Assessment of higher education in Brazil: from the Provão to Enade]. Ensaio: Avaliação e Políticas Públicas em Educação, 14(52), 291.Google Scholar
  50. Weiler, H. N. (2011). Knowledge and power. Journal of Educational Planning and Administration, 25(3), 205–221.Google Scholar
  51. Zlatkin-Troitschanskaia, O., Shavelson, R. J., & Kuhn, C. (2015). The international state of research on measurement of competency in higher education. Studies in Higher Education, 40(3), 393–411.CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • Danilo Leite Dalmon
    • 1
    Email author
  • Izabel Fonseca
    • 2
  • Cláudio Pondé Avena
    • 3
  • Martin Carnoy
    • 2
  • Tatiana Khavenson
    • 4
  1. 1.Ministry of EducationBrasiliaBrazil
  2. 2.Stanford UniversityStanfordUSA
  3. 3.Fundacão Getulio VargasRio de JaneiroBrazil
  4. 4.National Research University Higher School of EconomicsMoscowRussia

Personalised recommendations