Abstract
Despite the excitement around the Education with Community Participation (EDUCO) program during the early 1990s as it was being scaled up, the Ministry of Education and the representatives of the World Bank knew that they would need solid evidence which demonstrated that the program produced beneficial outcomes in order (a) to continue to promote the program as a central policy for education reform in El Salvador and (b) to be able to credibly promote the program internationally as a best practice. It was in this context of excitement and determination that the World Bank began to carry out evaluations of EDUCO. With this in mind, this chapter critically reviews six key studies that were carried out between 1994 and 2005 by the World Bank on the EDUCO program. These six studies are included here for critical review because they represent each of the studies that were produced as impact evaluations, all of which were generated by the World Bank. They represent the entire body of “legitimate” and “policy-relevant” knowledge that was created in order to evaluate whether the program worked by identifying the effects of the EDUCO intervention.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
On this point, it is necessary to recall the tremendous popularity of decentralization as a theme within the international development industry during the 1990s (Edwards & DeMatthews, 2014) as well the predominance of the World Bank during this time vis-à-vis other multilateral organizations, especially in the education sector (Mundy, 1998).
- 2.
Meaning that the EDUCO schools and the traditional schools were selected for comparison after the EDUCO intervention began.
- 3.
Controls were included for family income, gender, and father’s ability to read (World Bank, 1994).
- 4.
Until 1999, the MINED allowed EDUCO and traditional schools to coexist through what it called “mixed” schools (Lindo Fuentes, 1998). Here, a traditional school would also receive funding from the EDUCO program, and the community would establish an ACE. With the funding received, the community would hire a teacher for lower grades (grades 1–3). The school would then use its non-EDUCO budget to pay a teacher to provide grades 4–6. As mentioned, the EDUCO program was initially targeted to communities where educational services were not provided during the civil war. However, as the MINED began to scale up the program, it did so by creating “mixed” schools, as well.
- 5.
It should be noted that Emmanuel Jimenez was at that time a member of the World Bank’s influential Development Economics Vice-Presidency and was well respected among his peers. The fact that he took up the cause of EDUCO by doing research on it attracted the attention of others within the Bank.
- 6.
The Heckman two-step correction is employed where there are issues with sample selection bias. Sample selection bias is normally a problem for regression analysis because, when the treatment and comparison groups systematically differ on a range of characteristics, we cannot be sure that the effect observed for the treatment (e.g., EDUCO) is unbiased. That is, the effect (or lack of it) could be due to unobserved variables related to the treatment (labeled by economists as endogeneity). Under endogeneity, we can also not be sure of the extent to which the estimates for the other variables in equation are biased as well (Berk, 1983). The best way to address this issue is to first draw a representative random sample of participants from the population of interest. In most cases this is not possible, especially in social science research. Such was the case with EDUCO, where participating communities were not chosen randomly and where participating communities differed systematically from a nationally representative sample.
Thus, to deal with this common issue, James Heckman developed his now famous correction technique which treats sample selection bias as specification error (Heckman, 1979). Per this technique, a separate estimator term is added to the principal regression equation of interest (i.e., the “substantive equation”). This term is arrived at by estimating through a (probit) selection equation the likelihood of a student, for example, being placed in an EDUCO school, given a set of selected characteristics. This probability, then, is fed back into the substantive equation as part of a ratio (the inverse Mills ratio) that represents the probability that a child with given characteristics will be excluded from the sample from which results are generalized, conditional on participation in EDUCO (Berk, 1983). Ideally, then, the added estimator is a sample selection correction term that adjusts for the fact that certain characteristics are overrepresented in the sample being examined. From the regression output, one can then interpret a statistically significant result for this term’s coefficient as indicating that there is sample selection bias. That is, a statistically significant coefficient signals that certain characteristics are over or underrepresented.
Fundamental issues surface in practice, however. For a thorough review of the many issues that can and do arise in practice through the use of the Heckman two-step correction, see Bushway, Johnson, and Slocum (2007). For example, if the same variables appear in both the selection equation and the substantive equation, the issue of multicollinearity arises (Bushway et al., 2007). That is, in concrete terms, if the same variables used to predict a student’s participation in EDUCO are also used to predict a student’s test scores, then the correction term added to the equation for test scores may well covary with the other independent variables. What’s more is that this technique assumes that the error terms for both the selection and substantive equation are jointly normal, meaning that they are independent of each other (Heckman, 1979). However, the error terms will be correlated if there are unobserved characteristics (omitted variables) relevant to the dependent variables of both the selection and substantive equations. It should be noted that, in the case of EDUCO, this is almost certainly the case. The reason is as follows: socioeconomic conditions determined who participated in EDUCO; these same poverty-related characteristics always influence student achievement. Ultimately, then, although this fix works well in theory, in practice it actually worsens issues of misspecification because there are now two linked regression equations to specify correctly and which themselves must not be correlated.
- 7.
Instrumental variables are employed in regression analysis under conditions of endogeneity. The idea is that one seeks to “find a variable (or instrument) that is highly correlated with program placement or participation but that is not correlated with unobserved characteristics affecting outcomes” (Khandker, Koolwal, & Samad, 2010, p. 87). Instrumental variables must be selected thoughtfully, as a weak instrumental variable—one that is correlated with unobserved characteristics or omitted variables—can worsen the bias of coefficient estimates. Ideally, the instrumental variable serves as a proxy for program participation while eliminating issues of endogeneity. See the previous footnote for more on endogeneity.
- 8.
The parents’ associations for traditional schools were known as Sociedades de Padres de Familia (SdPF) or Family Parent Societies (Sawada, 2000, p. 3).
- 9.
A revised version of this study was later published in the journal, Economic Development and Cultural Change (Jimenez & Sawada, 2014).
- 10.
Propensity score matching could not be run using all 42 variables because the communities were too dissimilar, according to Sawada and Ragatz (2005).
- 11.
In the words of Kahan (2013), “motivated reasoning refers to the tendency of people to conform assessments of information to some goal or end extrinsic to accuracy … The goal of protecting one’s identity or standing in an affinity group that shares fundamental values can generate motivated cognition relating to policy-relevant facts” (p. 408).
References
Achenbach, J. (2015, August 27). May scientific studies can’t be replicated. That’s a problem. The Washington Post. Retrieved from https://www.washingtonpost.com/news/speaking-of-science/wp/2015/08/27/trouble-in-science-massive-effort-to-reproduce-100-experimental-results-succeeds-only-36-times/
Barrera-Osorio, F., Fasih, T., Patrinos, H., & Santibánez, L. (2009). Decentralized decision-making in schools: The theory and evidence on school-based management. Washington, DC: The World Bank.
Berk, R. (1983). An introduction to sample selection bias in sociological data. American Sociological Review, 48(3), 368–398.
Bushway, S., Johnson, B., & Slocum, L. (2007). Is the magic still there? The use of the Heckman two-step correction for selection bias in criminology. Journal of Quantitative Criminology, 23(2), 151–178.
Edwards, D. B., Jr. (2018). The trajectory of global education policy: Community-based management in El Salvador and the global reform agenda. New York: Palgrave Macmillan.
Edwards, D. B., Jr., & DeMatthews, D. (2014). Historical trends in educational decentralization in the United States and developing countries: A periodization and comparison in the post-WWII context. Education Policy Analysis Archives, 22(40), 1–36. https://doi.org/10.14507/epaa.v22n40.2014.
Edwards, D. B., Jr., & Higa, S. (not yet published). The global education policy of school-based management in conflict-affected contexts: Current reach, prominent rationales, future research. Policy Futures in Education.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.
Jimenez, E., & Sawada, Y. (1999). Do community-managed schools work? An evaluation of El Salvador’s EDUCO program. The World Bank Economic Review, 13(3), 415–441.
Jimenez, E., & Sawada, Y. (2003). Does community management help keep kids in schools? Evidence using panel data from El Salvador’s EDUCO program. Discussion Paper, Center for International Research on the Japanese Economy. Tokyo: University of Tokyo.
Jimenez, E., & Sawada, Y. (2014). Does community management help keep kids in schools? Evidence using panel data from El Salvador’s EDUCO program. Economic Development and Cultural Change, 62(2), 307–338.
Kahan, D. (2013). Ideology, motivated reasoning, and cognitive reflection. Judgment and Decision Making, 8(4), 407–424.
Khandker, S., Kookwal, G., & Samad, H. (Eds.). (2010). Handbook on impact evaluation Quantitative methods and practices. Washington, DC: World Bank.
Klees, S., & Edwards, D. B., Jr. (2014). Knowledge production and technologies of governance. In T. Fenwick, E. Mangez, & J. Ozga (Eds.), World yearbook of education 2014: Governing knowledge: Comparison, knowledge-based technologies and expertise in the regulation of education (pp. 31–43). New York: Routledge.
Lindo Fuentes, H. (1998). Comunidad, participación y escuelas: EDUCO en El Salvador. [Community, participation and schools: EDUCO in El Salvador.]. Washington, DC: World Bank.
Mundy, K. (1998). Educational multilateralism and world (dis)order. Comparative Education Review, 42(4), 448–478.
Sawada, Y. (2000). Community participation, teacher effort, and educational outcome: The case of El Salvador’s EDUCO program. Michigan: The William Davidson Institute, University of Michigan Business School.
Sawada, Y., & Ragatz, A. (2005). Decentralization of Education, Teacher Behavior, and Outcomes: The Case of El Salvador’s EDUCO Program. In E. Vegas (Ed.), Incentive to improve teaching: Lessons from Latin America (pp. 255–306). Washington, DC: World Bank.
Umanzor et al. (1997). El Salvador’s EDUCO program: A first report on parents’ participation in school-based management. Working Paper Series on Impact Evaluation of Education Reforms, No. 4. Washington, DC: World Bank.
World Bank. (1994). El Salvador: Community education strategy: Decentralized school management. Washington, DC: The World Bank.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2018 The Author(s)
About this chapter
Cite this chapter
Edwards, D.B. (2018). Impact Evaluations of EDUCO: A Critical Review. In: Global Education Policy, Impact Evaluations, and Alternatives. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-75142-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-75142-9_5
Published:
Publisher Name: Palgrave Macmillan, Cham
Print ISBN: 978-3-319-75141-2
Online ISBN: 978-3-319-75142-9
eBook Packages: Political Science and International StudiesPolitical Science and International Studies (R0)