Introduction

Accreditation of higher education in Chile has received considerable attention in the last decade but remains, as elsewhere, controversial (see Arango et al., 2016; Barroilhet, 2019; Ceaser, 2004; Espinoza and González, 2012, 2013; Jerez et al., 2018; Letelier and Carrasco, 2004; Rojas and López, 2016; Rodríguez-Ponce et al., 2010). It swiftly evolved from a low-stakes to a high-stakes model when institutional accreditation became an eligibility requirement for state-guaranteed students loans. Now the system functions in an increasingly large and diverse environment characterized by tensions between several groups of institutions (Barroilhet, 2019; Jerez et al., 2018).

One of the largest sources of friction exists between the group of traditional universities congregated in the Consejo de Rectores de Universidades Chilenas [Council of Rectors of Chilean Universities] (CRUCh) that congregates the oldest and most prestigious public and private universities, and the newer private universities that sprung up after Pinochet’s reforms of 1981. The military intervention transformed the previously non-competitive higher education sector into a highly segmented system driven by market competition (Capel, 2005; Cox, 1996). Under the pressure of competition, some of the newer private institutions excelled, while some traditional institutions experienced shortages of public funding. Financial problems also surfaced among less-selective private institutions as they opened regional branches, often with insufficient funding and poor infrastructure, becoming diploma mills (Garrido Rojas, 2011; Geoffroy Pitta, 2013; Rojas Marín, 2011). Accreditation was, in its inception, a soft-law approach to address the decreasing quality of these institutions at the bottom of the hierarchy in both CRUCh and newer institutions (Barroilhet, 2019).

As elsewhere, accreditation did not start in full force. The government initially adopted what Westerheijden, Stensaker, Rosa and Corbett describe as a catwalk quality assurance policy (Westerheijden et al., 2014). The fashion metaphor alludes to models that are adopted, for the most part, because they look good and fashionable in the eyes of some relevant constituencies, with a lesser emphasis on whether they focus on quality assurance. The choice was an experimental and low-stakes approach to the accreditation of institutions and academic programmes overviewed by government-appointed experts. No financial consequences were attached to these accreditation decisions.

During this initial phase, many new academic programmes walked the catwalk, but only a few institutions did (Comisión Nacional de Acreditación de Pregrado (CNAP), Ministerio de Educación, 2007). However, a massive new programme of state-guaranteed student loans, which established institutional accreditation as an eligibility requirement, was announced in 2004. This requirement made accreditation imperative because it could leave non-accredited institutions without demand in the near future (Bernasconi, 2015). The announcement launched a two-year debate over who would control the accreditation system. The debate ended in a compromise in Congress between those who wanted less direct public funding for CRUCh institutions and more loans, and those who wanted to increase public funding for public institutions.1 The compromise resulted in the creation of a public agency in charge of overviewing all accreditation procedures, the Comisión Nacional de Acreditación (CNA). Seven of the 15 board members (‘commissioners’) of the CNA council were required to be experienced faculty members in their own fields, some appointed by CRUCh universities and some others appointed by new private institutions. The remaining members came from civil society and public administration. The statute, anticipating potential conflict of interests, required CNA commissioners ‘not to act on the representation of the organizations that elected them’ (Law No. 20129 of 2006, Art. 7) and forbade them to participate in the designation of peers for their own institutions and their subsequent accreditation judgment. The composition of the CNA aimed to ensure that potential representational conflicts of interest natural to a multi-stakeholder process would balance out to prevent the natural tendency to lean towards their appointing conglomerate.

In some nations, accreditation is oriented towards quality enhancement and transparency and does involve stakeholders, but is not consequential in terms of financing (Beerkens, 2015). In others, stakeholders are involved in quality agencies, like Austria or Finland, but do not engage directly on decision-making bodies. However, in Chile accreditation judgments are mandatory for institutions wanting to enrol students with state-guaranteed student loans and scholarships, which is a must for those institutions wanting to remain competitive. Hence, closer scrutiny should be aimed at detecting and controlling conflicts of interest in the decisions to accredit or not an institution of higher education in nations where high-stakes accreditation is implemented.

Though the meaning of quality seems a never-ending theme (Harvey and Green, 1993; Milliken and Colohan, 2004; Newton, 2002, 47), the aim of accreditation is universally understood as quality assurance, or quality control, or quality improvement, or a combination thereof (Schindler et al., 2015). Whatever definition is adopted, accreditation as quality assurance can be compromised by commissioners’ biases and potential conflicts of interests (PCI). In this sense, our research is an exercise that seeks to ‘evaluate the evaluators’ and to engage academically in the ‘politics of quality’ (Newton, 2002). Accreditation and its standing outcomes rest on the premise that accreditation judgments reflect a quality measure, regardless of the evaluators’ inclination concerning the type of institution evaluated.

During the period analysed the fifteen commissioners of the CNA were selected according to quotas, as shown in Table 1.

Table 1 CNA’s commissioners: requisites, number of commissioners and appointing conglomerate.
Full size table

The CRUCh conglomerate designated the highest number of commissioners. No objective criteria defined the number of commissioners per conglomerate (i.e. size or number of associated institutions). The composition of commissioners was the result of a political negotiation (Geoffroy Pitta, 2013; Scharager Goldenberg, 2018). The term of appointment for the 15 commissioners was four years and limited to two consecutive terms.2 Commissioners had a fundamental and consequential task: deciding whether to grant (or not grant) accreditation and, indirectly, gatekeeping access to public funding of programmes and institutions (Law No 20027, 2005). Subsequently, commissioners were required to specify the number of years until reaccreditation, from 1 for the weakest institutions to 7 for those that could demonstrate excellence in teaching, research and outreach. Thus, their votes could affect the standing of institutions in the opinion of the public.

The institutional accreditation scheme devised by the government’s policy makers and approved by Congress is a four-stage model. It includes a self-evaluation carried out within the institution, followed by a peer-review report with a recommendation to accredit or not. Institutional peer reviewers are selected by the CNA from a pool of registered peers, in consultation with the interested university. The law contemplates potential disagreements over candidates. During the period covered in this study, institutions could veto candidates up to three times, and if disagreement remained, the institution could appeal to the Consejo Superior de Educación (CSE) or its successor the Consejo Nacional de Educación (CNED). The process was designed to minimize potential biases in the designation of peers. Nowadays, this veto power is limited.3 The process ends with an accreditation judgment decided by a majority vote of the CNA’s commissioners, which is then made available to the public (Cancino and Schmal, 2014; Espinoza and González, 2012, 2013, p. 28; López et al., 2018). Only this final step during the period 2013–2016 is the concern of this study.

CNA started its activities in late 2006 setting guidelines for programme and institutional accreditation. However, early in 2007, CNA’s commissioners reasoned that ‘… accreditation [was] a State function, and thus, [required] esprit de corps to exercise freedom [in judgment] …’. Consequently, they agreed unanimously on confidentiality, declaring that judgments had to appear ‘… as coming from the CNA itself…’ and not as the sum of its commissioners’ votes (Acta No 48, 2007). Protected by anonymity, the CNA’s unanimous institutional accreditation judgments between 2007 and March 2013 were uninformative regarding potential biases.

The modus operandi based on unanimity changed after an accreditation scandal. In 2011, a piece of investigative journalism revealed that a mid-sized university accredited for 1 year in 2010, Universidad del Mar (UDM), which had approximately 2000 students enrolled with guaranteed student loans, recruited faculty for their medical school who did not hold university degrees. UDM’s accreditation — journalists concluded — involved corruption (Riquelme and Guzmán, 2012). The investigation that followed revealed that the president of the CNA — the second to chair CNA and the head of a second-tier private university — had received bribes for accrediting UDM and other institutions (González and Guzmán, 2012a, 2012b).

The UDM scandal compromised public faith in the national accreditation system. As a consequence, in early 2013, a newly appointed president of CNA implemented changes in the CNA’s voting system which had already been agreed upon by commissioners after the scandal. Commissioners consented to record their votes starting in March 2013 (Acta No 615, 2012).

The incoming president retrospectively reflected:

It is a whole different ballgame when one casts an anonymous vote than when one has to stand by it. We had to “raise the bar” to restore public faith in the system.

(…) This probably was more important in Chile than elsewhere because of the link between access to public funding and accreditation. I was convinced that if voting had been public, UDM would have been denied accreditation. (Past President of CNA (name withheld), personal communication, March 8, 2018)

To speed the process up, the new president announced the change to the press and the Congress. He reminisced:

It was a strategy, for once announced there was no turning back. In my view, it was pretty evident that making votes public would act as a natural deterrent to conflict of interests in accreditation. (Past President of CNA, personal communication, March 8, 2018)

Before the procedural change, the percentage of all types of institutions that applied for accreditation and did not obtain it was 8.5 per cent. The rate of denial increased to 14.3 per cent in the period when votes became public, signalling a change towards stringency in accreditation judgments. However, albeit intriguing, the sole comparison between the outcomes of accreditation between the two periods does not provide conclusive evidence as to the nature of the change observed, since the pool of institutions that sought accreditation varied, as did the members of the board.

The present study aims to examine whether commissioners tended to vote more favourably towards universities belonging to the conglomerate that designated them compared to other commissioners, controlling for potentially confounding variables. If commissioners favoured institutions from their conglomerate, it would indicate that a potential conflict of interests (PCI) operated despite the changes in the voting system introduced in 2013 to curb it. To our knowledge, ours is the first study that aims to examine from an empirical perspective the presence of a potential conflict of interest in accreditation judgments.

Methods

We explored the PCI bias through a quantitative research approach. We compiled and analysed institutional accreditation votes by CNA commissioners from the start of the transparency regulation in March 2013 to October 2016, including minutes #632 to #1037. One of the authors extracted the data from the transcripts of the CNA’s regular weekly meetings and coded it. The transcripts contained, among other decisions, commissioners’ votes either denying or granting accreditation. In the latter case, the data also included the years of accreditation granted.

The process of building the data set involved analysing 406 minutes. The records provided 181 commissioners’ votes on institutional accreditations issued by 29 individuals grouped in 45 meetings in which the commission judged institutional accreditation of universities in first instance: 11 in 2013, 10 in 2014, 11 in 2015 and 13 in 2016. Of these, 25 corresponded to the accreditation of CRUCh institutions and 20 to newer private universities. In total, 159 (out of 181 votes) favoured the position to grant institutional accreditation.

The composition of the commission varied through the years allowing us to study the voting behaviour of the commissioners and to explore whether they tended to favour institutions from their conglomerates. The statute at the time forbade commissioners to vote in the accreditation process of their own institution. However, it did not preclude them from participating in the accreditation of other institutions belonging to the same conglomerate or group of institutions. We coded commissioners’ votes as coming from a CRUCh member (which included public and traditional private separately) or new private universities (see Fig. 1).

Figure 1
figure 1

Diagram of the different categories of universities (as coded).

Full size image

As defined, PCI might arise when a commissioner judges and votes favouring a university that belongs to her conglomerate or disfavouring those belonging to other conglomerates or groups. We employed logistic regression and multiple linear regression analysis to explore the associations between the voters’ conglomerate designation and their votes. Specifically, we focused on whether the commissioner’s affiliation could predict: (1) her vote for accreditation of universities within her conglomerate and (2) years of accreditation granted. We focused exclusively on testing PCI on accreditation of universities and omitted from the analysis commissioners appointed by other types of institutions (i.e. IPs and CFTs) since in preliminary analyses we did not find evidence suggesting the presence of bias for either of these groups.

We used the commissioners’ votes as the unit of analysis to test for PCI and the potentially confounding background and contextual variables in accreditation decisions (i.e. to accredit or not and, separately, the number of years granted). The CNA minutes provided additional mandatory self-reported institutional information relative to the internal operations of each institution and commissioners’ data (designating conglomerate and gender).

Institutional information was available on: the number of years of accreditation assigned in the previous accreditation process; the total institutional enrolment, an indicator of institutional size (0 = small; 1 = medium/large, where below 5000 students are considered to be small); the size of the faculty body (FTE); the number of degrees offered by the institution; the number of technical programmes offered, and the dropout rates.

Previous accreditation status — the number of years of accreditation assigned in the previous process — was included to account for past judgments of quality. However, we do not interpret the standing number of years of accreditation as an accurate or valid assessment of standing quality. We included it because the standing years of accreditation granted could influence or bias the commissioners’ decisions in subsequent processes (Barroilhet, 2019). We included gender as a control variable to assess whether the stringency or leniency of accreditation judgments was associated with the gender of the commissioner.

Among other control variables, the 5000-student threshold for institutional size deserves further explanation: we expected smaller and non-selective institutions to have a disadvantage, with commissioners tending to favour larger institutions in a ‘too-big-to-fail’ logic. Denying accreditation to a large institution entails denying state guarantee for student loans, an outcome that bears undesirable individual and administrative costs. The specific threshold of 5000 was decided a priori based on existing criteria. In his taxonomy of Chilean Universities, Brunner (2009) characterizes institutions by their enrolment, among other features. Small non-selective institutions had typically a total enrolment below 5000 students. Similarly, in a comparative context, the website Times Higher Education classifies as ‘small universities’ those with fewer than 5000 students (Times Higher Education, 2019).

We created a proxy categorical variable to measure PCI since it was not amenable to direct measurement. Hence, PCI was dummy coded: a value of 1 indicated that the classification type of the university pursuing accreditation coincided with the affiliation provenance of the commissioner. For instance, if a CRUCh-designated commissioner had to pass judgment on whether to accredit or not a CRUCh university, the value of PCI was one. If she had to face the accreditation process of a non-CRUCh institution, the value of PCI was zero.

We tested whether PCI had a statistically significant influence on the model after controlling for background and contextual variables. A statistically significant coefficient for the PCI variable indicated that the commissioners from the same conglomerate tended to vote more favourably towards the accreditation of institutions from their conglomerate.

We used accreditation decision and the number of years of accreditation as expressed in each vote as criterion variables. We ran logit regressions for the binary accreditation decision (accreditation granted = 1; accreditation denied = 0). We also computed Poisson regressions to model the number of years of accreditation voted by the commissioner. As in any regression, causal interpretation should be treated with caution. Consequently, all usual caveats entailed in the interpretation of regression results apply.

Results

We report descriptive statistics for the institutional background data included in the analyses in Table 2. Standing accreditation varied from zero to six years depending on the institution, with an average of 3.4 years of accreditation (the values can go up to 7 years of accreditation, but only two universities achieved a 7, and those two did not apply to reaccreditation during the period covered by the study). We observed variability in all background variables, signalling the diverse characteristics of institutions going for accreditation in that period.

Table 2 Descriptive statistics for institutional background variables (N = 181).
Full size table

Full-time faculty, new enrolment (NE), and retention had (unrealistic) values of zero in several records. Consequently, we opted to exclude these variables as covariates in the logistic regression.

The model reported in Table 3 corresponds to an exploratory logistic regression model, the criterion variable being whether to grant or not grant accreditation. We followed standard reporting procedure recommendations (Peng and So, 2002).

Table 3 Logistic regression of decision to accredit by selected predictors (N = 181).
Full size table

The results indicate that standing accreditation, PCI and both enrolment measures yield statistically significant coefficients (and significant corresponding marginal probabilities) and contribute in predicting the vote for accreditation. There is an apparent (and expected) consistency between past and current accreditation decisions since it is unlikely that an institution drastically changes its status quo in consecutive accreditations. The PCI showed a significant coefficient, indicating a tendency for commissioners to favour the conglomerate that designated them.

The size indicator dummy variable (equal to 1 for institutions with over 5000 enrolled students), jointly with the regressor for total enrolment (TE), revealed that the effect of enrolment on the determination is both relevant and nonlinear, lending support to the ‘too-big-to-fail’ notion. Thus, institutions above and below that threshold appear to be qualitatively different and merit separate analyses. If the number of observations available had been larger, we could have conducted a regression discontinuity design (RDD) analysis. However, given the small sample size, we opted to compute separate regressions above and below the threshold. Table 4 displays the results by enrolment size.

Table 4 Logistic regression of decision to accredit, for institutions with TE below and above 5000 students.
Full size table

The results from Table 4 reveal that the most relevant predictors for the decision to grant accreditation to smaller institutions are PCI and standing accreditation. The marginal probability of choosing to grant accreditation by a commissioner is increased by 31.78 per cent when she has a potential conflict of interest. For larger institutions, however, the estimate of the marginal effect of PCI is not statistically identified: in all 67 votes where a commissioner had a potential conflict of interest, the vote favoured granting accreditation—commissioners’ votes were affirmative whenever PCI was equal to 1.

Tables 5 and 6 present the results for commissioners’ decisions on the number of years of accreditation granted. Unlike the decision to grant accreditation shown in Table 4, PCI was not a significant predictor. However, the ‘too-big-to-fail’ size indicator in Table 5 is highly statistically significant, confirming the need for analysing small and large institutions separately as reported in Table 6. For both groups, standing accreditation is the key predictor, with total enrolment remaining statistically significant. The number of technical programmes has a significant, albeit negative impact on the years of accreditation of larger institutions. One plausible explanation is that having many technical programmes at a university may be seen by peer reviewers and commissioners as signalling lower overall quality than focusing on university careers.

Table 5 Poisson regression to analyse number of years of accreditation granted by voter
Full size table
Table 6 Poisson regression to analyse number of years of accreditation granted by voter for small (TE < 5000) and large (TE ≥ 5000) institutions
Full size table

Discussion

Our findings support the notion that transiting from anonymous to transparent votes in the CNA may not have achieved its full purpose: the weight of the PCI variable on the decision to accredit appears to be substantial. Thus, we interpret the findings as preliminary evidence of the pervasiveness and ubiquity of agency issues and the conflict of interest bias: it can still be detected even in an open-vote environment. Interestingly, PCI was not significant as a predictor of the number of years awarded to an institution. This intriguing finding may indicate that the two decisions — to accredit or not and the number of years granted to an institution — constitute distinct judgments governed by independent processes.

What if commissioners from competitor conglomerates were too harsh towards institutions from rival conglomerates? What if the differences observed may not be so much a reflection of a public–private divide but as an expression of veiled resistance of the powerful CRUCh conglomerate against institutions founded after 1981? Considering that the total enrolment of new private universities grew 30.4 per cent in the 2008–2013 period, compared to an exiguous 3.94 per cent for CRUCh public institutions, CRUCh commissioners’ more stringent accreditation criteria may be seen as instrumentally geared to slow down the proliferation of new universities (López et al., 2015). Such interpretation would also be consistent with a (somewhat Machiavellian) notion that institutions at the top tend to bond in a positional competition and play in new competitive scenarios to their advantage (Marginson, 2004, 2006). Either explanation appears to be coherent with a potential conflict of interest.

In terms of the ‘too-big-to-fail’ hypothesis, the evidence appears to support it, since institutional size was a consistently significant predictor for both the accreditation decision and the years of accreditation granted. Institutional size is a control variable that should be addressed in future studies, ideally through quantitative and qualitative methods. Among the latter type, retrospective think-aloud studies could be useful in understanding the reasoning processes in accreditation decisions that involve large and small institutions.

Yet, the findings — albeit convincing — need to be qualified due to their correlational nature. Accreditation studies by-and-large ignore conflict of interest as a relevant variable. Many accreditation agencies worldwide continue to operate using anonymous off-the-record voting schemes.

Given the difficulties entailed in the definition of ‘quality’, it may be safer to consider that accreditation addresses some valued socially legitimized quality institutional attributes. If accreditation aims to confer legitimacy to institutions of higher education, one should consider in the institutional design the potential conflicts of interest and the presence of other biases that can compromise the validity of accreditation judgments. The need is stronger in nations where accreditation has taken on a far more significant role in orienting financial aid. One salient example where accreditation and financing are linked is in the USA: “[s]tudents attending institutions that are not accredited are ineligible for federal financial aid, money that is indispensable to the budgets of most American colleges” (Conn, 2014).

As in any correlational study, the omission of relevant covariates can impact directly on the interpretations of the results. Thus, as in any exploratory study, the results should be considered tentative at this point. Some potentially relevant covariates could not be included due to incomplete or erroneous reporting of institutional data in the official CNA transcripts. Still, even if more variables had been available, there is a natural limit regarding how many predictors could be included in a model with small sample size. Future studies should aim to monitor subsequent accreditation decisions expanding the number of observations to allow for the inclusion of more covariates, both at the institutional and the individual or commissioner level. In terms of individual factors, although gender was not a significant predictor, other commissioner demographic characteristics should be considered, when possible, in the analysis. For example, age, disciplinary training and personality characteristics such as a tendency to leniency or strictness in general (Ganzach, 1993), and other perceptual biases, may influence accreditation judgments and might merit further exploration.

Accreditation, or any form of judgment, will never be completely bias-free, since bias is inherent to the human condition. The first step is to acknowledge the existence of bias, the second is to examine mechanisms that hold a promise to reduce it, and the third step is to put forward those mechanisms that have proved useful to control biases. The task is not easy, but merits attention. The literature on courts decision-making and judges’ biases, or sports referees, is enlightening about how pervasive and strong biases can be, even when it comes to applying well-defined rules.

Concluding Remarks

At present, Chile is undergoing an accreditation reform that was expected to be implemented in full in 2020. Among other aspects, the new reform modified the composition of the CNA, made institutional accreditation mandatory, changed accreditation's possible outcomes, suppressed individual programme accreditation and changed procedures and the criteria for making accreditation decisions. The accreditation reform, not fully operational due to the COVID-19 pandemic and delays in implementing guidelines, has extended standing accreditations. The reform aims to make accreditation judgments more valid and reliable in terms of assessing quality and avoiding potential conflicts of interest of commissioners. Although conflict of interest is a problem for any accreditation system, in the case of Chile, the high stakes attached to the outcomes of the accreditation process (i.e. financial consequences for institutions) make it more necessary to attend to and attenuate the effect of conflict of interest in accreditation decisions.

There is no substantive evidence that the Chilean higher education system has improved significantly after the introduction of accreditation. Moreover, since enrolment rates are stalling, some congress members are pushing for more stringent standards at the bottom. Nevertheless, this was not the argument used to support the need for an accreditation reform. Instead, the change was forwarded to avoid representational conflict associated with the CNA’s structure, where stakeholders had substantial weight on direct decisions affecting them financially. According to the executive’s message introducing the bill to reform CNA to congress:

[T]he current structure of the CNA favours conflicts of interest given that commissioners are appointed as representatives of the sectors of higher education they have to overview and accredit. (Biblioteca del Congreso Nacional, 2018, translated by the authors).

One significant change is that the new CNA has 6 out of its 12 members appointed by the President of the Republic, with the agreement of 3/5 of the Senate, two appointed solely by the President from a list proposed by the Council for High Public Administration (Consejo de Alta Dirección Pública - ADP), one elected by the National Corporation for Development (CORFO) and one elected by scientific societies, both from lists proposed by the ADP, and two selected by student organizations (Law No 21.091, Art.7). Thus, conglomerates no longer will designate commissioners. The question is, since chosen academics will still be affiliated to some of the institutions, and potential alliances are likely to persist, will this change contribute to eliminate or reduce conflict of interests’ biases in accreditation judgments?

The results of our empirical analyses suggest the existence of some degree of conflict of interest bias was still present after voting became transparent. Whether it can be further reduced is still a pending issue. Given the high stakes associated with the outcomes of accreditation for institutions, students and society, more research, both quantitative and qualitative on accreditation biases, is amply justified. Our findings have implications for Chile but may likely be of use to other nations, particularly those that have attached high stakes to their accreditation system or are considering moving towards that direction. Exploring the influence of PCI and other extraneous variables that can be a source of bias in the assessment of accreditation outcomes can improve the validity of accreditation judgments and their value to orient and inform public policy decisions. It remains to be seen if after 2022, all these changes will have the desired effect of improving Chile’s quality assurance system.

Notes

  1. 1

    Readers interested in learning in greater detail the contextual background in which accreditation operates in Chile can consult Barroilhet (2019), Bernasconi et al. (2020), Geoffroy Pitta (2013) and Scharager Goldenberg (2018).

  2. 2

    Law No 20129 of 2018 changed the composition of the CNA and changed the period from 4 to 6 years.

  3. 3

    After Law No 21091 of 2018 the procedure allows institutions to challenge CNA’s designated peers only once and the institution cannot longer appeal to the CNED.