How do admission committees select? Do applicants know how they select? Selection criteria and transparency at a Dutch University

Evaluative quality criteria for selection procedures

Disregarding these fundamental quality criteria in a selective admission process might have a wide range of negative consequences. Namely, using invalid and unreliable selective admission criteria might lead to a higher number of false-positive cases – students, who are admitted to a program, but are unable to complete it (Admissions to Higher Education Steering Group 2004), and a higher number of false-negative cases - rejected applicants, who would have been able to successfully complete the program, but did not get a chance to (Ooijen-van der Linden et al. 2017). Insufficient level of transparency or misleading information about selective admission criteria, documents and processes creates a disadvantage for insufficiently informed applicants such as failing to acquire knowledge and skills that are of importance for a graduate program, failing to realistically assess the likelihood to be accepted into a graduate program (Nauta 2000), and discouraging applicants with low self-confidence from applying (Admissions to Higher Education Steering Group 2004).

A call for more transparency by decentralized selection has been made by policy makers (Adnett et al. 2011; Admissions to Higher Education Steering Group 2004), higher education organizations (e.g. Utrecht University Education Guideline 2017), and by the students themselves (Interstedelijk Studenten Overleg 2014). Following this call, some universities began publishing minimum, median and top study scores (Bagshaw 2016), standardized test scores and background information of their admitted students such as race, ethnicity and geographical origins (La Noue 2003). These actions certainly contribute to articulating clearly to future applicants the kind of students that have a chance to be admitted. However, even if a university implements such policies, what still stays ambiguous to applicants is how the decision-makers actually select applicants who they will then accept (La Noue 2003).

How do admission committees select?

In the context of graduate education, where the admission process is often decentralized, the selective admission decisions are made by the academic staff members who participate in a program’s committee (further referred as admissions committee members). Research across different fields shows that in admissions applications, the admissions committee members are looking for indicators of both academic background and performance on one hand and of personality, motivation and attitudes, on the other.

For academic background and performance, admissions committee members consider important a match or fit between interests and skills of an applicant and what the program offers (Karazsia and McMurtry 2012; Karazsia et al. 2013; Posselt 2016; Walpole et al. 2002); the quality of applicants’ undergraduate experience (Walpole et al. 2002), such as prior institution - where and how an applicant received the undergraduate credits (Kogan et al. 2015), prior degree in a specific field, transcripts, applicant’s work experience, and grade point average (GPA) (MasterMind Europe 2017). In criteria related to personality, motivation and attitudes, admission committee members put attention on applicant’s integrity and honesty (Walpole et al. 2002), communication skills, critical thinking, ethical behavior (Conlon et al. 2012) and personal skills (MasterMind 2017).

The measurement of some of these criteria have been criticized for being prone to biases in selective admissions practice (Kira Talent 2018; Posselt 2016). Namely, when it comes to assessment of personality, motivation, and attitudes, such tools as the interviews, recommendation letters, CVs and motivation letters are often used. Interviews, CVs and motivation letters represent self-reported measures and when applied in a high-stake situation of selective admission, they are susceptible to faking by the applicants (Niessen et al. 2017). When interpreting recommendation letters, a measure which is not self-reported, the admission committee members sometimes indicate a need “to read between the lines” or “to detect an extra glow of enthusiasm about the student” (Walpole et al. 2002 p. 5), which indicates the susceptibility of this measure to biases as well.

Context of the study

Utrecht University is a higher education institution which offers research-oriented education in contrast to higher professional education that is offered by universities of applied sciences. A specific focus on research-oriented education means that the main goal of study programs is “independent academic participation or the professional use of academic knowledge” (Nuffic 2015, p. 11). For example, the mission of one of graduate schools participating in this study is “to train future scientists who will be capable of drawing on their acquired knowledge, understanding and skills to conduct top-class research around the world” in academia or industry (Graduate School of Life Sciences 2019). During a two-year master’s programs, a student focuses in a specific field and learns about research methods. A final thesis is an obligatory component of a research-oriented study program.

Two Graduate Schools of Utrecht University were involved in this research: Graduate School of Life Sciences (GSLS) and Graduate School of Natural Sciences (GSNS). The GSLS is organized by three faculties: medicine, veterinary sciences and science (biology, pharmaceutical sciences and chemistry); the GSNS is organized by the faculty of Science (mathematics, physics and information and computing sciences).

Not only has the number students attending Dutch research universities increased (Fig. 1), but also the numbers of applicants. For example, at the GSLS, there were over 2000 applicants in 2018, yet there was only room for 450 students. This brings urgency to evaluate the applied selective admission criteria and methods in terms of their compliance with evaluative quality criteria as described in the literature (Patterson and Ferguson 2010) and requested by the University itself (Utrecht University Education Guideline 2017).

Research questions

The curriculum of the life sciences research master’s programs is internship-focused, while the curriculum of the natural sciences research master’s programs is course-focused. We would expect that the relative importance of admission criteria is different for different types of curricula. Our first research question, therefore, is: What kind of selective admission criteria are reported to be important in decision making at the Life and Natural Sciences graduate schools by staff responsible? Not only the criteria themselves, but also their measurements should meet the evaluation quality criteria. We therefore examine how certain criteria are measured. This brings us to the second research question: What admission tools are most frequently applied? In addition, we aim to empirically analyse transparency of admissions and therefore, our third research question is: Are the applied criteria transparent for the applicants? These questions are tackled in a cross-sectional descriptive study that uses a survey methodology. We also discuss validity, reliability, and fairness of applied criteria based on prior literature in the discussion session.

Procedure and measures

The study was approved by the Dutch Association for Medical Education Ethical Review Board (#969). An invitation to participate in the survey, containing the URL for the online survey, an information letter and a consent form, were sent to the participants by email.

Participants and response

Data analysis

Analysis was conducted in SPSS 25. There were no missing values for the admissions committee members from both graduate schools; 0.2% of values in responses of the GSLS applicants and 0.4% of values in responses of the GSNS applicants were missing completely at random. To handle these missing values, the expectation-maximization (EM) algorithm in SPSS was used. In analysis, first, the descriptive statistics were obtained in order to rank the importance of admission criteria, and the frequencies of admission tools’ usage were obtained. To assess transparency, the importance of admission criteria, as reported by admissions committee members, was contrasted to the importance as perceived by applicants. This was done using Spearman correlation and 2-factorial analysis of variance (ANOVA) (1st factor = admission committees’ members versus applicants; 2nd factor = a domain of admission criteria).

Research question 1: What kind of selective admission criteria are reported to be important in decision-making at the Graduate School of Life Sciences and at the Graduate School of Natural Sciences?

Research question 2: What admission tools are most frequently applied?

At the GSLS, such admission tools as a motivation letter and a CV, an interview, a letter of recommendation and personal acquaintance, if applicable, are all used extensively for assessment of admission criteria. For example, a motivation letter is used for assessment not only of criteria related to motivation itself, but also of criteria related to personality, cognitive ability and previous academic background. Next, there are tools, namely, grade list and course content, that are used in a more focused way. For example, course content is used predominantly for assessment of academic background, but not for assessment of personality and motivation. Finally, a writing sample and an English language test are not frequently used: only a few members of admission committee (25%) indicate that they apply these tools.

At the GSNS, similarly to the GSLS, such admission tools as a motivation letter, a CV and a letter of recommendation are used frequently and for measurement of diverse admission criteria, while a grade list and a course content are also used extensively, but more focused (namely, for measurement of academic background). In contrast to the GSLS, interviews and inferences based on personal acquaintance are not of frequent use at all. Like at the GSLS, at the GSNS, an English language test and a writing sample are not often used for selective admission decisions.

Research question 3: Are the applied criteria transparent for the applicants?

The importance of selective admission criteria at the actual admission decisions and transparency of selective admissions for the applicants were examined on two levels: on the level of specific selective admission criteria and on the level of general admission domains.

For the GSLS, the correlation between the importance of specific admission criteria in decision making as reported by admission committees’ members and the perceived importance of the same criteria by the applicants, is of a moderate-to-large size, r_Pearson (51) = .53, p < .001. Two-way analysis of variance on admission domains revealed that there is significant interaction effect between the respondent group (admission committee members versus applicants) and the type of domain (academic background, grades, cognitive ability, research background, personality and personal competencies, and motivation factors) on their importance, F(5, 86) = 2.51, p = .036, η² = .13. As can be seen from Fig. 2, the extent to which the GSLS applicants perceive criteria related to academic background, grades, research background and motivation to be important is in coherence with reports from admission committee members. The GSLS applicants overestimate the importance of criteria related to cognitive ability and personality.

Open image in new window

For the GSNS, similar results were obtained. Reports of admission committees’ members on importance of specific admission criteria are correlated with applicants’ perceived importance to a large extent, r_Pearson (51) = .63, p < .001. On the level of admission domains, two-way ANOVA revealed a significant interaction effect: the applicants perceive the importance of admission criteria domains differently from what the admission committees’ members report, F(5, 86) = 2.98, p = .016, η² = .15. Figure 3 shows that four domains – academic background, grades, research background and motivation - were perceived by the GSNS applicants to have similar importance in admission decisions as was reported by the admission committee members. Like their GSLS peers, the GSNS applicants overestimate the importance of criteria related to cognitive ability and personality.

Open image in new window

Strength, limitations, further directions and conclusions

Among the strengths of this study are the availability of data from (a) both admission committee members and applicants and from (b) two graduate schools with different scope and different curricula designs. The information from admission committees’ members allowed us to describe the actual admission practices, and information from the applicants allowed us to assess whether these practices are clear to those for whom they are designed. Next, we used a comprehensive list of specific admission criteria, which allowed us to conduct analysis both on a level of specific criteria and on a level of domains – when several similar criteria were considered together. With this approach, we were able to capture the specific trends in applicants’ estimation of admission criteria importance. The fact that for both graduate schools we found either correct estimation or overestimation in the same admission criteria domains speaks for a possible generalizability of our results to research master’s programs. Furthermore, to the best of our knowledge, this is only the second study in the European context that explored admission practices both from admission committee members’ and applicants’ views (see MasterMind Europe 2017 for the survey across Europe), but the first one that directly contrasted perceived importance against actual practices of selective admissions.

Despite these strengths, the consideration of some limitations should be made when interpreting the findings. Firstly, a self-report by admission committee members leaves a possibility that the reported importance of certain admission criteria does not completely correspond with their real importance in the daily practice of selective admission decision making. Nonetheless, we assume that the staff members’ reports closely resemble their practice. Secondly, in the third part of the study, dedicated to transparency of selective admission criteria, most of the respondents were accepted applicants. Therefore, the results might have been affected by a selection bias: the accepted applicants might be those who had a chance to learn about the “unspoken” admission rules in contrast to students who did not have a chance to learn such implicit information from their environment. Thirdly, the study is conducted within one higher education institution that could narrow down the generalizability. Yet the study involves two large graduate schools, one of which is interfaculty, and multiple master’s programs within each of them, so we expect the results to be relevant for (life and natural) science programs more generally.

In summary, this study contributes to our knowledge on what admission committee members are looking for in future graduate students who apply for a research master’s program. As a practical implication, the study suggests that the quality of admissions should be a subject of evaluation and constant improvement in terms of scientific evidence, fairness and transparency. The results are relevant for applicants, researchers in higher education, in particular with a focus on admissions quality, and for professionals, who are involved in the (daily) practice of selective admissions in higher education institutions.