Regression Discontinuity Design: Recent Developments and a Guide to Practice for Researchers in Higher Education

Abstract

In an educational research climate, where the understanding of causal relationships between predictors and outcome variables is of primary interest, however, randomized experiments are either unethical or unfeasible, researchers have been turning to research designs and statistical techniques that produce causal results in a quasiexperimental manner. One such quasiexperimental design is regression discontinuity (RD), which has been shown to require fewer assumptions than most other designs and the assumptions it does require are verified with nearly the same ease as a randomized experiment. We provide a user friendly guide to RD designs for the educational researcher. We begin by discussing terminology associated with both experimental and RD designs. Next, we describe the process of checking the assumptions associated with an RD design and use an empirical example to illustrate this process. We then expand on the empirical example to show how to obtain estimates of treatment effects in both “sharp” and “fuzzy” RD designs employing both parametric and nonparametric frameworks. Finally, we discuss new developments in RD research and some future applications in educational research. A technical appendix is also included with additional details on the selection of parameters in the estimation procedure and we include a URL link to statistical code to conduct RD analysis.

Appendix: Local Polynomial Regression Estimates and Optimal Bandwidth Determination

In this appendix, we will describe in more detail how the nonparametric estimates are computed. Recall that the treatment effect $\tau$ for a RD design equals

$$ \tau =\frac{{{\lim }_{d\to 0}}E(Y|c+d>X>c)-{{\lim }_{d\to 0}}E(Y|c>X>c-d)}{{{\lim }_{d\to 0}}\Pr (T=1|c+d>X>c)-{{\lim }_{d\to 0}}\Pr (T=0|c>X>c-d)} $$

(5)

Now with the GMS program, no individuals below the cut-off score received a scholarship so ${{\lim }_{d\to 0}}\Pr (T=0|c>X>c-d)=0$ and

$$ \tau =\frac{{{\lim }_{d\to 0}}E(Y|c+d>X>c)-{{\lim }_{d\to 0}}E(Y|c>X>c-d)}{{{\lim }_{d\to 0}}\Pr (T=1|c+d>X>c)}. $$

(6)

To derive a consistent estimator of $\tau$ where consistency means that as the sample size gets large our estimate $\hat{\tau }$ approaches the true population value $\tau$ with probability one, we will need to consistently estimate the three terms in Eq. 6, $E(Y|c+d>X>c)$, $E(Y|c>X>c-d)$, and $\Pr (T=1|c+d>X>c)$, for some “small” value of d (i.e., using data close to the cut-off score). In order to do this, we will apply a technique called local polynomial regression (Fan and Gijbels 1992).

Consider the nonparametric regression model:

$$ Y=m(X)+\varepsilon . $$

(7)

In Eq. 7, m(X) is any continuous function of X instead of, e.g., a linear function of X: b ₀ + b ₁ X. So, instead of estimating two unknown population parameters b ₀ and b ₁, we are trying to estimate some unknown population function m(X). Local polynomial regression estimates the function m(x) by obtaining estimates of its value for a large number of values of x. For a particular value of x and x ₀, the value m(x ₀) is estimated by running a weighted polynomial regression using the sample where points in the sample that are closer to x ₀ receive larger weights in the regression. More formally, suppose that we wish to estimate the value m(x ₀) using a local polynomial regression of order p. Define the data matrix X by:

$$ X=\left[\begin{matrix} 1 & {{X}_{1}}-{{x}_{0}} & \cdots& {{({{X}_{1}}-{{x}_{0}})}^{p}}\\ \vdots& \vdots& \vdots& \vdots \\ 1 & {{X}_{n}}-{{x}_{0}} & \cdots& {{({{X}_{n}}-{{x}_{0}})}^{p}}\end{matrix} \right] $$

(8)

where each value of X in the sample is measured relative to its distance from x ₀. Let y be the vector of values of the dependent variable:

$$ y=\left( \begin{matrix} {{Y}_{1}}\\ {{Y}_{2}}\\ \vdots \\ {{Y}_{n}}\end{matrix} \right). $$

(9)

In local polynomial regression, we will want to run a weighted regression where those points closer to x ₀ receive larger weights in the regression. To this end, define the diagonal weighting matrix W by:

$$ \mathbf{W}=\text{diag}\left\{ {{K}_{h}}({{X}_{i}}-{{x}_{0}}) \right\}{.} $$

(10)

In Eq. 10, K _h is a kernel density weighting function with bandwidth h and is defined by ${{K}_{h}}(\cdot )={[K({\cdot }/{h}\;)]}/{h}\;$ for some kernel density function K and some bandwidth h. These kernel density functions reach the maximum height at 0 and decline monotonically as you move away from 0. While there are several possible choices K, we use the Gaussian kernel function which is defined by $K(u)={1}/{\sqrt{2\pi }}\;\exp ({-}({1}/{2}\;){{u}^{2}} )$. Once a bandwidth is determined then we estimate the local polynomial coefficients at x ₀,

$$ \hat{\beta }=\left( \begin{matrix} {{{\hat{\beta }}}_{0}}\\ {{{\hat{\beta }}}_{1}}\\ \vdots \\ {{{\hat{\beta }}}_{p}}\\\end{matrix} \right) $$

by minimizing the weighted sum of squared errors $\textbf{y}-\mathbf{{X}'}\beta$:

$$ \underset{{\hat{\beta }}}{\mathop{\min }}\,(\mathbf{y}-X\beta )'W(\mathbf{y}-X'\beta ). $$

(11)

For theoretical reasons (Fan and Gijbels 1996), it is preferable to estimate odd-ordered polynomial models. In our estimates of the treatment effect $\tau$ using the GMS data, we will simply estimate a local linear regression (p = 1). However, we need to estimate separate local linear regressions for $E(Y|c+d>X>c)$, $E(Y|c>X>c-d)$, and $\Pr (T=1|c+d>X>c)$. Here, we only use the data from the right of the cut-off value when estimating $E(Y|c+d>X>c)$ and $\Pr (T=1|c+d>X>c)$ and only use data to the left of the cut-off value when estimating $E(Y|c>X>c-d)$. Letting x be the closet point on our grid of x values to the left of c (which in our GMS application is c - 0.1) and x ₊ be the point closest on our grid of x values to the right of c (c + 0.1), the estimated value of $\tau$ equals

$$ \hat{\tau }=\frac{\hat{E}(Y|X={{x}_{+}})-\hat{E}(Y|X={{x}_{-}})}{\hat{\text{P}}\text{r}(T=1|X={{x}_{+}})}. $$

(12)

To implement this technique it is necessary to choose a bandwidth. Recall that the bandwidth determines the amount of smoothing. The larger the bandwidth, the bigger the potential for bias while a smaller bandwidth leads to an estimate with higher variance. We would like to choose a bandwidth which balances the potential of bias with variance. So, we choose the bandwidth that minimizes the mean square error which equals the sum bias squared plus variance.

Let ${{s}_{r,0}}=\int_{0}^{\infty }{K(u){{u}^{r}}du.}$

It can be shown that for a local linear regression model the optimal bandwidth, to the left of the cut-off point equals, (see Fan and Gijbels 1992, 1996) equals

$$ {{h}_{opt}}({{x}_{0}})=C(K){{\left[ \frac{{{\sigma }^{2}}({{x}_{0}})}{{{\left\{ {m}''({{x}_{0}}) \right\}}^{2}}f({{x}_{0}})n} \right]}^{{}^{1}\!\!\diagup\!\!{}_{5}\;}} $$

(13)

where

$$ C(K)={{\left[ \frac{\int_{0}^{\infty }{\left[ {{s}^{2}}_{2,0}-t{{s}_{1,0}} \right]{{K}^{2}}(t)dt}}{{{\left\{ {{s}^{2}}_{2,0}-{{s}_{1,0}}{{s}_{3,0}} \right\}}^{2}}} \right]}^{{}^{1}\!\!\diagup\!\!{}_{5}\;}}, $$

${{\sigma }^{2}}({{x}_{0}})$ is the variance of $\varepsilon$ at x ₀, ${m}''({{x}_{0}})$ is the second derivative of m at x ₀, and $f({{x}_{0}})$ is the density of x at x ₀.

The idea here is that the larger the variance of the error at x ₀, all else equal, the larger the optimal bandwidth is in order to minimize the mean square error since a larger bandwidth will smooth the data more and reduce the variance of the estimate. On the other hand, the larger ${m}''({{x}_{0}})$ the smaller the bandwidth, all else equal, because the slope of the function $m({{x}_{0}})$ changes more quickly and so a smaller bandwidth reduces the amount of bias.

For the Gaussian kernel density function that we employ in our estimates C(K) = 0.794. Several of these quantities are unknown; e.g., ${m}''({{x}_{0}})$ depends on m, which is what we are trying to estimate in the first place. So, we must employ a two-step method to obtain the optimal bandwidth.

In the first step, we compute what is termed as the “Rule of Thumb” (ROT) bandwidth which we denote by h _ROT. To compute h _ROT we first obtain a rough estimate of m(x) using a fourth-order (quartic) polynomial and weighting all the data equally. From these estimates we compute

$$ \tilde{m}(x)={{\tilde{\beta }}_{0}}+{{\tilde{\beta }}_{1}}x+\cdots +{{\tilde{\beta }}_{4}}{{x}^{4}} $$

which results in an estimate of ${m}''({{x}_{0}})$:

$$ {\tilde{m}}''(x)=2{{\tilde{\beta }}_{2}}+6{{\tilde{\beta }}_{3}}x+12{{\tilde{\beta }}_{4}}{{x}^{2}}\quad$$

and

$${{\sigma }^{2}}: {{\tilde{\sigma }}^{2}}=\sum\limits_{i=1}^{N}{\frac{{{( {{y}_{i}}-\tilde{m}(x) )}^{2}}}{N-5}}\quad$$

Then,

$$ {{h}_{\text{ROT}}}=0.794\times {{\left[ \frac{{{{\tilde{\sigma }}}^{2}}}{\sum\nolimits_{i=1}^{n}{{{\left\{ {\tilde{m}}''({{x}_{i}})-{{K}_{{{h}_{\text{ROT}}}}}({{x}_{i}}-{{x}_{0}}) \right\}}^{2}}}} \right]}^{{}^{1}\!\!\diagup\!\!{}_{5}\;}} $$

In the second step, we estimate a third-order local polynomial regression using the rule of thumb bandwidth h _ROT and the Gaussian kernel density function and obtain a refined estimate of m(x) at x ₀

$$ \hat{m}({{x}_{0}})={{\hat{\beta }}_{0}}+{{\hat{\beta }}_{1}}{{x}_{0}}+{{\hat{\beta }}_{2}}{{x}^{2}}_{0}+{{\hat{\beta }}_{3}}{{x}^{3}}_{0}{.} $$

From this we get:

$$ {\hat{m}}{''}({{x}_{0}})=2{{\hat{\beta}}_{2}}+6{{\hat{\beta}}_{3}}{{x}_{0}} $$

and

$$ {{\hat{\sigma }}^{2}}({{x}_{0}})=\frac{\sum\limits_{i=1}^{N}{{{( {{y}_{i}}-\hat{m}({{x}_{i}}) )}^{2}}{{K}_{{{h}_{\text{ROT}}}}}({{x}_{i}}-{{x}_{0}})}}{\text{tr}\left\{ \mathbf{W}-\mathbf{WX}{{\left\{ \mathbf{{X}'WX} \right\}}^{-1}}\mathbf{{X}'W} \right\}}. $$

From Eq. 13 the optimal bandwidth is then

$$ {{\hat{h}}_{\text{opt}}}({{x}_{0}})=0.794{{\left[ \frac{{{{\hat{\sigma }}}^{2}}({{x}_{0}})}{{{\left\{\hat{{m}}''({{x}_{0}}) \right\}}^{2}}\hat{f}({{x}_{0}})n} \right]}^{{}^{1}\!\!\diagup\!\!{}_{5}\;}} $$

where $\hat{f}({{x}_{0}})$ is estimated using a Gaussian kernel density estimator:

$$ \hat{f}({{x}_{0}})=\frac{1}{nh}\sum\limits_{i=1}^{n}{K\left( \frac{{{x}_{0}}-{{x}_{i}}}{h} \right)} $$

where h is chosen to minimize the mean squared error.

When computing the optimal bandwidth for $\hat{E}(y|{{x}_{+}})$ and $\hat{\text{P}}\text{r}(T=1|{{x}_{+}})$ we only use the data to the right of the cut-off point c when x ₀= x ₊. When computing the optimal bandwidth for $\hat{E}(y|{{x}_{-}})$ we use the data to the left of the cut-off point and x ₀= x _-. The Stata programs necessary to compute these estimates can be found at http://www-personal.umich.edu/~bpmccall/Programs/Handbook_Chapter/.