Methods for Examining the Effects of School Poverty on Student Test Score Achievement

Abstract

Measuring school effects has been an important inquiry for sociologists of education for at least 50 years. This chapter summarizes current research on the relationship between school poverty and student achievement, which relies heavily on cross-sectional associations. We then propose that scholars consider longitudinal approaches to estimating school effects in which changes in school outcomes are related to changes in school contexts. We present illustrative examples of both cross-sectional and longitudinal analyses using a census of North Carolina students and schools. Cross-sectional models indicate a significant negative association between school poverty and achievement. Our preferred specification—a three-level model of time within students cross-nested within schools—finds no relationship between school poverty and achievement, which raises important questions about the validity of school poverty effects on student test score growth. This model does, however, suggest that variation in test score growth across schools may be greater than variation in test score growth across students, which opens important avenues for understanding the sources of this variation.

Appendix: The Feasible Generalized Least Squares Method for Random Intercept Models

The first method to estimate within and between effects in a single model was feasible generalized least squares (FGLS), which is a process of transforming the variables to control the error structure and then fitting OLS models to the new data. The procedure outlined here is called the Swamy-Arora (1972)¹¹ method and is implemented in many software packages. This model requires estimates of conditional variance components.¹² To define conditional variance components, consider that the total conditional variance of the outcome (conditional on the values of fixed predictors and their fixed effects) is a combination of the variances of the error terms from the between and within models, var(y _ij| X _ij) = var (u _j) + var (e _ij). These two quantities are called variance components. Variance components can be rescaled to be an estimate of the intraclass correlation (ICC), defined as $\rho =\frac{\mathit{\operatorname{var}}\left({u}_j\right)}{\mathit{\operatorname{var}}\left({u}_j\right)+\mathit{\operatorname{var}}\left({e}_{ij}\right)}$, which is the proportion of the total conditional or unconditional variation that exists between groups is characterized by the intraclass correlation, ICC or ρ.¹³ The intraclass correlation is a measure of how much units within the same group resemble each other on their values of an outcome. The larger the ICC, the more correlated (i.e., more similar) two units are within the same group.

The ICC is an important parameter because it is a key contributor to the design effect (Kish 1965) of the variances of the between-group effects, contextual effects, and within effects. Design effects are measures of how much the sampling variance (the square of the standard error) of estimated effects of group level variables (such as between effects) change due to the estimation strategy (i.e., generalized least squares compared to ordinary least squares). The use of OLS naively on the original data produces sampling variances that ignore the design effects, leading to inflated standard errors and false tests of the null hypotheses.

Estimating a contextual effects model with feasible generalized least squares (FGLS) requires transforming the OLS equation with weights defined by a ratio of the variance components. For clustered data, the transformation de-means the value using a weight: For example, a variable z would be transformed by ${z}_{ij}^{\ast }={z}_{ij}-\widehat{\theta}\ {\overline{z}}_j$using the group mean of z and the parameter θ as the weight. The parameter θ is based on the within and between variance components (Cameron and Trivedi 2005):

$$ \widehat{\theta}=1-\sqrt{\frac{\mathit{\operatorname{var}}\left({e}_{ij}\right)}{n\left(\mathit{\operatorname{var}}\left({u}_j\right)\right)+\mathit{\operatorname{var}}\left({e}_{ij}\right)}.} $$

Thus, a model that estimates both within and between effects can be computed using the following regression,

$$ {\displaystyle \begin{array}{c}\left({y}_{ij}-\widehat{\theta}\ {\overline{y}}_j\right)={\beta}_0\left(1-\widehat{\theta}\right)+{\beta}_1\left(\ {\overline{x}}_j-\widehat{\theta}\ {\overline{x}}_j\right)\\ {}+{\beta}_2\left({x}_{ij}-{\overline{x}}_j\ \right)+\left({e}_{ij}-\overline{e_j}\right).\end{array}} $$

As between-unit variation increases relative to within-unit variation, then $\widehat{\theta}$ approaches 1 and the random effects estimator converges to the fixed effects estimator, which demeans with the entire portion of the group mean. Conversely, as within-unit variation increases relative to between-unit variation, $\widehat{\theta}$ approaches 0 and the random effects estimator converges to pooled OLS, in which group averages are irrelevant. In other words, the random effects estimator uses the information in the data to determine how much of the group mean to include in the estimate, more if between effects are large and less if between effects are small.

The θ parameter is also directly related to how the standard error increases when using the correct model compared to the naïve OLS estimator. For example, if we examine the “Random Within and Between Effects Model” in column 1 of Table 22.2 we see that the standard error of the between effect (0.513) is much larger than the standard error of the between effect from the OLS model in column 3 of Table 22.1 (0.136). The random effects variance is about 0.513^2/0.136^2 = 14 times higher than the OLS variance. There are about 55 students per school, and the conditional ICC is 0.23, so the expected design effect is about 1 + (55–1)*0.23 = 13, which is consistent with the observed inflation in the standard error of the between effect coefficient.