Abstract
For five elementary schools in Harford County, Maryland, this chapter probes the effects of comprehensive school reforms on change in student test scores from school year (SY) 1996–1997 through SY 1999–2000. External consultants employed by Co-nect, an educational design and professional development firm then based in Cambridge, Massachusetts, were the change agents. The consultants began their core services in December of 1997 and ended them three years later. Controlling for the intervening years, this evaluation uses the average test scores of the schools for May of 1997 as the prereform measures and the average test scores for May of 2000 as the postreform measures, thereby covering the major portion of the consultants’ basic contractual activities. These results pertain to this pre to post period and should not be generalized to time periods after the consultants ceased their activities.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
When selecting the matched schools Russell and Robinson (April 2000, 3) applied these criteria (quoted directly from their report):
a. Grades served had to be identical
b. If enrollment was less than 400, then it should be within 50 students, if enrollment was between 400 and 1000, it should be within 100 students, and if enrollment was above 1000, it should be within 200 students.
c. Differences in ethnic composition and free/reduced lunch should be within 10 percentage points.
d. Differences in LEP and mobility rate should be within 5 points.
The matched schools satisfied these criteria exactly or closely.
- 2.
A second chart was even more detailed; it separately reported the percentages passing each of the six tests during 1997 and 1999, for the target and its two matched schools, producing a total of 180 data elements (2 time periods times 3 target-matched schools times 6 tests times 5 target-matched comparisons).
- 3.
These measures of the schools’ performance are referred to as analytical properties of the schools because they are derived from mathematical operations on the data describing their individual students (Lazarsfeld and Menzel [1961] 1972, 227). The schools are characterized by an average based on the performance on tests of its own students.
- 4.
By having two confounded measures of school performance and reasonably strong effects of the reform program, the probability of at least one response showing a positive significant effect is higher than if only one measure is used. Let the null hypothesis be H 0: The school reforms enhance school performance. Let the alternative hypothesis be H a: The school reforms do not enhance school performance. Thus, H 0 is more likely to be accepted (and H a rejected) even if H 0 is false and H a is true. This is a Type II error. By combining the two highly correlated treatment effects and testing the composite for significance, this borrowing of strength will minimize the risk of the Type II error of accepting a false hypothesis.
- 5.
At the time of this study additional information about these measures was available at the School Improvement in Maryland Web site http://www.mdk12.org-and at the Maryland State Department of Education Web site http://www.msde.state.md.us.
- 6.
In order to accommodate the reversals due to the use of the Class statement, post is referred to in the actual SAS code as ryear20 and its zero (0) value indicates the post time period and its one (1) value indicates the baseline time period.
- 7.
The DDFM = KR specifies the Denominator Degrees of Freedom Method as Kenward–Roger (KR). This option corrects for unbalanced data, multiple random effects, and correlated errors. SAS (Littell et al. 2006, 188) recommends its use in repeated measures models in part because it reduces Type I errors, the probability of rejecting true hypotheses.
- 8.
Littell et al. (2006, 200) clarify the difference between R and V options as follows:
The V option on the RANDOM statement request that the marginal covariance matrix be displayed. The difference between the result of the R option in the REPEATED and the result of the V option in the RANDOM statement is simply that the latter displays Var[Y]=ZGZ’+R, whereas the former displays Var[Y|u] = R.
- 9.
The following SAS code implements the Covtest statements in Proc Glimmix for the normally distributed response variable, the SPI index. For explication of these tests for logistic regression, see the next chapter or SAS online documentation.
Title ‘Covariance Tests in Proc Glimmix for Harford Data';
Proc GLIMMIX data = standard ic = Q;
class school Trt Post;
model msapperform = Trt Post Trt*Post year98 year99 lowstgrd locate targett1 safe stratio femprob Blackp forredlp
/solution dist = normal link = identity;
random _residual_/type = unr sub = school(trt) s v vcorr;
weight impscore;
covtest ‘Ho: No G-Side Random Effects (UNR Parameters)’ ZeroG/cl;
covtest ‘Ho: Independence = No G-side, Diagonal R-side ' Indep/cl;
covtest ‘Ho: Homogeneous off-diagonal correlations, CSH ' General
0 0 0 0 1 - 1,
0 0 0 0 1 0 -1,
0 0 0 0 1 0 0 -1,
0 0 0 0 1 0 0 0 -1,
0 0 0 0 1 0 0 0 0 -1/estimates;
covtest ‘Ho: compound symmetry’ General
1 -1,
1 0 -1,
1 0 0 -1,
0 0 0 0 1 -1,
0 0 0 0 1 0 -1,
0 0 0 0 1 0 0 -1,
0 0 0 0 1 0 0 0 -1,
0 0 0 0 1 0 0 0 0 -1/estimates;
run;
- 10.
However, a likelihood-ratio test indicates that there is little difference between these models. The difference in -2 RLL produces a χ 2 of 2.38 (747.68 – 745.30), the difference in df = 3, and the p = .50. The null hypothesis of no difference is not rejected.
- 11.
Karney and Bradbury (1997) have applied Rosenthal's formulas to calculate effect sizes based on parameters from multilevel models.
- 12.
Benchmarks for effect sizes in educational research define a d =.20 as small and a d =.80 as large. Modest effects ranging from d =.10 to d =.20 should not be ignored. For further discussion see Borman et al. (2003).
- 13.
For each response variable the bottom four rows of the Appendix Tables 11.1 and 11.2 report the covariance parameter estimates for the full models and their statistical significance. Apparently, the KR corrections had no discernable effects on the statistical significance of these covariance parameters.
- 14.
The model composed of all of the predictors except the treatment effect coefficients also exhibits this pattern. But its BIC of 770.5 is higher than the BIC of 755.4 for the full model that includes the treatment effect coefficients––the latter model fits better.
- 15.
The models that use the Repeated statement do not readily produce the random-effects estimates for each school. This Random statement produced the estimates for Fig. 11.2 and for the other models mentioned here:
Random Intercept/Subject = School (Trt) Type = VC Solution V VCORR;
- 16.
Both composite effects have been standardized by dividing the treatment effects by the simple average of the overall ungrouped standard deviations of the SPI and CI. The SD of the SPI = 15.33 and that of the CI = 11.59; their simple average is 13.46.
- 17.
Because all of the effects of the reform treatment are positive and all of the effects of the null treatment of the not-matched group are negative, it is not reasonable to correct the set of six probabilities for multiplicity. If negative signs are attached to the probabilities for the negative effects, then Proc Multtest treats these probabilities as missing and only adjusts the probabilities for the three positive effects.
References
Aladjem, Daniel K., Kerstin Carlson LeFloch, Yu Zhang, Anja Kurki, Andrea Boyle, James E. Taylor, Suzannah Herrmann, Kazuaki Uekawa, Kerri Thomsen, and Olatokunbo Fashola. September 2006. Models matter—the final report of the national longitudinal evaluation of comprehensive school reform. Washington, DC: American Institutes of Research.
Borman, Geoffrey D., Gina M. Hewes, L.T. Overman, and S. Brown. 2003. Comprehensive school reform and achievement: A meta-analysis. Review of Educational Research 73: 125–230.
Fleiss, Joseph L. 1981. Statistical methods for rates and proportions. New York: Wiley.
Harkins, Nancy. 2001. Co-nect schools: Informational report to the Superintendent. Harford County, Maryland: Public Schools Office. July.
Karney, B.R., and T.N. Bradbury. 1997. Neuroticism, marital interaction, and the trajectory of marital satisfaction. Journal of Personality and Social Psychology 72: 1075–1092.
Lazarsfeld, Paul F., and Herbert Menzel. [1961] 1972. On the relation between individual and collective properties. In Continuities in the language of social research, ed. Paul F. Lazarsfeld, Ann K. Pasanella, and Morris Rosenberg, 219–237. New York: The Free Press.
Littell, Ramon C., George A. Milliken, Walter W. Stroup, Russell D. Wolfinger, and Oliver Schabenberger. 2006. SAS for mixed models, 2nd ed. Cary, NC: SAS Institute.
Rosenthal, Robert. 1991. Meta-analytic procedures for social research, Applied Social Research Methods Series, vol. 6, revisedth ed. Newbury Park, CA: Sage.
Russell, Michael, and Ryan Robinson. 2000. Co-nect retrospective outcomes study. Center for the study of testing, evaluation and educational policy. Chestnut Hill, MA: Boston College. April.
SAS Institute Inc. 1997. Chapter 18, the MIXED procedure. In SAS/STAT ® software: Changes and enhancements through release, vol. 6.12, 571–702. Cary, NC: SAS Institute.
SAS Institute. 2005. The Glimmix procedure, November. Cary, NC: SAS Institute.
Singer, Judith D., and John B. Willett. 2003. Applied longitudinal data analysis. New York: Oxford University Press.
Acknowledgements
The author thanks Sarah Cook and Dr. Bruce Goldberg of Co-nect for facilitating this research and encouraging its publication; Nancy Harkins for sharing information about her Harford County school system; and Philip Gibbs of the SAS Institute for clarifying nuances of PROC MIXED. Dr. Elizabeth Newton critiqued an early version of the research report; a latter version was favorably peer-reviewed by Aladjem et al. (September 2006), who included its results in their meta-analysis of comprehensive school reform. The views expressed in this chapter are the author’s own and do not necessarily reflect those of the above mentioned people or organizations.
Author information
Authors and Affiliations
Corresponding author
Appendices
Appendix
Endnotes
Footnote 1 Footnote 2 Footnote 3 Footnote 4 Footnote 5 Footnote 6 Footnote 7 Footnote 8 Footnote 9 Footnote 10 Footnote 11 Footnote 12 Footnote 13 Footnote 14 Footnote 15 Footnote 16 Footnote 17
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media B.V.
About this chapter
Cite this chapter
Smith, R.B. (2011). Target, Matched, and Not-Matched Schools. In: Multilevel Modeling of Social Problems. Springer, Dordrecht. https://doi.org/10.1007/978-90-481-9855-9_11
Download citation
DOI: https://doi.org/10.1007/978-90-481-9855-9_11
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-90-481-9854-2
Online ISBN: 978-90-481-9855-9
eBook Packages: Humanities, Social Sciences and LawSocial Sciences (R0)