Advertisement

Knowing is Half the Battle, or Is It? A Randomized Experiment of the Impact of Supplemental Notification Letters on Placement Exam Participation, Preparation, and Performance

  • Brian G. Moss
  • Peter Riley Bahr
  • Leigh Arsenault
  • Meghan Oster
Article
  • 22 Downloads

Abstract

Community college students often are unaware of the stakes involved in their performance on placement exams, used to sort the students into their first math and English courses. In our randomized experiment of approximately 13,000 newly admitted community college students, we test whether informing students, via a supplemental notification letter, about the implications of placement exam scores influences their placement exam participation, preparation, and performance. We also test whether message framing (loss-framed vs. gain-framed) alters the effect of the information on study outcomes. We find that neither type of message has an effect on placement exam preparation or performance. We find limited evidence to suggest that the loss-framed message may have a small effect on the avoidance of placement exams by increasing the likelihood of students’ submitting alternative evidence of academic preparation. Overall, our evidence suggests that students’ prior knowledge about the ramifications of placement exams is not enough to improve performance. Colleges should consider alternative means of assessing college readiness, such as measures of high school achievement, and should provide students who still have to take placement exams with detailed information about test content and study materials before testing.

Keywords

Community college Assessment Placement Developmental education Remedial Intervention Experiment Randomized trial 

Notes

Acknowledgements

An earlier version of this study was presented at the 2016 annual meeting of the Association for the Study of Higher Education in Columbus, Ohio. The authors thank the Editor and anonymous referees of the journal Research in Higher Education for feedback on this work. In addition, the authors thank the administration of the study college for support of this investigation, and also thank Susan Dynarski (University of Michigan) for the suggestion that ultimately led to this investigation.

References

  1. Alferes, V. R. (2012). Methods of randomization in experimental design (Vol. 171). Thousand Oaks, CA: Sage Publications.CrossRefGoogle Scholar
  2. Attewell, P., Lavin, D., Domina, T., & Levey, T. (2006). New evidence on college remediation. Journal of Higher Education, 77, 886–924.CrossRefGoogle Scholar
  3. Bahr, P. R. (2008). Does mathematics remediation work? A comparative analysis of academic attainment among community college students. Research in Higher Education, 49, 420–450.CrossRefGoogle Scholar
  4. Bahr, P. R. (2009). Revisiting the efficacy of postsecondary remediation: The moderating effects of depth/breadth of deficiency. Review of Higher Education, 33, 177–205.CrossRefGoogle Scholar
  5. Bahr, P. R. (2016). Replacing placement tests in Michigan’s community colleges. Ann Arbor: University of Michigan.Google Scholar
  6. Bahr, P. R., Fagioli, L. P., Hetts, J., Hayward, C., Willett, T., Lamoree, D., Newell, M. A., Sorey, K., & Baker, R. B. (in press). Improving placement accuracy in California’s community colleges using multiple measures of high school achievement. Community College Review.Google Scholar
  7. Bahr, P. R., & Gross, J. (2016). Community colleges. In M. N. Bastedo, P. G. Altbach, & P. J. Gumport (Eds.), American higher education in the 21st century (4th ed., pp. 462–502). Baltimore: John Hopkins University.Google Scholar
  8. Bailey, T., Jeong, D. W., & Cho, S. (2010). Referral, enrollment, and completion in developmental education sequences in community colleges. Economics of Education Review, 29, 255–270.CrossRefGoogle Scholar
  9. Banks, S. M., Salovery, P., Greener, S., Rothman, A. J., Moyer, A., Beauvais, J., et al. (1995). The effects of message framing on mammography utilization. Health Psychology, 14, 178–184.CrossRefGoogle Scholar
  10. Belfield, C. R., & Crosta, P. M. (2012). Predicting success in college: The importance of placement tests and high school transcripts (CCRC Working Paper No. 42). New York: Columbia University.Google Scholar
  11. Bettinger, E. P., Boatman, A., & Long, B. T. (2013). Student supports: Developmental education and other academic programs. The Future of Children, 23, 93–115.CrossRefGoogle Scholar
  12. Block, L. G., & Keller, P. A. (1995). When to accentuate the negative: The effects of perceived efficacy and message framing on intentions to perform a health-related behavior. Journal of Marketing Research, 32, 192–203.CrossRefGoogle Scholar
  13. Burdman, P. (2012). Where to begin? The evolving role of placement exams for students starting college. Boston: Jobs for the Future. Retrieved from Jobs for the Future website: http://www.jff.org/publications/where-begin-evolving-role-placement-exams-students-starting-college.
  14. DesJardins, S. L., & Toutkoushian, R. K. (2005). Are students really rational? The development of rational thought and its application to student choice. Higher Education: Handbook of Theory and Research (pp. 191–240). Berlin: Springer.CrossRefGoogle Scholar
  15. Fay, M. P., Bickerstaff, S., & Hodara, M. (2013). Why students do not prepare for math placement exams: Student perspectives (CCRC Brief No. 57). New York: Columbia University.Google Scholar
  16. Fields, R., & Parsad, B. (2012). Tests and cut scores used for student placement in postsecondary education: Fall 2011. Washington, DC: National Assessment Governing Board. Retrieved from https://www.nagb.org/content/nagb/assets/documents/commission/researchandresources/test-and-cut-scores-used-for-student-placement-in-postsecondary-education-fall-2011.pdf.
  17. Ginder, S. A., Kelly-Reid, J. E., & Mann, F. B. (2015). Enrollment and Employees in Postsecondary Institutions, Fall 2014; and Financial Statistics and Academic Libraries, Fiscal Year 2014: First Look (Provisional Data) (NCES 2016-005). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubsearch.
  18. Graubard, B. I., & Korn, E. L. (1999). Predictive margins with survey data. Biometrics, 55, 652–659.CrossRefGoogle Scholar
  19. Grewal, D., Gotlieb, J., & Marmorstein, H. (1994). The moderating effects of message framing and source credibility on the price-perceived risk relationship. Journal of Consumer Research, 21, 145–153.CrossRefGoogle Scholar
  20. Hodara, M., Jaggars, S. S., & Karp, M. M. (2012). Improving developmental education assessment and placement: Lessons from community colleges across the country (CCRC Working Paper No. 51). New York: Columbia University.Google Scholar
  21. Hughes, K. L., & Scott-Clayton, J. (2011). Assessing developmental assessment in community colleges. Community College Review, 39, 327–351.CrossRefGoogle Scholar
  22. Jones, B. D. (1999). Bounded rationality. Annual Review of Political Science, 2, 297–321.CrossRefGoogle Scholar
  23. Lauvey, D., & Rubin, M. (1990). Message framing, dispositional optimism, and follow-up for abnormal Papanicolaou tests. Research in Nursing & Health, 13, 199–207.CrossRefGoogle Scholar
  24. Lenhart, A. (2012). Teens, smartphones & texting. Pew Internet & American Life Project, 21, 1–34.Google Scholar
  25. Lesik, S. A. (2006). Applying the regression-discontinuity design to infer causality with non-random assignment. The Review of Higher Education, 30, 1–19.CrossRefGoogle Scholar
  26. Levin, I. P., & Gaeth, G. J. (1988). How consumers are affected by the framing of attribute information before and after consuming the product. Journal of Consumer Research, 15, 374–378.CrossRefGoogle Scholar
  27. Maheswaran, D., & Meyers-Levy, J. (1990). The influence of message framing and issue involvement. Journal of Marketing Research, 27, 361–367.CrossRefGoogle Scholar
  28. March, J. G. (1994). Primer on decision making: How decisions happen. New York: Simon and Schuster.Google Scholar
  29. Meyerowitz, B. E., & Chaiken, S. (1987). The effect of message framing on breast self-examination attitudes, intentions, and behavior. Journal of Personality and Social Psychology, 52, 500–510.CrossRefGoogle Scholar
  30. Moss, B. G., & Yeaton, W. H. (2006). Shaping policies related to developmental education: An evaluation using the regression-discontinuity design. Educational Evaluation and Policy Analysis, 28, 215–229.CrossRefGoogle Scholar
  31. Moss, B. G., & Yeaton, W. H. (2013). Evaluating effects of developmental education for college students using a regression discontinuity design. Evaluation Review, 37, 370–404.CrossRefGoogle Scholar
  32. Moss, B. G., Yeaton, W. H., & Lloyd, J. (2014). Estimating the effectiveness of developmental mathematics by embedding a randomized experiment within a regression discontinuity design. Educational Evaluation and Policy Analysis., 36, 170–185.CrossRefGoogle Scholar
  33. Rothman, A. J., Salovey, P., Antone, C., Keough, K., & Martin, C. D. (1993). The influence of message framing on intentions to perform health behaviors. Journal of Experimental Social Psychology, 29, 408–433.CrossRefGoogle Scholar
  34. Safran, S., & Visher, M. G. (2010). Case studies of three community colleges: The policy and practice of assessing and placing students in developmental education courses. New York: Columbia University.Google Scholar
  35. Scott-Clayton, J. (2012). Do high-stakes exams predict college success? (CCRC Working Paper No. 41). New York: Columbia University.Google Scholar
  36. Shadish, W. R., Cook, T. D., & Campbell, T. D. (2002). Experimental and quasi-experimental design for generalized causal inference. Boston: Houghton Mifflin.Google Scholar
  37. Simon, H. A. (1991). Bounded rationality and organizational learning. Organization Science, 2, 125–134.CrossRefGoogle Scholar
  38. Tversky, A., & Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453–458.CrossRefGoogle Scholar
  39. Tversky, A., & Kahneman, D. (1986). Rational choice and the framing of decisions. The Journal of Business, 59, 251–278.CrossRefGoogle Scholar
  40. Venezia, A., Bracco, K. R., & Nodine, T. (2010). One-shot deal? Students’ perceptions of assessment and course placement in California’s community colleges. San Francisco, CA: WestEd.Google Scholar
  41. Witte, K., & Allen, M. (2000). A meta-analysis of fear appeals: Implications for effective public health campaigns. Health Education and Behavior, 27, 591–615.CrossRefGoogle Scholar
  42. Yeaton, W. H., & Moss, B. G. (in press). A multiple-design, experimental strategy: Academic probation warning letter’s impact on student achievement. The Journal of Experimental Education.Google Scholar

Copyright information

© Springer Nature B.V 2018

Authors and Affiliations

  • Brian G. Moss
    • 1
  • Peter Riley Bahr
    • 2
  • Leigh Arsenault
    • 2
  • Meghan Oster
    • 2
  1. 1.Wayne State UniversityDetroitUSA
  2. 2.Center for the Study of Higher and Postsecondary EducationUniversity of MichiganAnn ArborUSA

Personalised recommendations