Abstract
As Federal agencies have increasingly specified the methodology expected in the program evaluations that they fund, the long-standing debate about what constitutes scientifically based research has been resurrected. In fact, there are no simple answers to questions about how well programs work, nor is there a single analytic approach to evaluate the wide variety of possible programs and their complexities. Evaluators need to be familiar with a range of analytic methods, and it is often necessary to use several methods simultaneously, including both quantitative and qualitative approaches. Some evaluation approaches are particularly helpful in the early developmental stages of a program, whereas others are more suited to situations in which the program has become more routinized and broadly implemented. One of the key points that we stress in this chapter is that the evaluation design should utilize the most rigorous method possible to address the questions posed and should be appropriately matched to the program’s developmental status. The warnings not to evaluate a developing program with an experimental design have been sounded for some time [Patton, M.Q. (2008). Utilization focused evaluation (4th ed). Thousand Oaks, CA: Sage; Lipsey, M. (2005). Improving evaluation of anticrime programs. Washington, DC: National Academies Press], but this is the first time to our knowledge that the developmental phases of program development have been specified in detail and linked to evaluation designs.
To guide design decisions, we present a framework that builds on the goal structure devised by the Department of Education’s (DoE) Institute of Education Sciences (IES). It is important to note however that designing a program evaluation is often a complicated process, and the ultimate design rarely comes straight out of a textbook. Rather, design decisions require responsiveness and judgment particular to each setting, given the practical constraints of time and resources. Our intent is therefore not to be prescriptive, but rather to provide guidance, tools, and resources to help novice evaluators develop evaluation designs that are responsive to the needs of the client and appropriate to the developmental status of the program.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Altschuld, J., & Kumar, D. (2010). Needs assessment: An overview. Thousand Oaks, CA: Sage.
American Educational Research Association. (2008). Definition of scientifically based research. Retrieved April 2011, from http://www.aera.net/opportunities/?id=6790
American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved February 2012, from http://www.eval.org/Publications/GuidingPrinciplesPrintable.asp
American Evaluation Association. (2009). An evaluation roadmap for a more effective government. Retrieved February 2012, from http://evaluationcenter.net/Documents/aea 09.eptf.eval.roadmap.pdf
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25.
Bickman, L., & Henchy, T. (1971). Beyond the laboratory: Field research in social psychology. New York, NY: McGraw Hill.
Century, J., Rudnick, M., & Freeman, C. (2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199–218.
Chatterji, M. (2004). Evidence on “What Works”: An argument for extended-term mixed method (ETMM) evaluation. Educational Researcher, 33(9), 3–13.
*Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum Associates.
Cook, T. (2002). Randomized experiments in education: Why are they so rare? Education Evaluation and Policy Analysis, 24(3), 175–200.
*Cordray, D. (2007). Assessment of treatment implementation and assessment of the control condition. Retrieved March 2012, from http://ies.ed.gov/ncer/whatsnew/conferences/rct_traininginstitute/presentations.asp
Dane, A., & Schneider, B. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45.
Eisenhart, M., & Towne, L. (2003). Contestation and change in national policy on “scientifically based” education research. Educational Researcher, 32(7), 31–38.
Erickson, F., & Gutierrez, K. (2002). Culture, rigor, and science in educational research. Educational Researcher, 31(8), 21–24.
Evaluation Research Society. (1982). Evaluation research society standards for program evaluation. New Directions for Program Evaluation, 15, 7–19.
Flagg, B. (1990). Formative evaluation for educational technologies. Hillsdale, NJ: Lawrence Erlbaum Associates.
Foundation, N. S. (2010). The 2010 user-friendly handbook for project evaluation. Washington, DC: Directorate for education and human resources.
Hedrick, T. E., Bickman, L., & Rog, D. J. (1993). Applied research design: A practical guide. Thousand Oaks, CA: Sage.
IES (2012a). Retrieved January 2012, from http://ies.ed.gov/aboutus
IES (2012b). Request for applications: Education research grants. CFDA number 84.305A. Retrieved March 2012, from http://ies.ed.gov/funding/pdf/2013_84305A.pdf
Joint Committee on Standards for Educational Evaluation. (2010). The program evaluation standards (3rd ed.). Thousand Oaks, CA: Sage.
Julnes, G., & Rog, D. (2007). Informing federal policies on evaluation methodology: Building the evidence base for method choice in government sponsored evaluation. San Francisco, CA: Jossey-Bass.
Kirk, R. (1995). Experimental design: Procedures for the behavioral sciences (3rd ed.). Pacific Grove, CA: Brooks/Cole Publishing.
Levin-Rozalis, M. (2003). Evaluation and research, differences and similarities. The Canadian Journal of Program Evaluation, 18(2), 1–31.
Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research. Thousand Oaks, CA: Sage.
Lipsey, M. (2005). Improving evaluation of anticrime programs. Washington, DC: National Academies Press.
Lofland, J., & Lofland, L. (1995). Analyzing social settings: A guide to qualitative observation and analysis. Belmont, CA: Wadsworth.
Madaus, G., Scriven, M., & Stufflebeam, D. (1996). Evaluation models: Viewpoints on educational and human services evaluation. Boston, MA: Kuwer-Nijhoff Publishing.
Mark, M., & Shotland, R. (1987). Multiple methods in program evaluation. San Francisco, CA: Jossey-Bass.
Mayeske, G., & Lambur, M. (2001). How to design better programs: A staff centered stakeholder approach to program logic modeling. Crofton, MD: The Program Design Institute.
McKenney, S. E., & Reeves, T. C. (2012). Conducting educational design research. New York, NY: Routledge.
O’Donnell, C. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78(1), 33–84.
Office of Juvenile Justice and Delinquency Prevention. (2003). Evaluability assessment: Examining the readiness of a program for evaluation. Justice Research and Statistics Association. Retrieved March 2012, from http://www.jrsa.org/pubs/juv-justice/evaluability-assessment.pdf
Olson, D. R. (2004). The triumph of hope over experience in the search for “What Works”: A response to slavin. Educational Researcher, 33(1), 24–26.
Patton, M. Q. (1987). Evaluation’s political inherency: Practical implications for design and use. In D. Palumbo (Ed.), The politics of program evaluation (pp. 100–145). Beverly Hills, CA: Sage.
*Patton, M. Q. (1990). Qualitative evaluation and research methods (2nd ed.), Thousand Oaks, CA: Sage.
Patton, M. Q. (2008). Utilization focused evaluation (4th ed.). Thousand Oaks, CA: Sage.
Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation. New York, NY: Guilford Press.
Reeves, T., & Hedberg, J. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology.
Resnick, B., Bellg, A., Borrelli, B., DeFrancesco, C., Breger, R., Hecht, J., et al. (2005). Examples of implementation and evaluation of treatment fidelity in the BCC studies: Where we are and where we need to go. Annals of Behavioral Medicine, 29, 46–54.
Rosenbaum, P., & Rubin, D. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Scheirer, M., & Rezmovic, E. (1983). Measuring the degree of program implementation: A methodological review. Evaluation Review, 7(5), 599–633.
Schoenfeld, A. (2006). What doesn’t work: The challenge and failure of the what works clearinghouse to conduct meaningful reviews of studies of mathematics curricula. Educational Researcher, 35(2), 13–21.
Scriven, M. (1981). The logic of evaluation. Inverness, CA: Edgepress.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage.
*Shadish, W., Cook, T., & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
Slavin, R. (2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31(7), 15–21.
Slavin, R. (2004). Education research can and must address “What Works” questions. Educational Researcher, 33(1), 27–28.
Smith, M. (1989). Evaluability assessment: A practical approach. Clemson: Kluwer.
St. Pierre, E. A. (2002). “Science” rejects postmodernism. Educational Research, 31(8), 25–27.
Stewart, D., & Shamdasani, P. (1990). Focus groups: Theory and practice. Newbury Park, CA: Sage.
Stufflebeam, D., & Shinkfield, A. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey-Bass.
Teijlingen, E., & Hundley, V. (2001). The importance of pilot studies. Social Research Update (35). UK: University of Surrey.
US Department of Education. (2002). Strategic plan 2002-2007. Washington, DC: Author.
US Department of Education. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user-friendly guide. Washington, DC: Institute for Education Science.
US Department of Education. (2004). New directions for program evaluation at the US Department of Education. Retrieved January 2012, from http://www2.ed.gov/news/pressreleases/2002/04/evaluation.html
US Department of Education. (2005). Scientifically based evaluation methods. RIN 1890-ZA00. Federal Register, 70(15), 3586–3589.
US Department of Education. (2007). Report of the academic competitiveness council. Washington, DC: Author.
US General Accounting Office. (1990). Case study evaluations. Washington, DC: Program Evaluation and Methodology Division.
W.K. Kellogg Foundation. (2004). Logic model development guide: Using logic models to bring together planning, evaluation, and action. Retrieved March 2011, from http:// www.wkkf.org /∼/ media /20 B 6036478 FA 46 C 580577970 AFC 3600 C.ashx
Weiss, C. H. (1972). Evaluation research: Methods for assessing program effectiveness. Englewood Cliffs, NJ: Prentice Hall.
*What Works Clearinghouse. (2011). What Works Clearinghouse procedures and standards handbook v. 2.1. Retrieved March 2012, from http://ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19
Yarbrough, D., Shulha, L., Hopson, R., & Caruthers, F. (2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.
Youtie, J., Bozeman, B., & Shapira, P. (1999). Using an evaluability assessment to select methods for evaluating state technology development programs: The case of the Georgia Research Alliance. Evaluation and Program Planning, 22(1), 55–64.
Acknowledgements
We would like to express our gratitude to Debra Rog and Cindy Tananis who provided input and friendly advice on this chapter. We also wish to thank the anonymous reviewers whose comments and questions greatly improved our work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this chapter
Cite this chapter
Hamilton, J., Feldman, J. (2014). Planning a Program Evaluation: Matching Methodology to Program Status. In: Spector, J., Merrill, M., Elen, J., Bishop, M. (eds) Handbook of Research on Educational Communications and Technology. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3185-5_20
Download citation
DOI: https://doi.org/10.1007/978-1-4614-3185-5_20
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-3184-8
Online ISBN: 978-1-4614-3185-5
eBook Packages: Humanities, Social Sciences and LawEducation (R0)