Advertisement

Research in Higher Education

, Volume 60, Issue 3, pp 392–413 | Cite as

Examining a Comprehensive College Transition Program: An Account of Iterative Mixed Methods Longitudinal Survey Design

  • Darnell Cole
  • Joseph A. KitchenEmail author
  • Adrianna Kezar
Research and Practice

Abstract

There are few accounts in the higher education literature of mixing methods at the survey design stage and very little guidance targeting higher education researchers and practitioners who want to implement a mixed methods approach to design survey tools. This article explores an eight-step, iterative, mixed methods approach for creating a longitudinal, multi-institutional survey to assess how participation in a comprehensive college transition program is related to students’ psychosocial and academic outcomes. In the context of a college transition program study, our mixed methods strategy to develop a survey instrument included initial qualitative data collection and review of psychosocial constructs, psychometric pilot, baseline survey, focus groups, case study research, cognitive interviews, follow-up pilot, and follow-up survey. This article makes a unique contribution to higher education research, providing a potential model for others seeking guidance in mixing methods at the study design and instrument development stage. Major lessons learned from the mixed methods survey design process are discussed.

Keywords

Mixed-methods Survey design College transition program Postsecondary program evaluation 

Notes

Acknowledgements

This work would not have been possible without the important contributions, input, and guidance of our research partners at USC, AIR, and ISU. We are especially indebted to Matthew Soldner and Mark Masterton of AIR for the foundational work they contributed to in the development and administration of the surveys described in this article.

References

  1. Bandura, A. (1969). Principles of behavior modification. New York: Holt, Rinehart & Winston.Google Scholar
  2. Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191.Google Scholar
  3. Barnhardt, C., Sheets, J., & Pasquesi, K. (2015). You expect what? Students’ perceptions as resources in acquiring commitments and capacities for civic engagement. Research in Higher Education, 56(6), 622–644.Google Scholar
  4. Bollen, K., & Hoyle, R. (1990). Perceived cohesion: A conceptual and empirical examination. Social Forces, 69(2), 479–504.Google Scholar
  5. Bourdieu, P. (1977). Outline of a theory of practice (Vol. 16). Cambridge: Cambridge University Press.Google Scholar
  6. Bourdieu, P. (1986). The forms of capital. In J. G. Richardson (Ed.), Handbook of theory and research for the sociology of education. New York: Greenwood.Google Scholar
  7. Cabrera, N. (2011). Using a sequential exploratory mixed method design to examine racial hyperprivilege in higher education. New Directions for Institutional Research, 151, 77–91.Google Scholar
  8. Callie, T., & Cheslock, J. (2008). The hiring and compensation practices of business school deans. The Review of Higher Education, 32(1), 25–49.Google Scholar
  9. Chesborough, R. (2011). College students and service: A mixed methods exploration of motivations, choices, and learning outcomes. Journal of College Student Development, 52(6), 687–705.Google Scholar
  10. Choi, N. (2005). Self-efficacy and self-concept as predictors of college students’ academic performance. Psychology in the Schools, 42(2), 197–205.Google Scholar
  11. Creswell, J. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Sage: Thousand Oaks, CA.Google Scholar
  12. Creswell, J. W., Plano Clark, V. L., Gutmann, M. L., & Hanson, W. E. (2003). Advanced mixed methods research designs. In A. Tashakkori & C. Teddlie (Eds.), Handbook on mixed methods in the behavioral and social sciences (pp. 209–240). Thousand Oaks, CA: Sage Publications.Google Scholar
  13. Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. New York: Wiley.Google Scholar
  14. Garmezy, N., & Masten, A. S. (1991). The protective role of competence indicators in children at risk. In E. M. Cummings, A. L. Greene, & K. H. Karraker (Eds.), Life-span developmental psychology: Perspectives on stress and coping (pp. 151–174). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  15. Gasiewski, J., Eagan, M., Garcia, G., Hurtado, S., & Chang, M. (2012). From gatekeeping to engagement: A multicontextual, mixed method study of student academic engagement in introductory STEM courses. Research in Higher Education, 53(2), 229–261.Google Scholar
  16. Griffin, K., & Museus, S. (2011). Application of mixed-methods approaches to higher education and intersectional analyses. New Directions for Institutional Research, 151, 15–26.Google Scholar
  17. Hausmann, L. R., Schofield, J. W., & Woods, R. L. (2007). Sense of belonging as a predictor of intentions to persist among African American and White first-year college students. Research in Higher Education, 48(7), 803–839.Google Scholar
  18. Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1(1), 104–121.Google Scholar
  19. Hora, M., & Anderson, C. (2012). Perceived norms for interactive teaching and their relationship to instructional decision-making: A mixed methods study. Higher Education, 64, 573–592.Google Scholar
  20. Hurtado, S., & Carter, D. F. (1997). Effects of college transition and perceptions of the campus racial climate on Latino college students’ sense of belonging. Sociology of Education, 70, 324–345.Google Scholar
  21. Inkelas, K. (2007). National survey of living-learning programs. Report of findings: 2007. Retrieved from http://drum.lib.umd.edu/handle/1903/8392.
  22. Inkelas, K. K., Vogt, K., Longerbeam, S., Owen, J., & Johnson, D. (2006). Measuring outcomes of living-learning programs: Examining college environments and student learning and development. Journal of General Education, 55(1), 40–76.Google Scholar
  23. Ivankova, N., & Stick, S. (2007). Students’ persistence in a distributed doctoral program in educational leadership in higher education: A mixed methods study. Research in Higher Education, 48(1), 93–135.Google Scholar
  24. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24(4), 602–611.Google Scholar
  25. Johnson, B., & Onwuegbuzie, A. (2004). Mixed methods research: A research paradigm whose time as home. Educational Researcher, 33(7), 14–26.Google Scholar
  26. Johnson, B., Onwuegbuzie, A., & Turner, L. (2007). Toward a definition of mixed methods research. Journal of Mixed Methods Research, 1(2), 112–133.Google Scholar
  27. Lazarsfeld, P. F. (1944). The controversy over detailed interviews: An offer for negotiation. Public Opinion Quarterly, 8, 38–60.Google Scholar
  28. Leech, N., & Onwuegbuzie, A. (2009). A typology of mixed methods research designs. Quality & Quantity, 43, 265–275.Google Scholar
  29. Lent, R. W., Brown, S. D., & Hackett, G. (2002). Social cognitive career theory. Career Choice and Development, 4, 255–311.Google Scholar
  30. Litwin, M. S., & Fink, A. (1995). How to measure survey reliability and validity (Vol. 7). Thousand Oaks: Sage.Google Scholar
  31. Masten, A., & Obradović, J. (2006). Competence and resilience in development. Annals of the New York Academy of Sciences, 1094(1), 13–27.Google Scholar
  32. Masten, A., & Obradović, J. (2008). Disaster preparation and recovery: Lessons from research on resilience in human development. Ecology and Society, 13(1), 9.Google Scholar
  33. Mertes, S., & Jankoviak, M. (2016). Creating a college-wide retention program: A mixed methods approach. Community College Enterprise, 22(1), 9–27.Google Scholar
  34. Ouimet, J. A., Bunnage, J. C., Carini, R. M., Kuh, G. D., & Kennedy, J. (2004). Using focus groups, expert advice, and cognitive interviews to establish the validity of a college student survey. Research in Higher Education, 45(3), 233–250.Google Scholar
  35. Pajares, F., & Schunk, D. H. (2001). Self-beliefs and school success: Self-efficacy, self-concept, and school achievement. Perception, 11, 239–266.Google Scholar
  36. Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407.Google Scholar
  37. Ray, A., & Margaret, W. (Eds.). (2003). PISA Programme for international student assessment (PISA) PISA 2000 technical report: PISA 2000 technical report. Paris: OECD Publishing.Google Scholar
  38. Ruthig, J., Haynes, T., Stupnisky, R., & Perry, R. (2009). Perceived academic control: Mediating the effects of optimism and social support on college students’ psychological health. Social Psychology of Education, 12(2), 233–249.Google Scholar
  39. Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19(4), 460–475.Google Scholar
  40. Smith, B. L., MacGregor, J., Matthews, R., & Gabelnick, F. (2004). Learning communities: Reforming undergraduate education. San Francisco: Jossey-Bass.Google Scholar
  41. Soldner, M., Rowan-Kenyon, H., Inkelas, K. K., Garvey, J., & Robbins, C. (2012). Supporting students’ intentions to persist in STEM disciplines: The role of living-learning programs among other social-cognitive factors. The Journal of Higher Education, 83(3), 311–336.Google Scholar
  42. Strayhorn, T. L. (2015). Factors influencing Black males’ preparation for college and success in STEM majors: A mixed methods study. Western Journal of Black Studies, 39(1), 45–63.Google Scholar
  43. Tashakkori, A., & Teddlie, C. (2010). Putting the human back in “human research methodology”: The researcher in mixed methods research. Journal of Mixed Methods Research, 4(4), 271–277.Google Scholar
  44. Teddlie, C., & Tashakkori, A. (2010). Overview of contemporary issues in mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (Vol. 2, pp. 1–44). Thousand Oaks: Sage.Google Scholar
  45. Torres, V. (2006). A mixed method study testing data-model fit of a retention model for Latino/a students at urban universities. Journal of College Student Development, 47(3), 299–318.Google Scholar
  46. Tourangeau, R. (1984). Cognitive sciences and survey methods. In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines (pp. 73–100). Washington, DC: National Academy Press.Google Scholar
  47. Vuong, M., Brown-Welty, S., & Tracz, S. (2010). The effects of self-efficacy on academic success of first-generation college sophomore students. Journal of College Student Development, 51(1), 50–64.Google Scholar
  48. Wells, R., Kolek, E., Williams, E., & Saunders, D. (2015). “How we know what we know”: A systematic comparison of research methods employed in higher education journals, 1996–-2000 v. 2006–2010. Journal of Higher Education, 86(2), 171–195.Google Scholar
  49. Willis, G. (1999). Reducing survey error through research on the cognitive and decision processes in surveys. Short course presented at the 1999 meeting of the American Statistical Association. Research Triangle Institute.Google Scholar
  50. Willis, G. (2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage Publications.Google Scholar
  51. Young, A., & Fry, J. D. (2008). Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2), 1–10.Google Scholar
  52. Zimmerman, M. A., & Arunkumar, R. (1994). Resiliency research: Implications for schools and policy. Social Policy Report, 8(4), 1–18.Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  • Darnell Cole
    • 1
  • Joseph A. Kitchen
    • 1
    Email author
  • Adrianna Kezar
    • 1
  1. 1.Rossier School of EducationUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations