Abstract
In responding to the need for internationally comparable data on higher education student learning outcomes, some modules of the HEIghten ® Outcomes Assessment Suite, developed by Educational Testing Service, have been translated and adapted for international use. This recent development points to a critical need to validate the use of translated and adapted HEIghten assessments in international contexts. This chapter reports on validating the use of the Russian HEIghten Quantitative Literacy (QL) assessment with a representative group of students majoring in electrical engineering and computer science from 34 higher education institutions in Russia. Our findings provided preliminary evidence in support of the use of the assessment for the target population as a measure of QL. Future research is suggested to further investigate the test’s ability of reflecting changes in the target construct as a function of learning in the context of Russia.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In the Russian higher education system, a university is classified as elite if it has the status of being a Federal University or a National Research University. All six elite universities in our sample were National Research Universities.
- 2.
The 75% test completion rate is used for the operational HEIghten assessments to screen out students with low testing motivation.
- 3.
Since there are no clear guidelines for the minimum sample size needed for estimating group-level reliability using this approach, we decided not to exclude schools due to small sample sizes.
- 4.
Relatively low reliabilities at the subscale level were expected and should not be a concern because, by design, HEIghten scores are not reported to individual students.
- 5.
In the Russian educational system, advanced high schools offer educational programs of higher level than regular high schools.
References
American College Testing. (2012). ACT CAAP technical handbook 2011–2012. Iowa City: CAAP Program Management. Retrieved from http://www.act.org/content/dam/act/unsecured/documents/CAAP-TechnicalHandbook.pdf
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Bennell, P., & Pierce, T. (2003). The internationalisation of tertiary education: Exporting education to developing and transitional economies. International Journal of Educational Development, 23, 215–232.
Busato, V. V., Prins, F. J., Elshout, J. J., & Hamaker, C. (2000). Intellectual ability, learning style, personality, achievement motivation and academic success of psychology students in higher education. Personality and Individual Differences, 29, 1057–1068.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale: L. Erlbaum Associates.
Council for Aid to Education. (2015). CLA+ technical FAQs. New York: Author. Retrieved from http://cae.org/images/uploads/pdf/CLA_Plus_Technical_FAQs.pdf
Educational Testing Service. (2010). ETS proficiency profile user’s guide. Princeton: Educational Testing Service. Retrieved from https://www.ets.org/s/proficiencyprofile/pdf/Users_Guide.pdf
Finney, S. J., & DiStefano, C. (2013). Nonnormal and categorical data in structural equation models. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 439–492). Charlotte: Information Age.
Haberman, S. J. (2008). When can subscores have value? Journal of Educational and Behavioral Statistics, 33, 204–229.
International Test Commission. (2005). International Test Commission guidelines for translating and adapting tests. Retrieved from http://www.intestcom.org/files/guideline_test_adaptation.pdf
Klein, S., Benjamin, R., Shavelson, R. J., & Bolus, R. (2007). The collegiate learning assessment: Facts and fantasies. Evaluation Review, 31, 415–439.
Klein, S., Liu, O. L., Sconing, J. A., Bolus, R., Bridgeman, B., Kugelmass, H., & Steedle, J. (2009). Test Validity Study (TVS) Report (ETS technical report). Supported by the Fund for Improvement of postsecondary education (FIPSE). Retrieved from https://cp-files.s3.amazonaws.com/26/TVSReport_Final.pdf
Knight, J. (2003). Updated definition of internationalization. International Higher Education, 33, 2–3.
Lizzio, A., Wilson, K., & Simons, R. (2002). University students’ perceptions of the learning environment and academic outcomes: Implications for theory and practice. Studies in Higher Education, 27, 27–52.
Muthén, L. K., & Muthén, B. O. (2010). Mplus user’s guide (6th ed.). Los Angeles: Authors.
Organization for Economic Co-operation and Development. (2008). Tertiary education for the knowledge society. Paris: OECD. Retrieved from https://oecd.org/edu/tertiary/review.
Organization for Economic Co-operation and Development. (2012). PISA 2012 technical report. Paris: OECD. Retrieved from https://www.oecd.org/pisa/pisaproducts/PISA-2012-technical-report-final.pdf.
Roohr, K. C., Graf, E. A., & Liu, O. L. (2014). Assessing quantitative literacy in higher education: An overview of existing research and assessments with recommendations for next-generation assessment (ETS RR-14-22). Princeton: Educational Testing Service.
Roohr, K. C., Lee, H., Xu, J., Liu, O. L., & Wang, Z. (2017). A preliminary evaluation of the psychometric quality of HEIghten™ quantitative literacy. Numeracy, 10(2), 3.
Sawaki, Y., Stricker, L. J., & Oranje, A. H. (2009). Factor structure of the TOEFL internet-based test. Language Testing, 26, 5–30.
Tremblay, K., Lalancette, D., & Roseveare, D. (2012). Assessment of higher education learning outcomes (AHELO) feasibility study report: Design and implementation (Vol. 1). Paris: Organization for Economic Co-operation and Development.
Trigwell, K., & Prosser, M. (1991). Improving the quality of student learning: The influence of learning context and student approaches to learning on learning outcomes. Higher Education, 22, 251–266.
Acknowledgments
Support from the Basic Research Program of the National Research University Higher School of Economics is gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Gu, L. et al. (2018). Validating the Use of Translated and Adapted HEIghten ® Quantitative Literacy Test in Russia. In: Zlatkin-Troitschanskaia, O., Toepper, M., Pant, H., Lautenbach, C., Kuhn, C. (eds) Assessment of Learning Outcomes in Higher Education. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-74338-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-74338-7_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-74337-0
Online ISBN: 978-3-319-74338-7
eBook Packages: EducationEducation (R0)