Abstract
This chapter considers how the process-based variables of test-taking strategies as reported by test-takers can help to explain the differences in the outcome of a reading comprehension test and serve to provide process level evidence of validity. With the process variables as the explanatory variables, test-takers’ performance was analyzed via a latent variable regression in a structural equation model (SEM), along with Pratt’s importance measures (Pratt, 1987) to assist in understanding the score variation in the latent outcome. We consider how understanding test-taking strategy can help inform test design and validation practices.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abbott, M. L. (2006). ESL reading strategies: Differences in Arabic and Mandarin speaker test performance. Language Learning, 56, 633–670.
Allen, S. (2003). An analytic comparison of three models of reading strategy instruction. International Review of Applied Linguistics in Language Teaching, 41, 319–338.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education [AERA, APA, & NCME]. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8, 41–66.
Cohen, A. D. (1984). On taking language tests what the students report. Language Testing, 1, 70–81.
Cohen, A. D. (2006). The coming of age of research on test-taking strategies. Language Assessment Quarterly, 3, 307–331.
Cohen, A. D. (2012a). Test-taking strategies. In C. Coombe, P. Davidson, B. O’Sullivan, & S. Stoynoff (Eds.), The Cambridge guide to second language assessment (pp. 96–104). Cambridge, UK: Cambridge University Press.
Cohen, A. D. (2012b). Test taker strategies and test design. In G. Fulcher & F. Davidson (Eds.), The Routledge handbook of language testing in a nutshell (pp. 262–277). Abingdon, UK: Routledge.
Cohen, A. D., & Upton, T. A. (2006). Strategies in responding to the new TOEFL reading tasks. In TOEFL Monograph Series, MS-33. Princeton, NJ: Educational Testing Services.
Cohen, A. D., & Upton, T. A. (2007). ‘I want to go back to the text’: Response strategies on the reading subtest of the new TOEFL®. Language Testing, 24, 209–250.
Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27, 209–226.
Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38, 319–342.
Lee, J. Y. (2011). Second language reading topic familiarity and test score: test-taking strategies for multiple-choice comprehension questions (Doctoral dissertation). The University of Iowa, Iowa City, IA. Retrieved from http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2717&context=etd&unstamped=1
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741–749.
Muthén, L. K., & Muthén, B. O. (1998–2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Authors.
Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6, 199–215.
Newton, P. E. (2016). Macro- and micro-validation: Beyond the ‘five sources’ framework for classifying validation evidence and analysis. Practical Assessment, Research & Evaluation, 21, 1–13. Available online: http://pareonline.net/getvn.asp?v=21&n=12
Pratt, J. W. (1987). Dividing the indivisible: Using simple symmetry to partition variance explained. In T. Pukilla & S. Duntaneu (Eds.), Proceedings of second Tampere conference in statistics (pp. 245–260). Tampere, Finland: University of Tampere.
Purpura, J. E. (1998). Investigating the effects of strategy use and second language test performance with high-and low-ability test takers: A structural equation modelling approach. Language Testing, 15, 333–379.
Roizen, M. A. (1984). Test performance vis a vis test-taking strategies in reading comprehension of English as a second language. ERIC Document Reproduction Service No. ED 224 350. Retrieved from http://ir.uiowa.edu/cgi/viewcontent.cgi?article=2717&context=etd&unstamped=1
Rupp, A. A., Ferne, T., & Choi, H. (2006). How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing, 23, 441–474.
Storey, P. (1997). Examining the test-taking process: A cognitive perspective on the discourse cloze test. Language Testing, 14, 214–231.
Thomas, D. R., Hughes, E., & Zumbo, B. D. (1998). On variable importance in linear regression. Social Indicators Research, 45, 253–275.
Wu, A. D., & Stone, J. E. (2015). Validation through understanding test-taking strategies: An illustration with the CELPIP-General reading pilot test using structural equation modeling. Journal of Psychoeducational Assessment, 34, 1–18.
Zumbo, B. D. (2005, July). Reflections on validity at the intersection of psychometrics, scaling, philosophy of inquiry, and language testing. Samuel J. Messick Memorial Award Lecture. Lecture provided at the 27th Language Testing Research Colloguium, Ottawa, Canada.
Zumbo, B. D. (2007). Validity: Foundational issues and statistical methodology. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Psychometrics (Vol. 26, pp. 45–79). Amsterdam, The Netherlands: Elsevier Science B.V.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendices
Appendix A
Statements of the test-taking strategy survey items and descriptive statistics
Type | Item # | When answering a question, … | M | SD |
---|---|---|---|---|
CM | 1 | I needed to understand the main ideas of the passage. | 3.81 | 1.07 |
2 | I needed to understand some specific sentences in the passage. | 3.73 | 1.00 | |
3 | I needed to read some parts again carefully. | 3.75 | 0.94 | |
SM | 4 | I quickly summarized or took notes. | 2.42 | 1.20 |
5 | I translated some words/sentences of the passage. | 2.32 | 1.20 | |
6 | I tried to guess from other sentences. | 3.19 | 1.18 | |
7 | I used clues in the other questions to guess the answer. | 2.84 | 1.22 | |
TW | 8 | I simply chose the answer that seemed the least wrong. | 2.55 | 1.19 |
9 | I selected an option that had an important word. | 2.49 | 1.16 | |
10 | I guessed blindly. | 2.01 | 1.19 |
Appendix B
-
1.
Note that texts in italic face followed by “!” are the descriptions of the Mplus syntax.
-
2.
Use Mplus standardized outputs under the heading of “STD” to obtain both the \( \hat{\beta} \) and \( \hat{r} \) for computing the Pratt’s measures.
Title: Regression of the Latent Task Performance (Task-1, Letter) on the 3 the Observed Strategy Types
Data: File is Strategy for Mplus.dat;
Format is 455F5;
Variable:
NAMES ARE id L1–L11 LCM LSM LTW;
! L1-L11 are scores for reading questions of Letter (Task-1).
! LCM, LSM and LTW are the observed scores for CM, SM and TW strategies.
USEVAR ARE L1-L11 LCM LSM LTW;
CATEGORICAL ARE L1-L11;
Missing are all (99);
Model:
P1 by L1-L11; ! Measurement model for latent performance (P1) for Letter (Task-1)
P1 on LCM LSM LTW; ! Latent variable regression (P1) on three types of Strategy
! To obtain correlations for P1 with LCM LSM and LTW, replace “on” by “with” (no “on” command in the model)
Output: Standardized; ! To obtain R 2 , \( \hat{\beta} \) and \( \hat{r} \) for computing Pratt’s measures (Use “STD” standardized)
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Wu, A.D., Zumbo, B.D. (2017). Understanding Test-Taking Strategies for a Reading Comprehension Test via Latent Variable Regression with Pratt’s Importance Measures. In: Zumbo, B., Hubley, A. (eds) Understanding and Investigating Response Processes in Validation Research. Social Indicators Research Series, vol 69. Springer, Cham. https://doi.org/10.1007/978-3-319-56129-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-56129-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56128-8
Online ISBN: 978-3-319-56129-5
eBook Packages: Social SciencesSocial Sciences (R0)