Abstract
Large-scale assessments are often conducted using complex sampling designs that include the stratification of a target population and multi-stage cluster sampling. To address the nested structure of item response data under complex sample designs, a number of previous studies proposed the multilevel/multidimensional item response models. However, incorporating sample weights into the item response models has been relatively less explored. The purpose of this study is to assess the performance of four approaches to analyzing item response data that are collected under complex sample designs: (1) single-level modeling without weights (ignoring complex sample designs), (2) the design-based (aggregate) method, (3) the model-based (disaggregate) method, and (4) the hybrid method that addresses both the multilevel structure and the sampling weights. A Monte Carlo simulation study is carried out to see whether the hybrid method can yield the least biased item/person parameter and level-2 variance estimates. Item response data are generated using the complex sample design that is adopted by PISA 2000, and bias in estimates and adequacy of standard errors are evaluated. The results highlight the importance of using sample weights in item analysis when a complex sample design is used.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, R. J., Wilson, M., & Wu, M. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76.
Anderson, J. O., Milford, T., & Ross, S. P. (2009). Multilevel modeling with HLM: Taking a second look at PISA. In Quality research in literacy and science education (pp. 263–286). Dordrecht: Springer Netherlands.
Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics - Theory and Methods, 35(3), 439–460.
Asparouhov, T., & Muthén, B. (2006). Multilevel modelling of complex survey data. In Proceedings of the Survey Research Methods Section, American Statistical Association 2006 (pp. 2718–2726).
Binder, D.  A. (1983). On the variances of asymptotically normal estimators from complex surveys. International Statistical Review, 51, 279–292.
Cai, L. (2013). Multiple-group item response theory analysis in the case of complex survey data. Contributed Paper for World Statistics Congress Session Latent Variable Modeling of Complex Survey Data, August 2013, Hong Kong.
Fox, J., & Glas, C. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 269–286.
Heeringa, S.  G., West, B.  T., & Berglund, P. A. (2010). Applied survey data analysis. Boca Raton, FL: CRC Press.
Jiao, H., Kamata, A., Wang, S., & Jin, Y. (2012). A multilevel testlet model for dual local dependence. Journal of Educational Measurement, 49(1), 82–100.
Kamata, A. (2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79–93.
Kish, L. (1965). Survey sampling. New York, NY: Wiley.
Laukaityte, I. (2013). The importance of sampling weights in multilevel modeling of international large-scale assessment data. Paper presented at the 9th Multilevel conference, Utrecht, March 27–29.
Mislevy, R.  J., Beaton, A. E., Kaplan, B. K., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133–161.
Pfeffermann, D. (1993). The role of sampling weights when modelling survey data. International Statistical Review, 61, 317–337.
Pfeffermann, D., Skinner, C.  J., Holmes, D.  J., Goldstein, H., & Rasbash, J. (1998). Weighting for unequal selection probabilities in multilevel models. Journal of the Royal Statistical Society. Series B, Statistical methodology, 60, 23–40.
Rabe-Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society, Series A, 169(4), 805–827.
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592.
Rust, K. (1985). Variance estimation for complex estimators in sample surveys. Journal of Official Statistics, 1(4), 381–397.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(Suppl. 17).
Skinner, C.  J. (1989). Domain means, regression and multivariate analysis. In C. J. Skinner, D. Holt & T. M. F. Smith (Eds.), Analysis of complex surveys (pp. 59–87). New York, NY: Wiley.
Stapleton, L. M. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling, 9(4), 475–502.
Acknowledgements
This research is supported in part by the Institute for Education Sciences, U.S. Department of Education, through grants R305D150052. The opinions expressed are those of the authors and do not represent the views of the Institute or the Department of Education. We would like to thank Dr. Li Cai for providing his unpublished paper on unidimensional model with weights and flexMIRT®; program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Zheng, X., Yang, J.S. (2016). Using Sample Weights in Item Response Data Analysis under Complex Sample Designs . In: van der Ark, L., Bolt, D., Wang, WC., Douglas, J., Wiberg, M. (eds) Quantitative Psychology Research. Springer Proceedings in Mathematics & Statistics, vol 167. Springer, Cham. https://doi.org/10.1007/978-3-319-38759-8_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-38759-8_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-38757-4
Online ISBN: 978-3-319-38759-8
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)