Skip to main content
SpringerLink
Log in

Automatic Item Generation Unleashed: An Evaluation of a Large-Scale Deployment of Item Models

  • Conference paper
  • First Online:
Artificial Intelligence in Education (AIED 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10947))

Included in the following conference series:

Abstract

Automatic item generation represents a potential solution to the increased item development demands in this era of continuous testing. However, the use of test items that are automatically generated on-the-fly poses significant psychometric challenges for item calibration. The solution that has been suggested by a small but growing number of authors is to replace item calibration with item model (or family) calibration and to adopt a multilevel approach where items are nested within item models. Past research on the feasibility of this approach was limited to simulations or small-scale illustrations of its potential. The purpose of this study was to evaluate the results of a large-scale deployment of automatic item generation in a low-stakes adaptive testing context, with a large number of item models, and a very large number of randomly generated item instances.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Arendasy, M.E., Sommer, M.: Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learn. Individ. Differ. 22(1), 112–117 (2012)

    Article  Google Scholar 

  • Arieli-Attali, M., Cayton-Hodges, G.A.: Expanding the CBAL™ Competency Model for Mathematics Assessments and Developing a Rational Number Learning Progression. Educational Testing Service, Princeton (2014)

    Google Scholar 

  • Attali, Y., Arieli-Attali, M.: Gamification in assessment: do points affect test performance? Comput. Educ. 83, 57–63 (2015)

    Article  Google Scholar 

  • Attali, Y., Arieli-Attali, M.: Validating predictions from learning progressions: framework and implementation. In: The Annual Meeting of the American Educational Research Association (AERA), San Antonio, TX (2017)

    Google Scholar 

  • Attali, Y., Powers, D.: Immediate feedback and opportunity to revise answers to open-ended questions. Educ. Psychol. Meas. 70(1), 22–35 (2010)

    Article  Google Scholar 

  • Bartram, D., Hambleton, R.: Computer-Based Testing and the Internet: Issues and Advances. Wiley, New York (2005)

    Google Scholar 

  • Bates, D., Mächler, M., Bolker, B., Walker, S.: Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67(1), 1–48 (2015). https://doi.org/10.18637/jss.v067.i01

  • Bejar, I.I.; Generative testing: from conception to implementation. In: Irvine, S.H., Kyllonen, P.C. (eds.) Item Generation for Test Development, pp. 199–217. Erlbaum, Mahwah (2002)

    Google Scholar 

  • Bennett, R.E.: CBAL: results from piloting innovative k–12 assessments. ETS Res. Rep. Ser. 2011(1) (2011)

    Google Scholar 

  • Cho, S.-J., De Boeck, P., Embretson, S., Rabe-Hesketh, S.: Additive multilevel item structure models with random residuals: item modeling for explanation and item generation. Psychometrika 79(1), 84–104 (2014)

    Article  MathSciNet  Google Scholar 

  • Downing, S.M., Haladyna, T.M.: Handbook of test development. Lawrence Erlbaum, Mahwah (2006)

    Google Scholar 

  • Embretson, S.: Generating items during testing: psychometric issues and models. Psychometrika 64(4), 407–433 (1999)

    Article  Google Scholar 

  • Embretson, S., Yang, X.: Automatic item generation and cognitive psychology. Handb. Stat. 26, 747–768 (2006)

    Article  Google Scholar 

  • Geerlings, H., Glas, C.A., van der Linden, W.J.: Modeling rule-based item generation. Psychometrika 76(2), 337–359 (2011)

    Article  MathSciNet  Google Scholar 

  • Geerlings, H., van der Linden, W.J., Glas, C.A.: Optimal test design with rule-based item generation. Appl. Psychol. Meas. 37(2), 140–161 (2013)

    Article  Google Scholar 

  • Gierl, M.J., Haladyna, T.M.: Automatic Item Generation: Theory and Practice. Routledge, New York (2013)

    Google Scholar 

  • Glas, C.A., van der Linden, W.J.: Computerized adaptive testing with item cloning. Appl. Psychol. Meas. 27(4), 247–261 (2003)

    Article  MathSciNet  Google Scholar 

  • Graf, A., Fife, J.H.: Difficulty modeling and automatic generation of quantitative items. In: Gierl, M.J., Haladyna, T.M. (eds.) Automatic Item Generation: Theory and Practice, pp. 157–179. Routledge, New York (2013)

    Google Scholar 

  • Haladyna, T.M.: Automatic item generation: a historical perspective. In: Gierl, M.J., Haladyna, T.M. (eds.) Automatic Item Generation: Theory and Practice, pp. 13–25. Routledge, New York (2013)

    Google Scholar 

  • Irvine, S.H., Kyllonen, P.C.: Item Generation for Test Development. Erlbaum, Mahwah (2002)

    Google Scholar 

  • Janssen, R., Schepers, J., Peres, D.: Models with item and item group predictors. In: De Boeck, P., Wilson, M. (eds.) Explanatory Item Response Models, pp. 189–212. Springer, New York (2004). https://doi.org/10.1007/978-1-4757-3990-9_6

    Chapter  Google Scholar 

  • Janssen, R., Tuerlinckx, F., Meulders, M., De Boeck, P.: A hierarchical IRT model for criterion-referenced measurement. J. Educ. Behav. Stat. 25(3), 285–306 (2000)

    Article  Google Scholar 

  • Pintrich, P.R., Schunk, D.H.: Motivation in Education: Theory, Research, and Applications, 2nd edn. Prentice Hall, Upper Saddle River (2002)

    Google Scholar 

  • Shermis, M.D., Burstein, J.: Handbook of Automated Essay Evaluation: Current Applications and New Directions. Routledge, New York (2013)

    Google Scholar 

  • Sinharay, S., Johnson, M.S.: Use of item models in a large-scale admissions test: a case study. Int. J. Test. 8(3), 209–236 (2008)

    Article  Google Scholar 

  • Sinharay, S., Johnson, M.S.: Statistical modeling of automatically generated items. In: Gierl, M.J., Haladyna, T.M. (eds.) Automatic Item Generation: Theory and Practice, pp. 183–195. Routledge, New York (2013)

    Google Scholar 

  • Sireci, S.G., Zenisky, A.L.: Innovative item formats in computer-based testing: in pursuit of improved construct representation. In: Downing, S.M., Haladyna, T.M. (eds.) Handbook of Test Development, pp. 329–347. Lawrence Erlbaum, Mahwah (2006)

    Google Scholar 

  • van der Linden, W.J., Glas, C.A.: Elements of Adaptive Testing. Springer, New York (2010). https://doi.org/10.1007/978-0-387-85461-8

    Book  MATH  Google Scholar 

  • Whitely, S.E.: Construct validity: construct representation versus nomothetic span. Psychol. Bull. 93(1), 179 (1983)

    Article  Google Scholar 

  • Wright, B.D., Linacre, J.M.: Reasonable mean-square fit values. Rasch Meas. Trans. 8(3), 370 (1994)

    Google Scholar 

  • Wright, B.D., Masters, G.N.: Rating Scale Analysis: Rasch Measurement. Mesa Press, Chicago (1982)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Educational Testing Service, Rosedale Road, MS-16-R, Princeton, NJ, 08541, USA

    Yigal Attali

Authors
  1. Yigal Attali
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yigal Attali .

Editor information

Editors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, USA

    Carolyn Penstein Rosé

  2. University of Technology, Sydney, NSW, Australia

    Roberto Martínez-Maldonado

  3. University of Duisburg-Essen, Duisburg, Germany

    H. Ulrich Hoppe

  4. UCL Institute of Education, London, UK

    Rose Luckin

  5. UCL Institute of Education, London, UK

    Manolis Mavrikis

  6. UCL Institute of Education, London, UK

    Kaska Porayska-Pomsta

  7. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce McLaren

  8. University of Sussex, Brighton, UK

    Benedict du Boulay

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Attali, Y. (2018). Automatic Item Generation Unleashed: An Evaluation of a Large-Scale Deployment of Item Models. In: Penstein Rosé, C., et al. Artificial Intelligence in Education. AIED 2018. Lecture Notes in Computer Science(), vol 10947. Springer, Cham. https://doi.org/10.1007/978-3-319-93843-1_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-93843-1_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-93842-4

  • Online ISBN: 978-3-319-93843-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation