Advertisement

Issues in Innovative Item Types

  • Cynthia G. Parshall
  • Judith A. Spray
  • John C. Kalohn
  • Tim Davey
Chapter
Part of the Statistics for Social and Behavioral Sciences book series (SSBS)

Abstract

The development of innovative item types, defined as items that depart from the traditional, discrete, text-based, multiple-choice format, is perhaps the most promising area in the entire field of computer-based testing. The reason for this is the great potential that item innovations have for substantively improving measurement. Innovative item types are items that include some feature or function made available due to their administration on computer. Items may be innovative in many ways. This chapter will include a discussion of the purpose or value of innovative item types, the five dimensions in which items may be innovative, the impact of level of complexity on the development and implementation of these item types, and a view toward the future of innovative item types.

Keywords

Response Item National Council Item Type Item Format Item Innovation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. ACT, Inc. (1998). Assessing listening comprehension: A Review of Recent Literature Relevant to an LSAT Listening Component. Unpublished manuscript, LSAC, Newton, PA.Google Scholar
  2. ACT, Inc. (1999). Technical Manual for the ESL Exam. Iowa City: Author.Google Scholar
  3. Baker, E. L., & O’Neil, H. F. Jr. (1995). Computer technology futures for the improvement of assessment. Journal of Science Education and Technology, 4, 37–45.CrossRefGoogle Scholar
  4. Balizet, S., Treder, D. W., & Parshall, C. G. (1999, April). The development of an audio computer-based classroom test of ESL listening skills. Paper presented at the annual meeting of the American Educational Research Association, Montreal.Google Scholar
  5. Bennett, R. E., & Bejar, I. I. (1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues & Practices, 17, 9–17.CrossRefGoogle Scholar
  6. Bennett, R. E., Goodman, M., Hessinger, J., Ligget, J., Marshall, G., Kahn, H., & Zack, J. (1997). Using Multimedia in Large-Scale Computer-Based Testing Programs (Research Rep. No. RR-97-3). Princeton, NJ: Educational Testing Service.Google Scholar
  7. Bennett, R. E., Morley, M., & Quardt, D. (1998, April). Three Response Types for Broadening the Conception of Mathematical Problem Solving in Computerized-Adaptive Tests. Paper presented at the annual meeting of the National Council of Measurement in Education, San Diego.Google Scholar
  8. Braun, H. (1994). Assessing technology in assessment. In Baker, E. A., & O’Neil, H. F. (eds.), Technology Assessment in Education and Training (pp. 231–246). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  9. Breland, H. M. (1998, April). Writing Assessment Through Automated Editing. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.Google Scholar
  10. Burstein, J., Kukich, K., Wolff, S., Lu, C, & Chodorow, M. (1998, April). Computer Analysis of Essays. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.Google Scholar
  11. Clauser, B. E., Margolis, M. J., Clyman, S. G., & Ross, L P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement, 34, 141–161.CrossRefGoogle Scholar
  12. Davey, T., Godwin, J., & Mittelholtz, D. (1997). Developing and scoring an innovative computerized writing assessment. Journal of Educational Measurement, 34, 21–41.CrossRefGoogle Scholar
  13. Drasgow, F., Olson-Buchanan, J. B., & Moberg, P. J. (1999). Development of an interactive video assessment: Trials and tribulations. In Drasgow, F., & Olson-Buchanan, J. B, (eds.), Innovations in Computerized Assessment, (pp. 177–196). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  14. Educational Testing Service (ETS). (1998). Computer-Based TOEFL Score User Guide. Princeton, NJ: Author.Google Scholar
  15. French, A., & Godwin, J. (1996, April). Using Multimedia Technology to Create Innovative Items. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.Google Scholar
  16. Harmes, J. C, & Parshall, C. G. (2000, November). An Iterative Process for Computerized Test Development: Integrating Usability Methods. Paper presented at the annual meeting of the Florida Educational Research Association, Tallahassee.Google Scholar
  17. Landauer, T. K., Laham, D., Rehder, B., & Schreiner, M. E. (1997). How well can passage meaning be derived without using word order? A comparison of latent semantic analysis and humans. In Shafto, G., & Langley, P. (eds.), Proceedings of the 19th Annual Meeting of the Cognitive Science Society (pp. 412–417). Mahwah, NJ: Erlbaum.Google Scholar
  18. Olson-Buchanan, J. B., Drasgow, F., Moberg, P. J., Mead, A. D., Keenan, P.A., & Donovan, M.A. (1998). Interactive video assessment of conflict resolution skills. Personnel Psychology, 51, 1–24.CrossRefGoogle Scholar
  19. Page, E. B. & Petersen N. S. (1995, March). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappen, 76, 561–565.Google Scholar
  20. Parshall, C. G. (1999, February). Audio CBTs: Measuring More through the use of Speech and Nonspeech Sound. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.Google Scholar
  21. Parshall, C. G., Davey, T., & Pashley, P. J. (2000). Innovative item types for computerized testing. In Van der Linden, W. J., & Glas, C. A. W. (eds.), Computerized Adaptive Testing: Theory and Practice, (pp. 129–148). Norwell, MA: Kluwer Academic Publisher.Google Scholar
  22. Parshall, C. G., Stewart, R, & Ritter, J. (1996, April). Innovations: Sound, Graphics, and Alternative Response Modes. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.Google Scholar
  23. Shavelson, R. J., Baxter, G. P., & Pine, J. (1992). Performance assessments: Political rhetoric and measurement reality. Educational Researcher, 21, 22–27.Google Scholar
  24. Vispoel, W. P., Wang, T., & Bleiler, T. (1997). Computerized adaptive and fixed-item testing of music listening skill: A comparison of efficiency, precision, and concurrent validity. Journal of Educational Measurement, 34, 43–63.CrossRefGoogle Scholar

Additional Readings

  1. ACT, Inc. (1995). Work Keys. Iowa City: Author.Google Scholar
  2. Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522–532.CrossRefGoogle Scholar
  3. Bennett, R.E. (1998). Reinventing Assessment. Princeton, NJ: Educational Testing Service.Google Scholar
  4. Bennett, R. E., & Sebrechts, M. M. (1997). A computer-based task for measuring the representational component of quantitative proficiency. Journal of Educational Measurement, 34, 64–77.CrossRefGoogle Scholar
  5. Bennett, R. E., Steffen, M., Singley, M. K., Morley, M., & Jacquemin, D. (1997). Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computer-adaptive tests. Journal of Educational Measurement, 34, 162–176.CrossRefGoogle Scholar
  6. Booth, J. (1991). The key to valid computer-based testing: The user interface. Revue Européenne de Psychologie Appliquée, 41, 281–293.Google Scholar
  7. Bosman, F., Hoogenboom, J., & Walpot, G. (1994). An interactive video test for pharmaceutical chemist’s assistants. Computers in Human Behavior, 10, 51–62.CrossRefGoogle Scholar
  8. Braun, H. I., Bennett, R. E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systems. Journal of Educational Measurement, 27, 93–108.CrossRefGoogle Scholar
  9. Breland, H.M. (1998, April). Writing Assessment through Automated Editing. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego.Google Scholar
  10. Bugbee, A. C. Jr., & Bernt, F. M. (1990). Testing by computer: Findings in six years of use. Journal of Research on Computing in Education, 23, 87–100.Google Scholar
  11. Buxton, W. (1987). There’s more to interaction than meets the eye: Some issues in manual input. In Baecker, R. M. & Buxton, W. A. S. (eds.) Readings in Human-Computer Interaction: A Multidisciplinary Approach (pp. 366–375). San Mateo, CA: Morgan Kaufmann.Google Scholar
  12. Clauser, B. E., Ross, L. P., Clyman, S. G., Rose, K. M., Margolis, M. J., Nungester, R. N., Piemme, T. E., Chang, L., El-Bayoumi, G., Malakoff, G. L., & Pincetl, P. S. (1997). Development of a scoring algorithm to replace expert rating for scoring a complex performance-based assessment. Applied Measurement in Education, 10, 345–358.CrossRefGoogle Scholar
  13. Dodd, B. G., & Fitzpatrick, S.J. (1998). Alternatives for scoring computer-based tests. Paper presented at the ETS Colloquium, Computer-Based Testing: Building The Foundation For Future Assessments, Philadelphia.Google Scholar
  14. Fitch, W. T., & Kramer, G. (1994). Sonifying the body electric: Superiority of an auditory over a visual display in a complex, multivariate system. In Kramer, G. (ed.), Auditory Display, (pp. 307–325). Reading, MA: Addison-Wesley.Google Scholar
  15. Gaver, W. W. (1989). The SonicFinder: An interface that uses auditory icons. Human-Computer Interaction, 4, 67–94.CrossRefGoogle Scholar
  16. Godwin, J. (1999, April). Designing the ACT ESL listening test. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.Google Scholar
  17. Gruber, J. S. (1998, October). [Interview with James Kramer, head of Virtual Technologies, Inc.] Gropethink. Wired, pp. 168–169.Google Scholar
  18. Koch, D. A. (1993). Testing goes graphical. Journal of Interactive Instruction Development, 5, 14–21.Google Scholar
  19. Luecht, R. M., & Clauser, B. E. (1998, September). Test methods for complex computer-based testing. Paper presented at the ETS Colloquium, Computer-Based Testing: Building The Foundation For Future Assessments, Philadelphia.Google Scholar
  20. Martinez, M. E. (1991). A comparison of multiple-choice and constructed figurai response items. Journal of Educational Measurement, 28, 131–145.CrossRefGoogle Scholar
  21. Martinez, M. E. (1993). Item formats and mental abilities in biology assessment. Journal of Computers in Mathematics and Science Teaching, 12, 289–301.Google Scholar
  22. Martinez, M. E. & Bennett, R. E. (1992). A review of automatically scorable constructed-response item types for large-scale assessment. Applied Measurement in Education, 5, 151–169.CrossRefGoogle Scholar
  23. Nissan, S. (1999, April). Incorporating Sound, Visuals, and Text for TOEFL on Computer. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.Google Scholar
  24. O’Neill, K., & Folk, V. (1996, April). Innovative CBT Item Formats in a Teacher Licensing Program. Paper presented at the annual meeting of the National Council on Measurement in Education, New York, NY.Google Scholar
  25. Perlman, M., Berger, K., & Tyler, L. (1993). An Application of Multimedia Software to Standardized Testing in Music. (Research Rep. No. 93-36) Princeton, NJ: Educational Testing Service.Google Scholar
  26. Shea, J. A., Norcini, J. J., Baranowski, R. A., Langdon, L. O., & Popp, R. L. (1992). A comparison of video and print formats in the assessment of skill in interpreting cardiovascular motion studies. Evaluation and the Health Professions, 15, 325–340.CrossRefGoogle Scholar
  27. Stone, B. (1998, March). Focus on technology: Are you talking to me? Newsweek, pp. 85–86.Google Scholar
  28. Taggart, W. R. (1995). Certifying pilots: Implications for medicine and for the future. In. Mancall, E. L. & Bashook, P. G. (eds.), Assessing Clinical Reasoning: The Oral Examination and Alternative Methods (pp. 175–182). Evanston, IL: American Board of Medical Specialties.Google Scholar
  29. Vicino, F. L., & Moreno, K. E. (1997). Human factors in the CAT system: A pilot study. In Sands, W. A., Waters, B. K, & McBride, J. R. (eds.), Computerized Adaptive Testing: From Inquiry To Operation (pp. 157–160). Washington, DC: APA.CrossRefGoogle Scholar
  30. Vispoel, W. P., & Coffman, D. (1992). Computerized adaptive testing of music-related skills. Bulletin of the Council for Research in Music Education, 112, 29–49.Google Scholar
  31. Vispoel, W. P., Wang, T., & Bleiler, T. (1997). Computerized adaptive and fixed-item testing of music listening skill: A comparison of efficiency, precision, and concurrent validity. Journal of Educational Measurement, 34, 43–63.CrossRefGoogle Scholar
  32. Williams, V. S. L., Sweeny, S. F., & Bethke, A. D. (1999). The Development and Cognitive Laboratory Evaluation of an Audio-Assisted Computer-Adaptive Test for Eighth-Grade Mathematics. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal.Google Scholar

Copyright information

© Springer Science+Business Media New York 2002

Authors and Affiliations

  • Cynthia G. Parshall
    • 1
  • Judith A. Spray
    • 2
  • John C. Kalohn
    • 2
  • Tim Davey
    • 3
  1. 1.University of South FloridaTampaUSA
  2. 2.ACT, Inc.Iowa CityUSA
  3. 3.Educational Testing ServicePrincetonUSA

Personalised recommendations

Cite chapter

Buy options