Evaluating Subject Matter Experts’ Learning and Use of an ITS Authoring Tool

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 1452)


Because ITSs offer knowledge-based instruction, ITS authoring shells can use knowledge acquisition systems to bring courseware development within the reach of subject-matter experts unskilled in instructional design and development. The Experimental Advanced Instructional Design Advisor (XAIDA), a simple ITS shell, was evaluated in a study of 17 Air Force medical-technician trainers. The study examined XAIDA’s learnability, usability, and acceptability to the participants using a one-week combined training and evaluation program. All participants were able to produce courseware using XAIDA. Measures of cognitive structure indicated that their conception of XAIDA’s features became more similar to that of an expert as the result of training. Performance tests, skill ratings and usability ratings indicated that XAIDA is both usable and learnable. Participant’s attitudes were generally positive except at a point when training required a difficult cognitive shift. Attitudes remained focused througout training on personal and low-level concerns, indicating the participants were not likely to be early adapters of the technology. The results have implications for the deployment of ITS shells in operational environments.


Instructional Design Intelligent Tutoring System Authoring Tool Instructional Delivery Educational Technology Publication 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bailey, D. B., & Palsha, S. A. (1992). Qualities of stages of concern questionnaire and implications for educational innovations. The Journal of Educational Research, 85, 226–232.CrossRefGoogle Scholar
  2. 2.
    Bloom, C., Linton, F., & Bell, B. (1997). Using evaluation in the design of an intelligent tutoring system. Journal of Interactive Learning Research, 8, 235–276.Google Scholar
  3. 3.
    Cannon-Bowers, J. A., Tannenbaum, S. I., Salas, E., & Converse, S. A. (1991). Toward an integration of training theory and technique. Human Factors, 33, 281–292.Google Scholar
  4. 4.
    Carroll, J. M. (1984). Minimalist Training. Datamation, November, 125–136.Google Scholar
  5. 5.
    Cooke, N. J. (1994). Varieties of knowledge elicitation techniques. International Journal of Human-Computer Studies, 41, 801–849.zbMATHCrossRefGoogle Scholar
  6. 6.
    Geoghegan, W. H. (1994). Whatever happened to instructional technology? Paper presented at the 22nd Annual Conference of the International Business Schools Computing Association. Baltimore, Maryland.Google Scholar
  7. 7.
    Glaser, R. (1983). Expertise and learning: How do we think about instructional processes now that we have discovered knowledge structures? In E. Klaher and K. Kotovsky (Eds.) Complex information processing: The impact of Herb Simon. Lawrence Erlbaum: Hillsdale, NJ.Google Scholar
  8. 8.
    Hall, G. E. (1979). The concerns-based approach to facilitating change. Educational Horizons, 4, 202–208.Google Scholar
  9. 9.
    Hall, G. E., George, A. A., & Rutherford, W. L. (1986). Measuring stages of concern about the innovation: A manual for the user of SoC Questionnaire. Austin, TX: Southwest Educational Development Laboratory.Google Scholar
  10. 10.
    Hickey, A. E., Spector, J. M., & Muraida, D. J. (1991). Design specifications for the advanced instructional design advisor (AIDA) (Vols 1 & 2) (AL-TR-1991-0085). Brooks AFB, TX: Technical Report for the Air Force Armstrong Laboratory, Human Resources Directorate.Google Scholar
  11. 11.
    Hsieh, P. Y, Halff, H. M., & Redfield, C. L. (1996). Four easy pieces: Development systems for knowledge-based generative instruction. (Manuscript submitted for publication).Google Scholar
  12. 12.
    Kraiger, J., Ford, K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.CrossRefGoogle Scholar
  13. 13.
    Kraiger, J., Salas, E. & Cannon-Bowers, J. (1995). Measuring knowledge organization as a method of assessing learning during training. Human Factors, 37, 804–816.CrossRefGoogle Scholar
  14. 14.
    Merrill, M. D. (1993). An integrated model for automating instructional design and delivery. In J. M. Spector, M. C. Polson, and D. J. Muraida (Eds.) Automating Instructional Design: Concepts and Issues (pp. 147–190). Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
  15. 15.
    Palmiter, S., & Elkerton, J. (1993). Animated Demonstrations for learning procedural computer-based tasks. Human-Computer Interaction, 8, 193–216.CrossRefGoogle Scholar
  16. 16.
    Redfield, C. (November, 1997) AAAI Fall Symposium on ITS Authoring Tools (Technical Report FS-97-01). Cambridge, MA, AAAI.Google Scholar
  17. 17.
    Rowe, A., Cooke, N., Hall, & Halgren, (1996). Toward an on-line knowledge assessment methodology: Building on the relationship between knowing and doing. Journal of Experimental Psychology: Applied, 2, 31–47.CrossRefGoogle Scholar
  18. 18.
    Royer, J. M., Cisero, C. A., & Carlo, M. S. (1993). Techniques and procedures for assessing cognitive skills. Review of Educational Research, 63, 201–243.CrossRefGoogle Scholar
  19. 19.
    Schvaneveldt, R. W. (Ed.). (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex.Google Scholar
  20. 20.
    Spector, J. M. (1990). Designing and developing an advanced instructional design advisor. (AFHRL-TP-90-52). Brooks AFB: TX: Technical Paper for the Training Systems Division of the Air Force Human Resources Laboratory.Google Scholar
  21. 21.
    Spector, J. M., Polson, M. C., & Muraida, D. J., Eds. (1993). Automating Instructional Design: Concepts and Issues. Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
  22. 22.
    Wenzel, B. M., Dirnberger, M. T., & Ellis, R. (1997). Evaluation of the effectiveness of distributed interactive multimedia courseware for emergency medical technician training. Manuscript submitted for publication.Google Scholar
  23. 23.
    Wenzel, B. M., Keeney, C., Licklider, J., Roberts, J. D., & Reyna, H. (1997). Evaluation of the courseware developed and delivered using the Experimental Advanced Instructional Design Advisor. Presentation at the annual meeting of the Texas Association of College Technical Editors. Austin, Texas.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1998

Authors and Affiliations

  1. 1.Mei Technology CorporationSan AntonioUSA

Personalised recommendations