Skip to main content
SpringerLink
Log in

Learning Traces, Measurement and Assessment Templates for AIS Interoperability

  • Conference paper
  • First Online:
Adaptive Instructional Systems (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12214))

Included in the following conference series:

Abstract

The current paper contains elements relevant for the conceptual modelling and interoperability of measurements and assessments across adaptive instructional systems (AIS). After an introduction, the first section presents a generic use case, where knowledge acquisition is supported by a sequence of training simulators of increasing complexity, and the need to capture regularity and variations in measurements and assessments across instructional systems. The second section briefly discusses the role of measurements and assessments as a core of functions of adaptive instructional systems. The section indicates that an adaptive instructional system needs minimally to capture references to learners’ performance, knowledge components, learning tasks, and learning attempts. The third section maps the main Generalized Intelligent Framework for Tutoring (GIFT) components to a generic feedback control system model. The mapping makes explicit the dual interpretation of measurements and assessments as both results (learning traces) and functions (computation templates). The fourth section examines some non-proprietary frameworks in terms of their capability to support the interoperability of measurements and assessments across adaptive instructional systems. The section briefly discusses xAPI, the Competency And Skill System (CASS), the Evidence Trace File (ETF), the Training Objective Package (TOP), and the Human Performance Markup Language (HPML). The main contributions of the paper are: 1) an AIS conceptual model based on a feedback control system model, and 2) a brief review of learning data frameworks as they relate to measurements and assessments as both learning traces and computation templates.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Santos, G.S., Jorge, J.: Interoperable intelligent tutoring systems as open educational resources. IEEE Trans. Learn. Technol. 6, 271–282 (2013)

    Article  Google Scholar 

  2. Sottilare, R.: Exploring methods to promote interoperability in adaptive instructional systems. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 227–238. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_19

    Chapter  Google Scholar 

  3. Thai, K.P., Tong, R.: Interoperability standards for adaptive instructional systems: vertical and horizontal integrations. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 251–260. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_21

    Chapter  Google Scholar 

  4. Adaptive Instructional Systems (AIS) Working Group: Adaptive Instructional Systems (C/LT/AIS) P2247.1. https://site.ieee.org/sagroups-2247-1/. Accessed 26 Nov 2019

  5. Robson, R., Barr, A.: The new wave of training technology standards. In: Proceeding of the Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), pp. 1–11. National Training and Simulation Association, Orlando (2018)

    Google Scholar 

  6. Atkinson, B.F.W., Killilea, J.: A review of the potential return on investment benefits of a human performance measurement standard: lessons learned in the Navy aviation community. In: 2015 Fall Simulation Interoperability Workshop, SIW 2015. SISO - Simulation Interoperability Standards Organization, Orlando (2015)

    Google Scholar 

  7. Brawner, K.W., Goodwin, G., Regan, D.: Scaling Across Domains and the Implications for GIFT. U.S. Army Research Laboratory, Orlando (2016)

    Google Scholar 

  8. Emond, B., Kondratova, I., Durand, G., Valdés, J.J.: A multi-role reconfigurable trainer for naval combat information operators. In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), p. 14. National Training and Simulation Association, Orlando (2018)

    Google Scholar 

  9. Emond, B., et al.: Adaptive training simulation using speech interaction for training navy officers. In: Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), pp. 2924–2934. National Training and Simulation Association, Orlando (2016)

    Google Scholar 

  10. Emond, B.: Interoperable assessments using HPML: a novice conning skills acquisition use case. In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), p. 11. National Training and Simulation Association, Orlando (2017)

    Google Scholar 

  11. Alessi, S.M.: Fidelity in the design of instructional simulations. J. Comput. Instr. (1988)

    Google Scholar 

  12. Martin, B., Mitrovic, A., Koedinger, K.R., Mathan, S.: Evaluating and improving adaptive educational systems with learning curves. User Model. User-Adapt. Interact. 21, 249–283 (2011). https://doi.org/10.1007/s11257-010-9084-2

    Article  Google Scholar 

  13. Corbett, A.T., Anderson, J.R.: Knowledge tracing: modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4, 253–278 (1995). https://doi.org/10.1007/BF01099821

    Article  Google Scholar 

  14. National Research Council: Knowing what students know: The science and design of educational assessment. National Academy Press, Washington, DC (2001)

    Google Scholar 

  15. Mislevy, R.J., Steinberg, L.S., Almond, R.G.: On the structure of educational assessments. Meas. Interdisc. Res. Perspect. 1, 3–67 (2003)

    Article  Google Scholar 

  16. Stacy, W., Merket, D., Freeman, J., Wiese, E., Jackson, C.: A language for rapidly creating performance measures in simulators. In: The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC) (2005)

    Google Scholar 

  17. Stacy, W., Ayers, J., Freeman, J., Haimson, C.: Representing human performance with human performance measurement language. In: Proceedings of the Fall 2006 Simulation Interoperability Workshop, pp. 570–580. SISO - Simulation Interoperability Standards Organization (2006)

    Google Scholar 

  18. Gagné, R.M.: The Conditions of Learning. Holt, Rinehart and Winston, NewYork (1965)

    Google Scholar 

  19. Bloom, B.S.: Taxonomy of Educational Objectives, Handbook: The Cognitive Domain. David McKay, New York (1956)

    Google Scholar 

  20. Krathwohl, D.R., Bloom, B.S., Masia, B.B.: Taxonomy of Educational Objectives: The Classification of Educational Goals, Handbook II: Affective Domain. David McKay, New York (1964)

    Google Scholar 

  21. Simpson, E.J.: The Classification of Educational Objectives in the Psychomotor Domain. Gryphon House, Washington DC (1972)

    Google Scholar 

  22. Burke, C.S., Sottilare, R., Gilbert, S.: Leveraging team taxonomic efforts for authoring. In: Sottilare, R.A., Graesser, A.C., Hu, X., Sinatra, A.M. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 6 - Team Tutoring, pp. 193–199. US Army Research Laboratory, Orlando (2018)

    Google Scholar 

  23. Hjelmervik, K., Nazir, S., Myhrvold, A.: Simulator training for maritime complex tasks: an experimental study. WMU J. Marit. Aff. 17(1), 17–30 (2018). https://doi.org/10.1007/s13437-017-0133-0

    Article  Google Scholar 

  24. Lathan, C.L., Tracey, M.R., Sebrechts, M.M., Clawson, D.M., Higgins, G.A.: Using virtual environments as training simulators: Measuring transfer. In: Stanney, K.M. (ed.) Handbook of Virtual Environments: Design, Implementation, and Applications, pp. 403–414. Lawrence Erlbaum, Mahwah (2002)

    Google Scholar 

  25. Brydges, R., Carnahan, H., Rose, D., Rose, L., Dubrowski, A.: Coordinating progressive levels of simulation fidelity to maximize educational benefit. Acad. Med. 85, 806–812 (2010). https://doi.org/10.1097/ACM.0b013e3181d7aabd

    Article  Google Scholar 

  26. Beaubien, J.M., Baker, D.P.: The use of simulation for training teamwork skills in health care: how low can you go? Qual. Saf. Heal. Care 13, i51–i56 (2004). https://doi.org/10.1136/qshc.2004.009845

    Article  Google Scholar 

  27. IEEE P2247 Working Group: Adaptive Instructional Systems (C/LT/AIS) P2247 Working Group. https://site.ieee.org/sagroups-2247-1/. Accessed 11 Dec 2019

  28. Koedinger, K.R., Corbett, A.T., Perfetti, C.: The knowledge-learning-instruction framework: bridging the science-practice chasm to enhance robust student learning. Cogn. Sci. 36, 757–798 (2012). https://doi.org/10.1111/j.1551-6709.2012.01245.x

    Article  Google Scholar 

  29. Hao, J., Mislevy, R.J.: The Evidence Trace File: A Data Structure for Virtual Performance Assessments Informed by Data Analytics and Evidence-Centered Design. Wiley, Hoboken (2018). https://doi.org/10.1002/ets2.12215

  30. Tatsuoka, K.K.: Rule space: an approach for dealing with misconceptions based on item response theory. J. Educ. Meas. 20, 345–354 (1983)

    Article  Google Scholar 

  31. Aleven, V., McLaughlin, E.A., Glenn, R.A., Koedinger, K.R.: Instruction based on adaptive learning technologies. In: Mayer, R.E., Alexander, P.A. (eds.) Handbook of Research on Learning and Instruction, pp. 538–576. Routledge (2016). https://doi.org/10.4324/9781315736419

  32. Goldin, I., Pavlik, P.I.J., Ritter, S.: Discovering domain models in learning curve data. In: Sottilare, R.A., Graesser, A.C., Hu, X., Olney, A.M., Nye, B.D., Sinatra, A.M. (eds.) Design Recommendations for Intelligent Tutoring Systems: Volume 4 Domain Modeling, pp. 115–126. US Army Research Laboratory, Orlando (2016)

    Google Scholar 

  33. Vanlehn, K.: The behavior of tutoring systems. Int. J. Artif. Intell. Ed. 16, 227–265 (2006)

    Google Scholar 

  34. Sottilare, R.A.: Developing standards for adaptive instructional systems: 2018 update. In: Proceedings of the 6th Annual GIFT Users Symposium, pp. 1–5. U.S. Army Research Laboratory, Orlando (2018)

    Google Scholar 

  35. Pavlik, P.I., Brawner, K.W., Olney, A., Mitrovic, A.: A review of learner models used in intelligent tutoring systems. In: Sottilare, R.A., Graesser, A., Hu, X., Holden, H. (eds.) Design Recommendations for Intelligent Tutoring Systems. Volume 1: Learner Modeling, pp. 39–68. US Army Research Laboratory (2013)

    Google Scholar 

  36. Sottilare, R.A., Brawner, K.W., Goldberg, B.S., Holden, H.K.: The Generalized Intelligent Framework for Tutoring (GIFT) (2012)

    Google Scholar 

  37. Abdelzaher, T., Diao, Y., Hellerstein, J.L., Lu, C., Zhu, X.: Introduction to control theory and its application to computing systems. Perform. Model. Eng. 185–215 (2008). https://doi.org/10.1007/978-0-387-79361-0_7

  38. Hu, X., et al.: Capturing AIS behavior using xAPI-like statements. In: Sottilare, R.A., Schwarz, J. (eds.) HCII 2019. LNCS, vol. 11597, pp. 204–216. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22341-0_17

    Chapter  Google Scholar 

  39. Rustici Software: What is xAPI (the Experience API) (2019)

    Google Scholar 

  40. Poeppelman, T.R., Ayers, J., Hruska, M., Long, R., Amburn, C., Bink, M.: Interoperable Performance Assessment Using the Experience API. Interservice/Industry Train (2013)

    Google Scholar 

  41. Hruska, M., Long, R., Amburn, C.: Human performance interoperability via xAPI: current military outreach efforts. In: Fall Simulation Interoperability Workshop 2014 Fall SIW, pp. 207–216 (2014)

    Google Scholar 

  42. Goodwin, G.A., Murphy, J.S., Medford, A.L.: Support for training effectiveness assessment and data interoperability. In: Interservice/Industry Training, Simulation, Education Conference, pp. 1–11 (2016)

    Google Scholar 

  43. Advanced Distributed Learning, ADLnet.gov: xAPI. https://www.adlnet.gov/xAPI. Accessed 19 May 2017

  44. CASS Project: CASS Documentation. http://docs.cassproject.org/index.html. Accessed 09 Aug 2017

  45. Stacy, W., Freeman, J.: Training objective packages: enhancing the effectiveness of experiential training. Theor. Issues Ergon. Sci. 17, 149–168 (2016). https://doi.org/10.1080/1463922X.2015.1111459

    Article  Google Scholar 

  46. Simulation Interoperability Standards Organization: Human Performance Markup Language (HPML) Development Group (PDG). https://www.sisostds.org/StandardsActivities/DevelopmentGroups/HPMLPDG-HumanPerformanceMarkupLanguage.aspx

  47. Advanced Distributed Learning: GitHub - adlnet/xAPI-Spec. https://github.com/adlnet/xAPI-Spec. Accessed 09 June 2017

  48. IMS Global Learning Consortium: Caliper Analytics. https://www.imsglobal.org/activity/caliper. Accessed 15 Sept 2019

  49. Walker, A., Tolland, M., Stacy, W.: Using a human performance markup language for simulator-based training. In: Simulation Interoperability Standards Organization (SISO) Simulation Interoperability Workshop (SIW) Meeting. SISO - Simulation Interoperability Standards Organization, Orlando (2015)

    Google Scholar 

  50. Simulation Interoperability Standards Organization: Product Nomination for Human Performance Markup Language (SISO-PN-015-2016) (2016). https://www.sisostds.org/DigitalLibrary.aspx?Command=Core_Download&EntryId=43990

Download references

Author information

Authors and Affiliations

  1. National Research Council Canada, Ottawa, ON, K1A 0R6, Canada

    Bruno Emond

Authors
  1. Bruno Emond
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruno Emond .

Editor information

Editors and Affiliations

  1. Soar Technology, Inc., Orlando, FL, USA

    Robert A. Sottilare

  2. Fraunhofer FKIE, Wachtberg, Germany

    Jessica Schwarz

Rights and permissions

Reprints and permissions

Copyright information

© 2020 NRC Canada

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Emond, B. (2020). Learning Traces, Measurement and Assessment Templates for AIS Interoperability. In: Sottilare, R.A., Schwarz, J. (eds) Adaptive Instructional Systems. HCII 2020. Lecture Notes in Computer Science(), vol 12214. Springer, Cham. https://doi.org/10.1007/978-3-030-50788-6_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50788-6_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50787-9

  • Online ISBN: 978-3-030-50788-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation