Designing a Multimodal Analytics System to Improve Emergency Response Training

  • Hemant PurohitEmail author
  • Samantha Dubrow
  • Brenda Bannan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11590)


The role of high-fidelity simulation-based training is critical for preparing first responders to perform effectively in emergency response and firefighting environments. However, training simulation instructors currently rely on direct observation and radio-based audio communication methods to observe behaviors and evaluate interactions of trainees to provide feedback to trainees during debriefing sessions. Such human-driven evaluative methods, while valuable, lose fidelity of information for learning due to working memory limitations and the mind’s inability to capture all relevant behavioral information in realtime and to provide relevant detailed information that is easily interpretable to instructors. Thus, there is a need and the potential to explore alternative data collection and processing methods leveraging advanced wireless technologies to attempt to enhance the effectiveness of simulation training process for emergency response, to ultimately help save lives and better prepare our communities building emergency and disaster resilience. In this paper, we present a multimodal streaming analytics system to support learning by leveraging a user-centered design approach in consultation with a regional fire and rescue training academy. This conceptualized system provides real-time collection, visualization, and analysis of heterogeneous data streams including location sensing using Internet of Things (IoT) devices, audio communication, video observations, as well as social media and 911 call log streams. We describe the associated design challenges and lessons learned from the initial prototyping activities to strive toward enhancing the situational awareness and learning of emergency response personnel and leadership instructors with iterative design cycles at the regional fire and rescue training academy.


Learning Realtime streams IoT Smart cities Disaster resilience 



Authors would like to thank undergraduate research assistant Mohammad Rana for implementing the preliminary system and U.S. National Science Foundation grants DRL-1637263 and IIS-1815459 for partially supporting this research. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.


  1. Al-Shehri, S., Gitsaki, C.: Online reading: a preliminary study of the impact of integrated and split-attention formats on L2 students’ cognitive load. ReCALL 22(3), 356–375 (2010)CrossRefGoogle Scholar
  2. Ayres, P., Cierniak, G.: Split-attention effect. In: Seel, N.M. (ed.) Encyclopedia of the Sciences of Learning, pp. 3172–3175. Springer, Boston (2012). Scholar
  3. Bannan, B., Dubrow, S., Dobbins, C., Zaccaro, S., Purohit, H., Rana, M.: Toward wearable devices for multiteam systems learning. In: Buchem, I., Klamma, R., Wild, F. (eds.) Perspectives on Wearable Enhanced Learning: Current Trends Research and Practice, pp. 1–29. Springer, New York (2019, in press)Google Scholar
  4. Buck, D.A., Trainor, J.E., Aguirre, B.E.: A critical evaluation of the incident command system and NIMS. J. Homel. Secur. Emerg. Manag. 3(3), 1–27 (2006)Google Scholar
  5. Buller, M., Welles, A., Jenkins, O.C., Hoyt, R.: Extreme health sensing: the challenges, technologies, and strategies for active health sustainment of military personnel during training and combat missions. In: Sensors, and Command, Control, Communications, and Intelligence (C3I) Technologies for Homeland Security and Homeland Defense IX, vol. 7666, p. 766610. International Society for Optics and Photonics (2010)Google Scholar
  6. California Hospital Association: Emergency Preparedness: Types of Exercises. Accessed 1 Mar 2019
  7. Chen, O., Woolcott, G., Sweller, J.: Using cognitive load theory to structure computer-based learning including MOOCs. J. Comput. Assist. Learn. 33(4), 293–305 (2017)CrossRefGoogle Scholar
  8. Dubrow, S., et al.: Using IoT sensors to enhance simulation and training in multiteam systems. In: Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) Published, pp. 1–10 (2017)Google Scholar
  9. Dubrow, S., Bannan, B.: Toward improving interagency learning in emergency response simulations contexts with wearable technologies. To be Presented at the 21st Annual Conference of Human-Computer Interaction, Orlando, FL (2019)Google Scholar
  10. Feese, S., Arnrich, B., Troster, G., Burtscher, M., Meyer, B., Jonas, K.: CoenoFire: monitoring performance indicators of firefighters in real-world missions using smartphones. In: Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 83–92. ACM (2013)Google Scholar
  11. Glassman, M., Kang, M.J.: Intelligence in the internet age: the emergence and evolution of Open Source Intelligence (OSINT). Comput. Hum. Behav. 28(2), 673–682 (2012)CrossRefGoogle Scholar
  12. Jenvald, J., Magnus, M.: Simulation-supported live training for emergency response in hazardous environments. Simul. Gaming 35(3), 363–377 (2004)CrossRefGoogle Scholar
  13. Jones, D.G.: A practical perspective on the utility of situation awareness. J. Cogn. Eng. Decis. Mak. 9(1), 98–100 (2015)CrossRefGoogle Scholar
  14. Kalyuga, S., Chandler, P., Sweller, J.: Managing split-attention and redundancy in multimedia instruction. Appl. Cogn. Psychol.: Off. J. Soc. Appl. Res. Mem. Cogn. 13(4), 351–371 (1999)CrossRefGoogle Scholar
  15. Kranzfelder, M., Schneider, A., Gillen, S., Feussner, H.: New technologies for information retrieval to achieve situational awareness and higher patient safety in the surgical operating room: the MRI institutional approach and review of the literature. Surg. Endosc. 25(3), 696–705 (2011)CrossRefGoogle Scholar
  16. Liu, T.C., Lin, Y.C., Tsai, M.J., Paas, F.: Split-attention and redundancy effects on mobile learning in physical environments. Comput. Educ. 58(1), 172–180 (2012)CrossRefGoogle Scholar
  17. Pandey, R., Purohit, H.: CitizenHelper-adaptive: expert-augmented streaming analytics system for emergency services and humanitarian organizations. In: 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 630–633. IEEE (2018)Google Scholar
  18. Rajkumar, R., Lee, I., Sha, L., Stankovic, J.: Cyber-physical systems: the next computing revolution. In: Design Automation Conference, pp. 731–736. IEEE (2010)Google Scholar
  19. Stanton, N.A., Salmon, P.M., Walker, G.H., Salas, E., Hancock, P.A.: State-of-science: situation awareness in individuals, teams and systems. Ergonomics 60(4), 449–466 (2017). Scholar
  20. Stanton, N.A., et al.: Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology. Ergonomics 49(12–13), 1288–1311 (2006)CrossRefGoogle Scholar
  21. Van de Walle, B., Brugghemans, B., Comes, T.: Improving situation awareness in crisis response teams: an experimental analysis of enriched information and centralized coordination. Int. J. Hum.-Comput. Stud. 95, 66–79 (2016)CrossRefGoogle Scholar
  22. Volunteer Fairfax: Fairfax County Community Resiliency Collaboration: Tabletop Exercise Series - Braddock District Situation Manual. Accessed 1 Mar 2019
  23. Wallach, D., Scholz, S.C.: User-centered design: why and how to put users first in software development. In: Maedche, A., Botzenhardt, A., Neer, L. (eds.) Software for People, pp. 11–38. Springer, Heidelberg (2012). Scholar
  24. Xiaodong, J., Hong, J.I., Takayama, L.A., Landay, J.A.: Ubiquitous computing for firefighters: field studies and prototypes of large displays for incident command. In: Proceedings of CHI Conference on Human Factors in Computing Systems, Vienna, Austria (2004)Google Scholar
  25. Zaccaro, S.J., Marks, M.A., DeChurch, L.A.: Multiteam systems: an introduction. In: Multiteam Systems, pp. 18–47. Routledge (2012)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hemant Purohit
    • 1
    Email author
  • Samantha Dubrow
    • 2
  • Brenda Bannan
    • 3
  1. 1.Information Sciences and Technology, Volgenau School of EngineeringGeorge Mason UniversityFairfaxUSA
  2. 2.Department of Psychology, College of Humanities and Social SciencesGeorge Mason UniversityFairfaxUSA
  3. 3.Division of Learning Technologies, College of Education and Human DevelopmentGeorge Mason UniversityFairfaxUSA

Personalised recommendations