Advertisement

Interruption Timing Prediction via Prosodic Task Boundary Model for Human-Machine Teaming

  • Nia PetersEmail author
Conference paper
First Online:
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 70)

Abstract

Human-machine teaming aims to meld the human cognitive strengths with the unique capabilities of smart machines to create intelligent teams adaptive to rapidly changing circumstances. One major problem within human-machine teaming is a lack of communication skills on the part of the machine such as the inability to know when to communicate information to or interrupt human teammates. To address this issue, an intelligent interruption system that monitors the speech within human-machine teaming interactions and predicts when to interrupt based on where human teammates are within the primary task is proposed. The intelligent interruption system leverages the raw audio within a simulated human-machine teaming interaction, extracts prosodic information, and predicts task boundaries as candidate interruption timings. Various machine learning techniques are evaluated as a prosody-only task boundary model and their task boundary detection performance is compared. The prosody-only task boundary model is implemented in real-time and the system latency and task boundary detection performance is evaluated. The final results indicate that although prosodic information processes with a low latency making it tractable in real-time, the prosody-only task boundary model performance is degraded in robust dialogues of human-machine teaming interactions.

Keywords

Human machine teaming Intelligent interruption Speech prosody Collaborative communication 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Adams, J.: Human machine teaming research (2017)Google Scholar
  2. 2.
    Stefik, M.: Half-human, half-computer? meet the modern centaur (2017)Google Scholar
  3. 3.
    Arroyo, E., Selker, T.: Attention and intention goals can mediate disruption in human-computer interaction, pp. 454–470 (2011)CrossRefGoogle Scholar
  4. 4.
    Bailey, B.P., Konstan, J.A.: On the need for attention-aware systems: measuring effects of interruption on task performance, error rate, and affective state. Comput. Hum. Behav. 22(4), 685–708 (2006)CrossRefGoogle Scholar
  5. 5.
    Czerwinski, M., Cutrell, E., Horvitz, E.: Instant messaging and interruption: influence of task type on performance. In: OZCHI 2000 Conference Proceedings, vol. 356, pp. 361–367 (2000)Google Scholar
  6. 6.
    Monk, C.A., Boehm-Davis, D.A., Gregory Trafton, J.: The attentional costs of interrupting task performance at various stage. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 46, pp. 1824–1828. SAGE Publications, Los Angeles, CA (2002)CrossRefGoogle Scholar
  7. 7.
    Czerwinski, M., Cutrell, E., Horvitz, E.: Instant messaging: effects of relevance and timing. People and Computers XIV: Proceedings of HCI 2, 71–76 (2000)Google Scholar
  8. 8.
    Adamczyk, P.D., Bailey, B.P.: If not now, when?: the effects of interruption at different moments within task execution. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 271–278. ACM (2004)Google Scholar
  9. 9.
    Zijlstra, F.R.H., Roe, R.A., Leonora, A.B., Krediet, I.: Temporal factors in mental work: effects of interrupted activities. J. Occup. Organ. Psychol. 72(2), 163–185 (1999)CrossRefGoogle Scholar
  10. 10.
    Scott McCrickard, D., Chewar, C.M., Somervell, J.P., Ndiwalana, A.: A model for notification systems evaluation?assessing user goals for multitasking activity. ACM Trans. Comput.-Hum. Interact. (TOCHI), 10, 312–338 (2003)CrossRefGoogle Scholar
  11. 11.
    McFarlane, D.C., Latorella, K.A.: The scope and importance of human interruption in human-computer interaction design. Hum.-Comput. Interact. 17(1), 1–61 (2002)CrossRefGoogle Scholar
  12. 12.
    Dabbish, L., Kraut, R.E., Controlling interruptions: awareness displays and social motivation for coordination. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 182–191. ACM (2004)Google Scholar
  13. 13.
    Latorella, K.A.: Investigating interruptions: an example from the flightdeck. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 40, pp. 249–253. SAGE Publications, Los Angeles, CA (1996)CrossRefGoogle Scholar
  14. 14.
    Kreifeldt, J.G., McCarthy, M.E.: Interruption as a test of the user-computer interface (1981)Google Scholar
  15. 15.
    Rubinstein, J.S., Meyer, D.E., Evans, J.E.: Executive control of cognitive processes in task switching. J. Exp. Psychol.: Hum. Percept. Perform. 27(4), 763 (2001)Google Scholar
  16. 16.
    Altmann, E.M., Trafton, J.G.: Task interruption: resumption lag and the role of cues. In: Proceedings of the Cognitive Science Society, vol. 26 (2004)Google Scholar
  17. 17.
    Sasse, A., Johnson, C., et al.: Coordinating the interruption of people in human-computer interaction. Hum.-Comput. Interact. 99, 295 (1999)Google Scholar
  18. 18.
    Horvitz, E.C.M.C.E., Notification, disruption, and memory: effects of messaging interruptions on memory and performance. In: Human-Computer Interaction: INTERACT’01: IFIP TC. 13 International Conference on Human-Comupter Interaction, 9th–13th July 2001, Tokyo, Japan, p. 263. IOS Press (2001)Google Scholar
  19. 19.
    McFarlane, D.C.: Interruption of people in human-computer interaction: a general unifying definition of human interruption and taxonomy. Technical Report, Office of Naval Research, Arlington VA (1997)CrossRefGoogle Scholar
  20. 20.
    Bailey, B.P., Iqbal, S.T.: Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Trans. Comput.-Hum. Interact. (TOCHI), 14(4), 21 (2008)CrossRefGoogle Scholar
  21. 21.
    Iqbal, S.T., Bailey, B.P.: Investigating the effectiveness of mental workload as a predictor of opportune moments for interruption. In: CHI’05 Extended Abstracts on Human Factors in Computing Systems, pp. 1489–1492. ACM (2005)Google Scholar
  22. 22.
    Peters, N., Romigh, G., Bradley, G., Raj, B.: When to interrupt: a comparative analysis of interruption timings within collaborative communication tasks. In: Advances in Human Factors and System Interactions, pp. 177–187. Springer (2017)Google Scholar
  23. 23.
    Horvitz, E., Principles of mixed-initiative user interfaces. In: Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 159–166. ACM (1999)Google Scholar
  24. 24.
    Horvitz, E., Apacible, J.: Learning and reasoning about interruption. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 20–27. ACM (2003)Google Scholar
  25. 25.
    Horvitz, E., Koch, P., Kadie, C.M., Jacobs, A.: Coordinate: probabilistic forecasting of presence and availability. In: Proceedings of the Eighteenth Conference on Uncertainty in Artificial Intelligence, pp. 224–233. Morgan Kaufmann Publishers Inc. (2002)Google Scholar
  26. 26.
    Begole, J.B., Matsakis, N.E., Tang, J.C.: Lilsys: sensing unavailability. In: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work, pp. 511–514. ACM (2004)Google Scholar
  27. 27.
    Fogarty, J., Lai, J., Christensen, J.: Presence versus availability: the design and evaluation of a context-aware communication client. Int. J. Hum.-Comput. Stud. 61(3), 299–317 (2004)CrossRefGoogle Scholar
  28. 28.
    Iqbal, S.T., Bailey, B.P., Leveraging characteristics of task structure to predict the cost of interruption. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 741–750. ACM (2006)Google Scholar
  29. 29.
    Miyata, Y., Norman, D.A.: Psychological issues in support of multiple activities. In: User Centered System Design: New Perspectives on Human-Computer Interaction, pp. 265–284 (1986)Google Scholar
  30. 30.
    Bannon, L., Cypher, A., Greenspan, S., Monty, M.L.: Evaluation and analysis of users’ activity organization. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 54–57. ACM (1983)Google Scholar
  31. 31.
    Miller, S.L.: Window of opportunity: Using the interruption lag to manage disruption in complex tasks. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 46, pp. 245–249. SAGE Publications, Los Angeles, CA (2002)CrossRefGoogle Scholar
  32. 32.
    Peters, N., Romigh, G., Bradley, G., Raj, B., A comparative analysis of human-mediated and system-mediated interruptions for multi-user, multitasking interactions. In: International Conference on Applied Human Factors and Ergonomics, pp. 339–347. Springer (2017)Google Scholar
  33. 33.
    Mushin, I., Stirling, L., Fletcher, J., Wales, R.: Discourse structure, grounding, and prosody in task-oriented dialogue. Discourse Process. 35(1), 1–31 (2003)CrossRefGoogle Scholar
  34. 34.
    Mixdorff, H., Quantitative analysis of prosody in task-oriented dialogs. In: Speech Prosody 2004, International Conference (2004)Google Scholar
  35. 35.
    Syrdal, A.K., Kim, Y.-J.: Dialog speech acts and prosody: considerations for TTS. In: Proceedings of Speech Prosody, pp. 661–665 (2008)Google Scholar
  36. 36.
    Hastie, H.W., Poesio, M., Isard, S.: Automatically predicting dialogue structure using prosodic features. Speech Commun. 36(1), 63–79 (2002)zbMATHCrossRefGoogle Scholar
  37. 37.
    Carletta, J., Isard, S., Doherty-Sneddon, G., Isard, A., Kowtko, J.C., Anderson, A.H.: The reliability of a dialogue structure coding scheme. Comput. Linguist. 23(1), 13–31 (1997)Google Scholar
  38. 38.
    Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in opensmile, the munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838. ACM (2013)Google Scholar
  39. 39.
    Banerjee, S.: Simple example code and generic function for random forests. https://www.mathworks.com/matlabcentral/fileexchange/51985-simple-example-code-and-generic-function-for-random-forests (2016)
  40. 40.
    Ho, T.K.: Random decision forests. In: Proceedings of the Third International Conference on Document Analysis and Recognition, vol. 1, pp. 278–282. IEEE (1995)Google Scholar
  41. 41.
    Peters, N., Raj, B., Romigh, G.: Topic and prosodic modeling for interruption management in multi-use multitasking communication interactions. In: AI-HRI (2018)Google Scholar
  42. 42.
    Russell, S., Norvig, P., Intelligence, Artificial: “A modern approach”. In: Artificial Intelligence, vol. 25, p. 27. Prentice-Hall, Englewood Cliffs (1995)Google Scholar
  43. 43.
    Cortes, C., Vapnik, V.: Support-vector networks. Machine Learn. 20(3), 273–297 (1995)Google Scholar
  44. 44.
    Swerts, M., Ostendorf, M.: Prosodic and lexical indications of discourse structure in human-machine interactions. Speech Commun. 22(1), 25–41 (1997)CrossRefGoogle Scholar
  45. 45.
    Shriberg, E., Stolcke, A., Hakkani-Tür, D., Tür, G.: Prosody-based automatic segmentation of speech into sentences and topics. Speech Commun. 32(1), 127–154 (2000)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Air Force Research Laboratory, Wright Patterson Air Force BaseDaytonUSA

Personalised recommendations

Cite paper

Buy options