Advertisement

The Challenge of Automatic Eating Behaviour Analysis and Tracking

  • Dagmar M. Schuller
  • Björn W. SchullerEmail author
Chapter
Part of the Intelligent Systems Reference Library book series (ISRL, volume 170)

Abstract

Computer-based tracking of eating behaviour is recently finding great interest by a broader choice of modalities such as by audio and video, or movement sensors, in particular in wearable every-day settings. Here, we provide an extensive insight into the current state-of-play for automatic tracking with a broader view on sensors and information used up to this point. The chapter is largely guided by and including results from the Interspeech 2015 Computational Paralinguistics Challenge (ComParE) Eating Sub-Challenge and the audio/visual Eating Analysis and Tracking (EAT) 2018 Challenge, both co-organised by the last author. The relevance is given by use-cases in health care and wellbeing including, amongst others, assistive technologies for individuals with eating disorders potentially leading either to under- or overeating or special health conditions such as diabetes. The chapter touches upon different feature representations including feature brute-forcing, bag-of-audio-word representations, and deep end-to-end learning from a raw sensor signal. It further reports on machine learning approaches used in the field including deep learning and conventional approaches. In the conclusion, the chapter discusses also usability aspects to foster optimal adherence, such as sensor placement, energy consumption, explainability, and privacy aspects.

Keywords

Eating analysis Eating disorders mHealth Health informatics Deep learning Artificial intelligence Multimodality Computational paralinguistics Assistive technologies 

References

  1. 1.
    Alshurafa, N., Kalantarian, H., Pourhomayoun, M., Liu, J.J., Sarin, S., Shahbazi, B., Sarrafzadeh, M.: Recognition of nutrition intake using time-frequency decomposition in a wearable necklace using a piezoelectric sensor. IEEE Sens. J. 15(7), 3909–3916 (2015)CrossRefGoogle Scholar
  2. 2.
    Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: An open source facial behavior analysis toolkit. In: Proceedings IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, pp. 1–10 (2016)Google Scholar
  3. 3.
    Bedri, A., Byrd, D., Presti, P., Sahni, H., Gue, Z., Starner, T.: Stick it in your ear: building an in-ear jaw movement sensor. In: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers, Osaka, Japan, pp. 1333–1338 (2015)Google Scholar
  4. 4.
    Cummins, N., Schuller, B.W., Baird, A.: Speech analysis for health: current state-of-the-art and the increasing impact of deep learning. Methods 151, 41–54 (2018). (Special Issue on Translational data analytics and health informatics)CrossRefGoogle Scholar
  5. 5.
    Deng, J., Schuller, B.: Confidence measures in speech emotion recognition based on semi-supervised learning. In: Proceedings Thirteenth Annual Conference of the International Speech Communication Association (Interspeech), Portland, OR, USA, pp. 2226–2229 (2012)Google Scholar
  6. 6.
    Drennan, M.: An assessment of linear wrist motion during the taking of a bite of food. Ph.D. Thesis. Clemson University, Clemson, SC, USA (2010)Google Scholar
  7. 7.
    Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings 21st ACM International Conference on Multimedia, Barcelona, Spain, pp. 835–838 (2013)Google Scholar
  8. 8.
    Fontana, J.M., Farooq, M., Sazonov, E.: Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behaviour. IEEE Trans. Biomed. Eng. 61(6), 1772–1779 (2014)CrossRefGoogle Scholar
  9. 9.
    Gao, Y., Zhang, N., Wang, H., Ding, X., Ye, X., Chen, G., Cao, Y.: iHear food: eating detection using commodity bluetooth headsets. In: Proceedings IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington, DC, USA, pp. 163–172 (2016)Google Scholar
  10. 10.
    Gosztolya, G., Tóth, L.: A feature selection-based speaker clustering method for paralinguistic tasks. Pattern Anal. Appl. 21(1), 193–204 (2018)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Guo, Y., Han, J., Zhang, Z., Schuller, B., Ma, Y.: Exploring a new method for food likability rating based on DT-CWT theory. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 569–573 (2018)Google Scholar
  12. 12.
    Haider, F., Pollak, S., Zarogianni, E., Luz, S.: SAAMEAT: active feature transformation and selection methods for the recognition of user eating conditions. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 564–568 (2018)Google Scholar
  13. 13.
    Hantke, S., Weninger, F., Kurle, R., Ringeval, F., Batliner, A., El-Desoky Mousa, A., Schuller, B.: I hear you eat and speak: automatic recognition of eating condition and food types, use-cases, and impact on ASR performance. PLoS ONE 11(5), 1–24 (2016)CrossRefGoogle Scholar
  14. 14.
    Hantke, S., Schmitt, M., Tzirakis, P., Schuller, B.: EAT—the ICMI 2018 eating analysis and tracking challenge. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 569–563 (2018)Google Scholar
  15. 15.
    Kaya, H., Karpov, A.A., Salah, A.A.: Fisher vectors with cascaded normalization for paralinguistic analysis. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 909–913 (2015)Google Scholar
  16. 16.
    Kim, J., Nasir, M., Gupta, R., van Segbroeck, M., Bone, D., Black, M.P., Skordilis, Z.I., Yang, Z., Georgiou, P.G., Narayanan, S.S.: Automatic estimation of Parkinson’s disease severity from diverse speech tasks. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 914–918 (2015)Google Scholar
  17. 17.
    Kitamura, K., de Silva, C., Yamasaki, T., Aizawa, K.: Image processing based approach to food balance analysis for personal food logging. In: Proceedings IEEE International Conference on Multimedia and Expo (ICME), Singapore, pp. 625–630 (2010)Google Scholar
  18. 18.
    Merck, C., Maher, C., Mirtchouk, M., Zheng, M., Huang, Y., Kleinberg, S.: Multimodality sensing for eating recognition. In: Proceedings 10th EAI International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, pp. 130–137 (2016)Google Scholar
  19. 19.
    Milde, B., Biemann, C.: Using representation learning and out-of-domain data for a paralinguistic speech task. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 904–908 (2015)Google Scholar
  20. 20.
    Oviatt, S., Schuller, B., Cohen, P., Sonntag, D., Potamianos, G.: The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition, vol. 2. Morgan & Claypool (2018)Google Scholar
  21. 21.
    Pellegrini, T.: Comparing SVM, Softmax, and shallow neural networks for eating condition classification. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 899–903 (2015)Google Scholar
  22. 22.
    Pir, D., Brown, T.: Acoustic group feature selection using wrapper method for automatic eating condition recognition. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 894–898 (2015)Google Scholar
  23. 23.
    Pir, D.: Functional-based acoustic group feature selection for automatic recognition of eating condition. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 579–583 (2018)Google Scholar
  24. 24.
    Prasad, A., Gosh, P.K.: Automatic classification of eating conditions from speech using acoustic feature selection and a set of hierarchical support vector machine classifiers. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 884–888 (2015)Google Scholar
  25. 25.
    Rahman, S.A., Merck, C., Huang, Y., Kleinberg, S.: Unintrusive eating recognition using Google Glass. In: Proceedings IEEE 9th International Conference Pervasive Computing Technologies for Healthcare (PervasiveHealth), Istanbul, Turkey, pp. 108–111 (2015)Google Scholar
  26. 26.
    Sazonov, E.S., Makeyev, O., Schuckers, S., Lopez-Meyer, P., Melanson, E.L., Neuman, M.R.: Automatic detection of swallowing events by acoustical means for applications of monitoring of ingestive behaviour. IEEE Trans. Biomed. Eng. 57(3), 626–633 (2010)CrossRefGoogle Scholar
  27. 27.
    Schmitt, M., Schuller, B.: OpenXBOW: introducing the Passau open-source crossmodal bag-of-words toolkit. J. Mach. Learn. Res. 18(1), 3370–3374 (2017)MathSciNetzbMATHGoogle Scholar
  28. 28.
    Schuller, B., Steidl, S., Batliner, A., Hantke, S., Hönig, F., Orozco-Arroyave, J. R., Nöth, E., Zhang, Y, Weninger, F.: The INTERSPEECH 2015 computational paralinguistics challenge: nativeness, Parkinson’s & eating condition. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 478–482 (2015)Google Scholar
  29. 29.
    Sertolli, B., Cummins, N., Sengur, A., Schuller, B.: Deep end-to-end representation learning for food type recognition from speech. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 574–578 (2018)Google Scholar
  30. 30.
    Thomaz, E., Zhang, C., Essa, I., Abowd, G.D.: Inferring meal eating activities in real world settings from ambient sounds: a feasibility study. In: Proceedings 20th ACM International Conference on Intelligent User Interfaces (IUI), Atlanta, GA, USA, pp. 427–431 (2015)Google Scholar
  31. 31.
    Tzirakis, P., Zafeiriou, S., Schuller, B.W.: End2You—The Imperial Toolkit for Multimodal Profiling by End-to-End Learning (2018). arXiv preprint arXiv:1802.01115
  32. 32.
    Val-Laillet, D., Aarts, E., Weber, B., Ferrari, M., Quaresima, V., Stoeckel, L.E., Alonso-Alonso, M., Audette, M., Malbert, C.H., Stice, E.: Neuroimaging and neuromodulation approaches to study eating behavior and prevent and treat eating disorders and obesity. Neuro Image Clin. 8, 1–31 (2015)Google Scholar
  33. 33.
    Wagner, J., Seiderer, A., Lingenfelser, F., André, E.: Combining hierarchical classification with frequency weighting for the recognition of eating conditions. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 889–893 (2015)Google Scholar
  34. 34.
    World Health Organization: Obesity and Overweight (2018). https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight
  35. 35.
    World Health Organization: Diabetes (2019). https://www.who.int/news-room/fact-sheets/detail/diabetes. Accessed 3 Feb 2019

Additional Reading Section (Resource List)

  1. 36.
    Amft, O., Junker, H., Troster, G.: Detection of eating and drinking arm gestures using inertial body-worn sensors. In: Proceedings Ninth IEEE International Symposium on Wearable Computers (ISWC), Osaka, Japan, pp. 160–163 (2005)Google Scholar
  2. 37.
    Dong, Y., Scisco, J., Wilson, M., Muth, E., Hoover, A.: Detecting periods of eating during free-living by tracking wrist motion. IEEE J. Biomed. Health Inf. 18(4), 1253–1260 (2014)CrossRefGoogle Scholar
  3. 38.
    Liu, J., Johns, E., Atallah, L., Pettitt, C., Lo, B., Frost, G., Yang, G.Z.: An intelligent food-intake monitoring system using wearable sensors. In: 2012 Ninth IEEE International Conference on Wearable and Implantable Body Sensor Networks (BSN), London, UK, pp. 154–160 (2012)Google Scholar
  4. 39.
    Mirtchouk, M., Merck, C., Kleinberg, S.: Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In: Proceedings 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), Heidelberg, Germany, pp. 451–462 (2016)Google Scholar
  5. 40.
    Nguyen, D.T., Cohen, E., Pourhomayoun, M., Alshurafa, N.: SwallowNet: recurrent neural network detects and characterizes eating patterns. In: Proceedings IEEE International Conference on Pervasive Computing and Communications Workshops, PerCom Workshops, Kona, HI, USA, pp. 401–406 (2017)Google Scholar
  6. 41.
    Schuller, B., Batliner, A.: Computational Paralinguistics. John Wiley & Sons (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.audEERING GmbHGilchingGermany
  2. 2.ZD.B Chair of Embedded Intelligence for Health Care and WellbeingUniversity of AugsburgAugsburgGermany
  3. 3.GLAM - Group on Language, Audio & Music, Imperial College LondonLondonUK

Personalised recommendations