Skip to main content
SpringerLink
Log in

The Challenge of Automatic Eating Behaviour Analysis and Tracking

  • Chapter
  • First Online:
Recent Advances in Intelligent Assistive Technologies: Paradigms and Applications

Part of the book series: Intelligent Systems Reference Library ((ISRL,volume 170))

Abstract

Computer-based tracking of eating behaviour is recently finding great interest by a broader choice of modalities such as by audio and video, or movement sensors, in particular in wearable every-day settings. Here, we provide an extensive insight into the current state-of-play for automatic tracking with a broader view on sensors and information used up to this point. The chapter is largely guided by and including results from the Interspeech 2015 Computational Paralinguistics Challenge (ComParE) Eating Sub-Challenge and the audio/visual Eating Analysis and Tracking (EAT) 2018 Challenge, both co-organised by the last author. The relevance is given by use-cases in health care and wellbeing including, amongst others, assistive technologies for individuals with eating disorders potentially leading either to under- or overeating or special health conditions such as diabetes. The chapter touches upon different feature representations including feature brute-forcing, bag-of-audio-word representations, and deep end-to-end learning from a raw sensor signal. It further reports on machine learning approaches used in the field including deep learning and conventional approaches. In the conclusion, the chapter discusses also usability aspects to foster optimal adherence, such as sensor placement, energy consumption, explainability, and privacy aspects.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alshurafa, N., Kalantarian, H., Pourhomayoun, M., Liu, J.J., Sarin, S., Shahbazi, B., Sarrafzadeh, M.: Recognition of nutrition intake using time-frequency decomposition in a wearable necklace using a piezoelectric sensor. IEEE Sens. J. 15(7), 3909–3916 (2015)

    Article  Google Scholar 

  2. Baltrušaitis, T., Robinson, P., Morency, L.P.: OpenFace: An open source facial behavior analysis toolkit. In: Proceedings IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, pp. 1–10 (2016)

    Google Scholar 

  3. Bedri, A., Byrd, D., Presti, P., Sahni, H., Gue, Z., Starner, T.: Stick it in your ear: building an in-ear jaw movement sensor. In: Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers, Osaka, Japan, pp. 1333–1338 (2015)

    Google Scholar 

  4. Cummins, N., Schuller, B.W., Baird, A.: Speech analysis for health: current state-of-the-art and the increasing impact of deep learning. Methods 151, 41–54 (2018). (Special Issue on Translational data analytics and health informatics)

    Article  Google Scholar 

  5. Deng, J., Schuller, B.: Confidence measures in speech emotion recognition based on semi-supervised learning. In: Proceedings Thirteenth Annual Conference of the International Speech Communication Association (Interspeech), Portland, OR, USA, pp. 2226–2229 (2012)

    Google Scholar 

  6. Drennan, M.: An assessment of linear wrist motion during the taking of a bite of food. Ph.D. Thesis. Clemson University, Clemson, SC, USA (2010)

    Google Scholar 

  7. Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings 21st ACM International Conference on Multimedia, Barcelona, Spain, pp. 835–838 (2013)

    Google Scholar 

  8. Fontana, J.M., Farooq, M., Sazonov, E.: Automatic ingestion monitor: a novel wearable device for monitoring of ingestive behaviour. IEEE Trans. Biomed. Eng. 61(6), 1772–1779 (2014)

    Article  Google Scholar 

  9. Gao, Y., Zhang, N., Wang, H., Ding, X., Ye, X., Chen, G., Cao, Y.: iHear food: eating detection using commodity bluetooth headsets. In: Proceedings IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE), Washington, DC, USA, pp. 163–172 (2016)

    Google Scholar 

  10. Gosztolya, G., Tóth, L.: A feature selection-based speaker clustering method for paralinguistic tasks. Pattern Anal. Appl. 21(1), 193–204 (2018)

    Article  MathSciNet  Google Scholar 

  11. Guo, Y., Han, J., Zhang, Z., Schuller, B., Ma, Y.: Exploring a new method for food likability rating based on DT-CWT theory. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 569–573 (2018)

    Google Scholar 

  12. Haider, F., Pollak, S., Zarogianni, E., Luz, S.: SAAMEAT: active feature transformation and selection methods for the recognition of user eating conditions. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 564–568 (2018)

    Google Scholar 

  13. Hantke, S., Weninger, F., Kurle, R., Ringeval, F., Batliner, A., El-Desoky Mousa, A., Schuller, B.: I hear you eat and speak: automatic recognition of eating condition and food types, use-cases, and impact on ASR performance. PLoS ONE 11(5), 1–24 (2016)

    Article  Google Scholar 

  14. Hantke, S., Schmitt, M., Tzirakis, P., Schuller, B.: EAT—the ICMI 2018 eating analysis and tracking challenge. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 569–563 (2018)

    Google Scholar 

  15. Kaya, H., Karpov, A.A., Salah, A.A.: Fisher vectors with cascaded normalization for paralinguistic analysis. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 909–913 (2015)

    Google Scholar 

  16. Kim, J., Nasir, M., Gupta, R., van Segbroeck, M., Bone, D., Black, M.P., Skordilis, Z.I., Yang, Z., Georgiou, P.G., Narayanan, S.S.: Automatic estimation of Parkinson’s disease severity from diverse speech tasks. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 914–918 (2015)

    Google Scholar 

  17. Kitamura, K., de Silva, C., Yamasaki, T., Aizawa, K.: Image processing based approach to food balance analysis for personal food logging. In: Proceedings IEEE International Conference on Multimedia and Expo (ICME), Singapore, pp. 625–630 (2010)

    Google Scholar 

  18. Merck, C., Maher, C., Mirtchouk, M., Zheng, M., Huang, Y., Kleinberg, S.: Multimodality sensing for eating recognition. In: Proceedings 10th EAI International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, pp. 130–137 (2016)

    Google Scholar 

  19. Milde, B., Biemann, C.: Using representation learning and out-of-domain data for a paralinguistic speech task. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 904–908 (2015)

    Google Scholar 

  20. Oviatt, S., Schuller, B., Cohen, P., Sonntag, D., Potamianos, G.: The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition, vol. 2. Morgan & Claypool (2018)

    Google Scholar 

  21. Pellegrini, T.: Comparing SVM, Softmax, and shallow neural networks for eating condition classification. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 899–903 (2015)

    Google Scholar 

  22. Pir, D., Brown, T.: Acoustic group feature selection using wrapper method for automatic eating condition recognition. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 894–898 (2015)

    Google Scholar 

  23. Pir, D.: Functional-based acoustic group feature selection for automatic recognition of eating condition. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 579–583 (2018)

    Google Scholar 

  24. Prasad, A., Gosh, P.K.: Automatic classification of eating conditions from speech using acoustic feature selection and a set of hierarchical support vector machine classifiers. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 884–888 (2015)

    Google Scholar 

  25. Rahman, S.A., Merck, C., Huang, Y., Kleinberg, S.: Unintrusive eating recognition using Google Glass. In: Proceedings IEEE 9th International Conference Pervasive Computing Technologies for Healthcare (PervasiveHealth), Istanbul, Turkey, pp. 108–111 (2015)

    Google Scholar 

  26. Sazonov, E.S., Makeyev, O., Schuckers, S., Lopez-Meyer, P., Melanson, E.L., Neuman, M.R.: Automatic detection of swallowing events by acoustical means for applications of monitoring of ingestive behaviour. IEEE Trans. Biomed. Eng. 57(3), 626–633 (2010)

    Article  Google Scholar 

  27. Schmitt, M., Schuller, B.: OpenXBOW: introducing the Passau open-source crossmodal bag-of-words toolkit. J. Mach. Learn. Res. 18(1), 3370–3374 (2017)

    MathSciNet  MATH  Google Scholar 

  28. Schuller, B., Steidl, S., Batliner, A., Hantke, S., Hönig, F., Orozco-Arroyave, J. R., Nöth, E., Zhang, Y, Weninger, F.: The INTERSPEECH 2015 computational paralinguistics challenge: nativeness, Parkinson’s & eating condition. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 478–482 (2015)

    Google Scholar 

  29. Sertolli, B., Cummins, N., Sengur, A., Schuller, B.: Deep end-to-end representation learning for food type recognition from speech. In: Proceedings 20th ACM International Conference on Multimodal Interaction (ICMI), Boulder, Colorado, pp. 574–578 (2018)

    Google Scholar 

  30. Thomaz, E., Zhang, C., Essa, I., Abowd, G.D.: Inferring meal eating activities in real world settings from ambient sounds: a feasibility study. In: Proceedings 20th ACM International Conference on Intelligent User Interfaces (IUI), Atlanta, GA, USA, pp. 427–431 (2015)

    Google Scholar 

  31. Tzirakis, P., Zafeiriou, S., Schuller, B.W.: End2You—The Imperial Toolkit for Multimodal Profiling by End-to-End Learning (2018). arXiv preprint arXiv:1802.01115

  32. Val-Laillet, D., Aarts, E., Weber, B., Ferrari, M., Quaresima, V., Stoeckel, L.E., Alonso-Alonso, M., Audette, M., Malbert, C.H., Stice, E.: Neuroimaging and neuromodulation approaches to study eating behavior and prevent and treat eating disorders and obesity. Neuro Image Clin. 8, 1–31 (2015)

    Google Scholar 

  33. Wagner, J., Seiderer, A., Lingenfelser, F., André, E.: Combining hierarchical classification with frequency weighting for the recognition of eating conditions. In: Proceedings Sixteenth Annual Conference of the International Speech Communication Association (Interspeech), Dresden, Germany, pp. 889–893 (2015)

    Google Scholar 

  34. World Health Organization: Obesity and Overweight (2018). https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight

  35. World Health Organization: Diabetes (2019). https://www.who.int/news-room/fact-sheets/detail/diabetes. Accessed 3 Feb 2019

Additional Reading Section (Resource List)

  1. Amft, O., Junker, H., Troster, G.: Detection of eating and drinking arm gestures using inertial body-worn sensors. In: Proceedings Ninth IEEE International Symposium on Wearable Computers (ISWC), Osaka, Japan, pp. 160–163 (2005)

    Google Scholar 

  2. Dong, Y., Scisco, J., Wilson, M., Muth, E., Hoover, A.: Detecting periods of eating during free-living by tracking wrist motion. IEEE J. Biomed. Health Inf. 18(4), 1253–1260 (2014)

    Article  Google Scholar 

  3. Liu, J., Johns, E., Atallah, L., Pettitt, C., Lo, B., Frost, G., Yang, G.Z.: An intelligent food-intake monitoring system using wearable sensors. In: 2012 Ninth IEEE International Conference on Wearable and Implantable Body Sensor Networks (BSN), London, UK, pp. 154–160 (2012)

    Google Scholar 

  4. Mirtchouk, M., Merck, C., Kleinberg, S.: Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In: Proceedings 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp), Heidelberg, Germany, pp. 451–462 (2016)

    Google Scholar 

  5. Nguyen, D.T., Cohen, E., Pourhomayoun, M., Alshurafa, N.: SwallowNet: recurrent neural network detects and characterizes eating patterns. In: Proceedings IEEE International Conference on Pervasive Computing and Communications Workshops, PerCom Workshops, Kona, HI, USA, pp. 401–406 (2017)

    Google Scholar 

  6. Schuller, B., Batliner, A.: Computational Paralinguistics. John Wiley & Sons (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. audEERING GmbH, Gilching, Germany

    Dagmar M. Schuller & Björn W. Schuller

  2. ZD.B Chair of Embedded Intelligence for Health Care and Wellbeing, University of Augsburg, Augsburg, Germany

    Björn W. Schuller

  3. GLAM - Group on Language, Audio & Music, Imperial College London, London, UK

    Björn W. Schuller

Authors
  1. Dagmar M. Schuller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Björn W. Schuller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Björn W. Schuller .

Editor information

Editors and Affiliations

  1. Faculty of Medical Bioengineering, University of Medicine and Pharmacy, Iași, Romania

    Hariton Costin

  2. University of Augsburg, Augsburg, Germany

    Björn Schuller

  3. Faculty of Automatic Control and Computers, University Polytechnica of Bucharest, Bucharest, Romania

    Adina Magda Florea

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Schuller, D.M., Schuller, B.W. (2020). The Challenge of Automatic Eating Behaviour Analysis and Tracking. In: Costin, H., Schuller, B., Florea, A. (eds) Recent Advances in Intelligent Assistive Technologies: Paradigms and Applications. Intelligent Systems Reference Library, vol 170. Springer, Cham. https://doi.org/10.1007/978-3-030-30817-9_8

Download citation

Publish with us

Policies and ethics

Search

Navigation