Detection and Recognition of Food in Photo Galleries for Analysis of User Preferences

  • Evgeniy MiasnikovEmail author
  • Andrey Savchenko
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12131)


Food analysis is one of the most important parts of user preference prediction engines for recommendation systems in the travel domain. In this paper, we describe and study the neural network method that allows you to recognize food in a gallery of photos taken with mobile devices. The described method consists of three main stages, including the classification of scenes, food detection, and subsequent classification. An essential feature of the developed method is the use of lightweight neural network models, which allows its usage on mobile devices. The development of the method was carried out using both known open data and a proprietary data set.


Scene recognition Food detection Food recognition Convolutional neural networks (CNN) 



This research is based on the work supported by Samsung Research, Samsung Electronics.


  1. 1.
    Savchenko, A.V., Demochkin, K.V., Grechikhin, I.S.: User preference prediction in visual data on mobile devices. arXiv preprint 1907.04519 (2019)Google Scholar
  2. 2.
    Matsuda, Y., Yanai, K.: Multiple-food recognition considering co-occurrence employing manifold ranking. In: Proceedings of the 21st International Conference on Pattern Recognition, ICPR 2012, pp. 2017–2020, November 2012Google Scholar
  3. 3.
    Kitamura, K., Yamasaki, T., Aizawa, K.: FoodLog: capture, analysis and retrieval of personal food images via web. In: Proceedings of the ACM Multimedia 2009 Workshop on Multimedia for Cooking and Eating Activities, CEA 2009, pp. 23–30. Association for Computing Machinery, New York (2009)Google Scholar
  4. 4.
    Farinella, G.M., Allegra, D., Stanco, F., Battiato, S.: On the exploitation of one class classification to distinguish food vs non-food images. In: Murino, V., Puppo, E., Sona, D., Cristani, M., Sansone, C. (eds.) ICIAP 2015. LNCS, vol. 9281, pp. 375–383. Springer, Cham (2015). Scholar
  5. 5.
    Ragusa, F., Tomaselli, V., Furnari, A., Battiato, S., Farinella, G.M.: Food vs non-food classification. In: Proceedings of the International Workshop on Multimedia Assisted Dietary Management (MADiMa), pp. 77–81. ACM (2016)Google Scholar
  6. 6.
    Myers, A., et al.: Im2Calories: towards an automated mobile vision food diary. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1233–1241, December 2015Google Scholar
  7. 7.
    Anzawa, M., Amano, S., Yamakata, Y., Motonaga, K., Kamei, A., Aizawa, K.: Recognition of multiple food items in a single photo for use in a buffet-style restaurant. IEICE Trans. Inf. Syst. E102.D(2), 410–414 (2019)Google Scholar
  8. 8.
    Kagaya, H., Aizawa, K., Ogawa, M.: Food detection and recognition using convolutional neural network. In: Proceedings of the 22nd ACM International Conference on Multimedia, MM 2014, pp. 1085–1088. Association for Computing Machinery, New York (2014)Google Scholar
  9. 9.
    Singla, A., Yuan, L., Ebrahimi, T.: Food/non-food image classification and food categorization using pre-trained GoogLeNet model. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management. MADiMa 2016. Association for Computing Machinery, New York (2016)Google Scholar
  10. 10.
    Aguilar, E., Bolaños, M., Radeva, P.: Exploring food detection using CNNs. In: Moreno-Díaz, R., Pichler, F., Quesada-Arencibia, A. (eds.) EUROCAST 2017. LNCS, vol. 10672, pp. 339–347. Springer, Cham (2018). Scholar
  11. 11.
    Oliveira, L., Costa, V., Neves, G., Oliveira, T., Jorge, E., Lizarraga, M.: A mobile, lightweight, poll-based food identification system. Pattern Recogn. 47(5), 1941–1952 (2014)CrossRefGoogle Scholar
  12. 12.
    Martinel, N., Piciarelli, C., Micheloni, C., Foresti, G.L.: A structured committee for food recognition. In: 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), pp. 484–492 (2015)Google Scholar
  13. 13.
    Zheng, J., Wang, Z., Zhu, C.: Food image recognition via superpixel based low-level and mid-level distance coding for smart home applications. Sustainability 9(5), 856 (2017)CrossRefGoogle Scholar
  14. 14.
    Bettadapura, V., Thomaz, E., Parnami, A., Abowd, G.D., Essa, I.: Leveraging context to support automated food recognition in restaurants. In: Proceedings of the Winter Conference on Applications of Computer Vision (WACV), pp. 580–587. IEEE (2015)Google Scholar
  15. 15.
    Bolanos, M., P., R.: Simultaneous food localization and recognition. In: International Conference on Pattern Recognition, pp. 3140–3145 (2017)Google Scholar
  16. 16.
    Wu, H., Merler, M., Uceda-Sosa, R., Smith, J.R.: Learning to make better mistakes: semantics-aware visual food recognition. In: Proceedings of the 24th International Conference on Multimedia (MM), pp. 172–176. ACM (2016)Google Scholar
  17. 17.
    Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: a new dataset, experiments, and results. IEEE J. Biomed. Health Inform. 21(3), 588–598 (2016)CrossRefGoogle Scholar
  18. 18.
    Kaur, P., Sikka, K., Wang, W., Belongie, S., Divakaran, A.: FoodX-251: a dataset for fine-grained food classification. arXiv preprint 1907.06167 (2019)Google Scholar
  19. 19.
    Ming, Z.-Y., Chen, J., Cao, Y., Forde, C., Ngo, C.-W., Chua, T.S.: Food photo recognition for dietary tracking: system and experiment. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10705, pp. 129–141. Springer, Cham (2018). Scholar
  20. 20.
    Aguilar, E., Bolaños, M., Radeva, P.: Food recognition using fusion of classifiers based on CNNs. In: Battiato, S., Gallo, G., Schettini, R., Stanco, F. (eds.) ICIAP 2017. LNCS, vol. 10485, pp. 213–224. Springer, Cham (2017). Scholar
  21. 21.
    Xin Wang, Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: Proceedings of the International Conference on Multimedia Expo Workshops (ICMEW), pp. 1–6. IEEE (2015)Google Scholar
  22. 22.
    Liu, C., Cao, Y., Luo, Y., Chen, G., Vokkarane, V., Ma, Y.: DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. In: Chang, C.K., Chiari, L., Cao, Y., Jin, H., Mokhtari, M., Aloulou, H. (eds.) ICOST 2016. LNCS, vol. 9677, pp. 37–48. Springer, Cham (2016). Scholar
  23. 23.
    Farinella, G.M., Allegra, D., Stanco, F.: A benchmark dataset to study the representation of food images. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 584–599. Springer, Cham (2015). Scholar
  24. 24.
    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014). Scholar
  25. 25.
    Zhang, H., Cissé, M., Dauphin, Y.N., Lopez-Paz, D.: Mixup: beyond empirical risk minimization. CoRR abs/1710.09412 (2017)Google Scholar
  26. 26.
    Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. CoRR abs/1709.01507 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.St. Petersburg Department of Steklov Institute of MathematicsSt. PetersburgRussia
  2. 2.Laboratory of Algorithms and Technologies for Network AnalysisNational Research University Higher School of EconomicsNizhny NovgorodRussia

Personalised recommendations