Design in Everyday Cooking: Challenges for Assisting with Menu Planning and Food Preparation

  • Atsushi HashimotoEmail author
  • Jun Harashima
  • Yoko Yamakata
  • Shinsuke Mori
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9741)


In this study, we introduce challenges for assisting with everyday cooking activities. Menu planning is the first step in daily cooking, and there are many commercial services available. We introduce the case study of “cookpad,” one of the largest recipe portal sites, and illustrate their efforts to maintain an up-to-date recipe search system. As an academic challenge, situated recipe recommendation is also introduced. Food preparation is another important topic. We present our perspective based on the relationship between recipe texts and cooking activities, along with related studies.


Recipe Cooking activity 



This work was supported by JSPS KAKENHI Grant Numbers 24240030, 26280039, 26280084.


  1. 1.
    Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
  2. 2.
    Anthimopoulos, M., Gianola, L., Scarnato, L., Diem, P., Mougiakakou, S.: A food recognition system for diabetic patients based on an optimized bag of features approach. IEEE J. Biomed. Health Inform. 18(4), 1261–1271 (2014)CrossRefGoogle Scholar
  3. 3.
    Sudo, K., Murasaki, K., Shimamura, J., Taniguchi, Y.: Estimating nutritional value from food images based on semantic segmentation. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. UbiComp 2014 Adjunct, pp. 571–576 (2014)Google Scholar
  4. 4.
    Kitamura, K., Yamasaki, T., Aizawa, K.: Foodlog: capture, analysis and retrieval of personal food images via web. In: Proceedings of the ACM Multimedia 2009 Workshop on Multimedia for Cooking and Eating Activities, pp. 23–30 (2009)Google Scholar
  5. 5.
    Khanna, N., Boushey, C.J., Kerr, D., Okos, M., Ebert, D.S., Delp, E.J.: An overview of the technology assisted dietary assessment project at purdue university. In: Proceedings of 2010 IEEE International Symposium on Multimedia, pp. 290–295 (2010)Google Scholar
  6. 6.
    Kawano, Y., Yanai, K.: Food image recognition with deep convolutional features. In: Proceedings of ACM UbiComp Workshop on Workshop on Smart Technology for Cooking and Eating Activities (CEA), September 2014Google Scholar
  7. 7.
    Takeuchi, T., Fujii, T., Narumi, T., Tanikawa, T., Hirose, M.: Considering individual taste in social feedback to improve eating habits. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
  8. 8.
    Harashima, J., Ariga, M., Murata, K., Ioki, M.: A large-scale recipe and meal data collection as infrastructure for food research. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (2016, to appear)Google Scholar
  9. 9.
    Mynavi Corporation: Cooking related questionary investi-gation reported by Mynavi woman on 27th (in Japanese). Accessed 1 Feb 2016
  10. 10.
    Oyama, S., Kokubo, T., Ishida, T.: Domain-specific web search with keyword spices. IEEE Trans. Knowl. Data Eng. 16(1), 17–27 (2004)CrossRefGoogle Scholar
  11. 11.
    Tsukuda, K., Yamamoto, T., Nakamura, S., Tanaka, K.: Plus one or minus one: a method to browse from an object to another object by adding or deleting an element. In: Bringas, P.G., Hameurlain, A., Quirchmayr, G. (eds.) DEXA 2010, Part II. LNCS, vol. 6262, pp. 258–266. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Chung, Y.: Finding food entity relationships using user-generated data in recipe service. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 2611–2614 (2012)Google Scholar
  13. 13.
    Ai-Land Co., Ltd.: Recipe blog (in Japanese). Accessed 1 Feb 2016
  14. 14.
    Kadowaki, T., Mori, S., Yamakata, Y., Tanaka, K.: Recipe search for blog-type recipe articles based on a users situation. In: Proceedings of ACM Conference on Ubiquitous Computing, pp. 497–506 (2014)Google Scholar
  15. 15.
    Zhang, K., Jiang, T.: Some MAX SNP-hard results concerning unordered labeled trees. Inf. Process. Lett. 49(5), 249–254 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Yamakata, Y., Imahori, S., Sugiyama, Y., Mori, S., Tanaka, K.: Feature extraction and summarization of recipes using flow graph. In: Jatowt, A., et al. (eds.) SocInfo 2013. LNCS, vol. 8238, pp. 241–254. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  17. 17.
    Wang, L., Li, Q., Li, N., Dong, G., Yang, Y.: Substructure similarity measurement in chinese recipes. In: Proceedings of the 17th International Conference on World Wide Web, pp. 979–988 (2008)Google Scholar
  18. 18.
    Hashimoto, A., Inoue, J., Funatomi, T., Minoh, M.: How does user’s access to object make HCI smooth in recipe guidance? In: Rau, P.L.P. (ed.) CCD 2014. LNCS, vol. 8528, pp. 150–161. Springer, Heidelberg (2014)Google Scholar
  19. 19.
    Shimada, A., Kondo, K., Deguchi, D., Morin, G., Stern, H.: Kitchen scene context based gesture recognition: a contest in ICPR2012. In: Jiang, X., Bellon, O.R.P., Goldgof, D., Oishi, T. (eds.) WDIA 2012. LNCS, vol. 7854, pp. 168–185. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  20. 20.
    Iscen, A., Duygulu, P.: Knives are picked before slices are cut: recognition through activity sequence analysis. In: Proceedings of the 5th International Workshop on Multimedia for Cooking and Eating Activities, pp. 3–8 (2013)Google Scholar
  21. 21.
    Rohrbach, M., Amin, S., Andriluka, M., Schiele, B.: A database for fine grained activity detection of cooking activities. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1194–1201 (2012)Google Scholar
  22. 22.
    Packer, B., Saenko, K., Koller, D.: A combined pose, object, and feature model for action understanding. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1378–1385 (2012)Google Scholar
  23. 23.
    Lei, J., Ren, X., Fox, D.: Fine-grained kitchen activity recognition using RGB-D. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 208–211 (2012)Google Scholar
  24. 24.
    Hashimoto, A., Inoue, J., Nakamura, K., Funatomi, T., Ueda, M., Yamakata, Y., Minoh, M.: Recognizing ingredients at cutting process by integrating multimodal features. In: Proceedings of the ACM Multimedia 2012 Workshop on Multimedia for Cooking and Eating Activities, pp. 13–18 (2012)Google Scholar
  25. 25.
    Ueda, M., Funatomi, T., Hashimoto, A., Watanabe, T., Minoh, M.: Developing a real-time system for measuring the consumption of seasoning. In: Proceedings of IEEE ISM 2011 Workshop on Multimedia for Cooking and Eating Activities, pp. 393–398 (2011)Google Scholar
  26. 26.
    Hashimoto, A., Mori, N., Funatomi, T., Mukunoki, M., Kakusho, K., Minoh, M.: Tracking food materials with changing their appearance in food preparing. In: Proceedings of ISM 2010 Workshop on Multimedia for Cooking and Eating Activities, pp. 248–253. IEEE (2010)Google Scholar
  27. 27.
    Miyawaki, K., Sano, M.: A virtual agent for a cooking navigation system using augmented reality. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 97–103. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  28. 28.
    Yamasaki, T., Yoshino, K., Maeta, H., Sasada, T., Hashimoto, A., Funatomi, T., Yamakata, Y., Mori, S.: Procedual text generation from a flow graph. IPSJ J. 57(3) (to appear). Written in JapaneseGoogle Scholar
  29. 29.
    Kiddon, C., Ponnuraj, G.T., Zettlemoyer, L., Choi, Y.: Mise en place: unsupervised interpretation of instructional recipes. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 982–992 (2015)Google Scholar
  30. 30.
    Maeta, H., Sasada, T., Mori, S.: A framework for procedural text understanding. In: Proceedings of the 14th International Conference on Parsing Technologies (2015)Google Scholar
  31. 31.
    Karikome, S., Fujii, A.: Improving structural analysis of cooking recipe text. IEICE Tech. Rep. Data Eng. 112(75), 43–48 (2012)Google Scholar
  32. 32.
    Sato, A., Watanabe, K., Rekimoto, J.: Shadow cooking: situated guidance for a fluid cooking experience. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part III. LNCS, vol. 8515, pp. 558–566. Springer, Heidelberg (2014)Google Scholar
  33. 33.
    Matsushima, Y., Funabiki, N., Zhang, Y., Nakanishi, T., Watanabe, K.: Extensions of cooking guidance function on android tablet for homemade cooking assistance system. In: IEEE 2nd Global Conference on Consumer Electronics, pp. 397–401 (2013)Google Scholar
  34. 34.
    Halupka, V., Almahr, A., Pan, Y., Cheok, A.D.: Chop chop: a sound augmented kitchen prototype. In: Nijholt, A., Romão, T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 494–497. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  35. 35.
    Uriu, D., Namai, M., Tokuhisa, S., Kashiwagi, R., Inami, M., Okude, N.: Panavi: recipe medium with a sensors-embedded pan for domestic users to master professional culinary arts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 129–138 (2012)Google Scholar
  36. 36.
    Hamada, R., Okabe, J., Ide, I., Sakai, S., Tanaka, H.: Cooking navi: assistant for daily cooking in kitchen. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 371–374 (2005). Written in JapaneseGoogle Scholar
  37. 37.
    Bradbury, J.S., Shell, J.S., Knowles, C.B.: Hands on cooking: towards an attentive kitchen. In: Proceedings of CHI 2003 Extended Abstracts on Human Factors in Computing Systems, pp. 996–997 (2003)Google Scholar
  38. 38.
    Ju, W., Hurwitz, R., Judd, T., Lee, B.: Counteractive: an interactive cookbook for the kitchen counter. In: Proceedings of CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 269–270. ACM, New York (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Atsushi Hashimoto
    • 1
    Email author
  • Jun Harashima
    • 2
  • Yoko Yamakata
    • 3
  • Shinsuke Mori
    • 1
  1. 1.Kyoto UniversitySakyo-kuJapan
  2. 2.Cookpad IncShibuya-kuJapan
  3. 3.The University of TokyoBunkyo-kuJapan

Personalised recommendations