Vision Augmented Robot Feeding

  • Alexandre CandeiasEmail author
  • Travers Rhodes
  • Manuel Marques
  • João P. Costeira
  • Manuela Veloso
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11134)


Researchers have over time developed robotic feeding assistants to help at meals so that people with disabilities can live more autonomous lives. Current commercial feeding assistant robots acquire food without feedback on acquisition success and move to a preprogrammed location to deliver the food. In this work, we evaluate how vision can be used to improve both food acquisition and delivery. We show that using visual feedback on whether food was captured increases food acquisition efficiency. We also show how Discriminative Optimization (DO) can be used in tracking so that the food can be effectively brought all the way to the user’s mouth, rather than to a preprogrammed feeding location.


Assistive technologies Manipulation aids Computer vision Feeding assistance 



We very specially thank Henny Admoni for generously making available the arm robots in her HARP lab for our work. We specially thank Jayakorn Vongkulbhisal for the help with the implementation of DO. We thank the CMU-Portugal program for their partial support of this work through an ERI grant, as well as for supporting the visit of Alexandre Candeias to CMU, which led to this exciting and successful joint collaboration and work. This work was also funded by FCT grant [UID/EEA/50009/2013] and FCT project IF/00879/2012 (Projecto Exploratório). Finally, the authors thank the anonymous reviewers for their comments. The views and conclusions of this document are those of the authors only.

Supplementary material

Supplementary material 1 (mp4 23109 KB)


  1. 1.
    Al-Halimi, R.K., Moussa, M.: Performing complex tasks by users with upper-extremity disabilities using a 6-DOF robotic arm: a study. IEEE Trans. Neural Syst. Rehabil. Eng. 25(6), 686–693 (2017)CrossRefGoogle Scholar
  2. 2.
    Topping, M.: Early experience in the use of the ‘handy 1’ robotic aid to eating. Robotica 11(6), 525–527 (1993)CrossRefGoogle Scholar
  3. 3.
    Obi: Obi, robotic feeding device designed for home care. Accessed 19 May 2018
  4. 4.
    Camanio Care: Bestic. Accessed 09 May 2018
  5. 5.
    Performance Health: Meal buddy. Accessed 18 Feb 2018
  6. 6.
    Park, D., Kim, Y.K., Erickson, Z.M., Kemp, C.C.: Towards assistive feeding with a general-purpose mobile manipulator. CoRR abs/1605.07996 (2016)Google Scholar
  7. 7.
    Park, D., Kim, H., Hoshi, Y., Erickson, Z., Kapusta, A., Kemp, C.C.: A multimodal execution monitor with anomaly classification for robot-assisted feeding. In: 2016 IEEE International Conference on Robots and Systems (IROS) (2017)Google Scholar
  8. 8.
    Hawkins, K.P., Grice, P.M., Chen, T.L., King, C.H., Kemp, C.C.: Assistive mobile manipulation for self-care tasks around the head. In: IEEE SSCI 2014–2014 IEEE Symposium Series on Computational Intelligence - CIR2AT 2014: 2014 IEEE Symposium on Computational Intelligence in Robotic Rehabilitation and Assistive Technologies, Proceedings, pp. 16–25 (2014)Google Scholar
  9. 9.
    Perera, C.J., Lalitharatne, T.D., Kiguchi, K.: EEG-controlled meal assistance robot with camera-based automatic mouth position tracking and mouth open detection. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1760–1765. IEEE (2017)Google Scholar
  10. 10.
    Schröer, S., et al.: An autonomous robotic assistant for drinking. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 6482–6487. IEEE (2015)Google Scholar
  11. 11.
    Vongkulbhisal, J., De La Torre, F., Costeira, J.P.: Discriminative optimization: theory and applications to point cloud registration. In: Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 3975–3983, January 2017Google Scholar
  12. 12.
    Silva, C., Vongkulbhisal, J., Marques, M., Costeira, J.P., Veloso, M.: Feedbot - a robotic arm for autonomous assisted feeding. In: Oliveira, E., Gama, J., Vale, Z., Lopes Cardoso, H. (eds.) EPIA 2017. LNCS (LNAI), vol. 10423, pp. 486–497. Springer, Cham (2017). Scholar
  13. 13.
    Herlant, L.V.: Algorithms, implementation, and studies on eating with a shared control robot arm. Ph.D. dissertation, Carnegie Mellon University (2016)Google Scholar
  14. 14.
    Ragusa, F., Tomaselli, V., Furnari, A., Battiato, S., Farinella, G.M.: Food vs. non-food classification. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management - MADiMa 2016, pp. 77–81 (2016)Google Scholar
  15. 15.
    Besl, P.J., McKay, N.D.: A Method for Registration of 3-D Shapes (1992)Google Scholar
  16. 16.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, pp. I-511–I-518 (2001)Google Scholar
  17. 17.
    Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Process. Lett. 23(10), 1499–1503 (2016)CrossRefGoogle Scholar
  18. 18.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)CrossRefGoogle Scholar
  19. 19.
    Diankov, R.: Automated construction of robotic manipulation programs. Ph.D. thesis, Carnegie Mellon University, Robotics Institute, August 2010Google Scholar
  20. 20.
    Argall, B.D., Chernova, S., Veloso, M., Browning, B.: A survey of robot learning from demonstration. Robot. Auton. Syst. 57(5), 469–483 (2009)CrossRefGoogle Scholar
  21. 21.
    Bhattacharjee, T., Song, H., Lee, G., Srinivasa, S.S.: Food manipulation: a cadence of haptic signals. arXiv preprint arXiv:1804.08768 (2018)
  22. 22.
    Eggert, D.W., Lorusso, A., Fisher, R.B.: Estimating 3-D rigid body transformations: a comparison of four major algorithms. Mach. Vis. Appl. 9, 272–290 (1997)CrossRefGoogle Scholar
  23. 23.
    Daszykowski, M., Walczak, B.: Density-based clustering methods. Compr. Chemom. 2, 635–654 (2010)Google Scholar
  24. 24.
    Baltrusaitis, T., Robinson, P., Morency, L.P.: OpenFace: an open source facial behavior analysis toolkit. In: 2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.ISR - IST Universidade de LisboaLisbonPortugal
  2. 2.Carnegie Mellon UniversityPittsburghUSA

Personalised recommendations