Advertisement

Dealing with Ambiguity in Robotic Grasping via Multiple Predictions

  • Ghazal GhazaeiEmail author
  • Iro Laina
  • Christian Rupprecht
  • Federico Tombari
  • Nassir Navab
  • Kianoush Nazarpour
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11364)

Abstract

Humans excel in grasping and manipulating objects because of their life-long experience and knowledge about the 3D shape and weight distribution of objects. However, the lack of such intuition in robots makes robotic grasping an exceptionally challenging task. There are often several equally viable options of grasping an object. However, this ambiguity is not modeled in conventional systems that estimate a single, optimal grasp position. We propose to tackle this problem by simultaneously estimating multiple grasp poses from a single RGB image of the target object. Further, we reformulate the problem of robotic grasping by replacing conventional grasp rectangles with grasp belief maps, which hold more precise location information than a rectangle and account for the uncertainty inherent to the task. We augment a fully convolutional neural network with a multiple hypothesis prediction model that predicts a set of grasp hypotheses in under 60 ms, which is critical for real-time robotic applications. The grasp detection accuracy reaches over \(90\%\) for unseen objects, outperforming the current state of the art on this task.

Keywords

Robotic grasping Deep learning Multiple hypotheses 

Notes

Acknowledgments

This work is supported by UK Engineering and Physical Sciences Research Council (EP/R004242/1). We also gratefully acknowledge the support of NVIDIA Corporation with the donation of a Titan Xp GPU used for the experiments.

References

  1. 1.
    Asif, U., Bennamoun, M., Sohel, F.A.: RGB-D object recognition and grasp detection using hierarchical cascaded forests. IEEE Trans. Rob. 33(3), 547–564 (2017)CrossRefGoogle Scholar
  2. 2.
    Belagiannis, V., Zisserman, A.: Recurrent human pose estimation. In: International Conference on Automatic Face & Gesture Recognition (FG 2017) (2017)Google Scholar
  3. 3.
    Bicchi, A., Kumar, V.: Robotic grasping and contact: a review. In: Proceedings of 2000 IEEE International Conference on Robotics and Automation (ICRA), vol. 1, pp. 348–353. IEEE (2000)Google Scholar
  4. 4.
    Bulat, A., Tzimiropoulos, G.: Human pose estimation via convolutional part heatmap regression. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 717–732. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46478-7_44CrossRefGoogle Scholar
  5. 5.
    Bulat, A., Tzimiropoulos, G.: Super-fan: integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  6. 6.
    Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  7. 7.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B (Methodol.) (1977)Google Scholar
  8. 8.
    Du, X., et al.: Articulated multi-instrument 2D pose estimation using fully convolutional networks. IEEE Trans. Med. Imaging (2018)Google Scholar
  9. 9.
    Guo, D., Sun, F., Liu, H., Kong, T., Fang, B., Xi, N.: A hybrid deep architecture for robotic grasp detection. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2017)Google Scholar
  10. 10.
    Guzman-Rivera, A., et al.: Multi-output learning for camera relocalization. In: Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  11. 11.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  12. 12.
    Jiang, Y., Moseson, S., Saxena, A.: Efficient grasping from RGB-D images: learning using a new rectangle representation. In: International Conference on Robotics and Automation (ICRA). IEEE (2011)Google Scholar
  13. 13.
    Kehoe, B., Patil, S., Abbeel, P., Goldberg, K.: A survey of research on cloud robotics and automationGoogle Scholar
  14. 14.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  15. 15.
    Kumra, S., Kanan, C.: Robotic grasp detection using deep convolutional neural networks. arXiv preprint arXiv:1611.08036 (2016)
  16. 16.
    Laina, I., et al.: Concurrent segmentation and localization for tracking of surgical instruments. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 664–672. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66185-8_75CrossRefGoogle Scholar
  17. 17.
    Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., Navab, N.: Deeper depth prediction with fully convolutional residual networks. In: 2016 Fourth International Conference on 3D Vision (3DV). IEEE (2016)Google Scholar
  18. 18.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE (1998)Google Scholar
  19. 19.
    Lee, S., Prakash, S.P.S., Cogswell, M., Ranjan, V., Crandall, D., Batra, D.: Stochastic multiple choice learning for training diverse deep ensembles. In: Advances in Neural Information Processing Systems (2016)Google Scholar
  20. 20.
    Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Rob. Res. (2015)Google Scholar
  21. 21.
    Levine, S., Pastor, P., Krizhevsky, A., Quillen, D.: Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. arXiv preprint arXiv:1603.02199 (2016)
  22. 22.
    Li, Z., Chen, Q., Koltun, V.: Interactive image segmentation with latent diversity. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2018)Google Scholar
  23. 23.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  24. 24.
    Mahler, J., et al.: Dex-net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312 (2017)
  25. 25.
    McLachlan, G., Peel, D.: Finite Mixture Models. Wiley, Hoboken (2004)zbMATHGoogle Scholar
  26. 26.
    Merget, D., Rock, M., Rigoll, G.: Robust facial landmark detection via a fully-convolutional local-global context network. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  27. 27.
    Miller, A.T., Allen, P.K.: Graspit! a versatile simulator for robotic grasping. IEEE Rob. Autom. Mag. (2004)Google Scholar
  28. 28.
    Papandreou, G., et al.: Towards accurate multi-person pose estimation in the wild. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  29. 29.
    Redmon, J., Angelova, A.: Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2015)Google Scholar
  30. 30.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  31. 31.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (2015)Google Scholar
  32. 32.
    Rochan, M., Ye, L., Wang, Y.: Video summarization using fully convolutional sequence networks. arXiv preprint arXiv:1805.10538 (2018)
  33. 33.
    Rupprecht, C., et al.: Learning in an uncertain world: representing ambiguity through multiple hypotheses. In: International Conference on Computer Vision (ICCV) (2017)Google Scholar
  34. 34.
    Saxena, A., Driemeyer, J., Ng, A.Y.: Robotic grasping of novel objects using vision. Int. J. Rob. Res. (2008)Google Scholar
  35. 35.
    Varley, J., DeChant, C., Richardson, A., Nair, A., Ruales, J., Allen, P.: Shape completion enabled robotic grasping. arXiv preprint arXiv:1609.08546 (2016)
  36. 36.
    Vedaldi, A., Lenc, K.: MatConvNet - convolutional neural networks for MATLAB. In: Proceeding of the ACM International Conference on Multimedia (2015)Google Scholar
  37. 37.
    Viereck, U., Pas, A., Saenko, K., Platt, R.: Learning a visuomotor controller for real world robotic grasping using simulated depth images. In: Conference on Robot Learning (2017)Google Scholar
  38. 38.
    Wang, Z., Li, Z., Wang, B., Liu, H.: Robot grasp detection using multimodal deep convolutional neural networks. Adv. Mech. Eng. (2016)Google Scholar
  39. 39.
    Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y.: Convolutional pose machines. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  40. 40.
    Zapata-Impata, B.S.: Using geometry to detect grasping points on 3D unknown point cloud. In: International Conference on Informatics in Control, Automation and Robotics (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ghazal Ghazaei
    • 1
    • 2
    Email author
  • Iro Laina
    • 2
  • Christian Rupprecht
    • 2
  • Federico Tombari
    • 2
  • Nassir Navab
    • 2
  • Kianoush Nazarpour
    • 1
    • 3
  1. 1.School of EngineeringNewcastle UniversityNewcastleUK
  2. 2.Technische Universität MünchenMunichGermany
  3. 3.Institute of NeuroscienceNewcastle UniversityNewcastleUK

Personalised recommendations