Skip to main content
SpringerLink
Log in
Book cover

Advanced Interfacing Techniques for Sensors pp 269–290Cite as

Harnessing Vision and Touch for Compliant Robotic Interaction with Soft or Rigid Objects

  • Chapter
  • First Online:

  • 1683 Accesses

Part of the book series: Smart Sensors, Measurement and Instrumentation ((SSMI,volume 25))

Abstract

The chapter discusses recent research achievements related to sensing issues and interfacing techniques to enable safe interaction of commercial-grade robot manipulators with objects exhibiting rigid or soft surfaces. The main challenges are described, including the identification of proper combinations of vision and touch sensor technologies, and their placement and trajectory with respect to the objects of interest to enable safe navigation and close interaction. Various selective data acquisition procedures are also examined to ensure fast and sufficient monitoring of the interaction behaviour of the object under forces imposed by a robotic manipulator or a multi-finger gripper. Issues related to sensor calibration and data fusion are detailed. Potential solutions are presented in the context of various interaction tasks, including adaptive surface and contour following, object characteristics identification, and dexterous robot hand manipulation of soft objects using the Barrett hand. Numerous experiments demonstrate the validity of the proposed solutions.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. M. Seitz, Towards autonomous robotic servicing: using an integrated hand-arm-eye system for manipulating unknown objects. J. Robot. Auton. Syst. 26, 23–42 (1999)

    Article  Google Scholar 

  2. D. Henrich, H. Worn (eds.), Robot Manipulation of Deformable Objects (Springer, London, 2000)

    Google Scholar 

  3. M. Saadat, P. Nan, Industrial applications of automatic manipulation of flexible materials. Int. J. Ind. Robots 29, 434–442 (2002)

    Article  Google Scholar 

  4. F.F. Khalil, P. Payeur, Dexterous robotic manipulation of deformable objects with multi-sensory feedback—a review. Robot Manipulators, Trends and Development, ed. by A. Jimenez, B.M. Al Hadithi (Vukovar, Croatia, In-Tech, 2010), pp 587–619

    Google Scholar 

  5. L. Zaidi, B. Bouzgarrou, L. Sabourin, Y. Menzouar, Interaction modeling in the grasping and manipulation of 3D deformable objects, in Proceedings of IEEE International Conference on Advanced Robotics, Istanbul, Turkey (2015), pp. 504–509

    Google Scholar 

  6. A.-M. Cretu, P. Payeur, E.M. Petriu, Learning and prediction of soft object deformation using visual analysis of robot interactions, in Proceedings of International Symposium on Visual Computing, Las Vegas, Nevada, US, ed. by G. Bebis et al. LNCS 6454 (2010), pp. 232–241

    Google Scholar 

  7. M. Krainin, P. Henry, X. Ren, D. Fox, Manipulator and object tracking for in-hand 3D object modeling. Int. J. Robot. Res. 1311–1327 (2011)

    Google Scholar 

  8. D. Navarro-Alarcon, H.M. Yip, Z. Wang, Y.-H. Liu, F. Zhong, T. Zhang, P. Li, Automatic 3-D manipulation of soft object by robotic arms with an adaptive deformation model. IEEE Trans. Robot. 32(2), 429–441 (2016)

    Article  Google Scholar 

  9. M.-H. Choi, S.C. Wilber, M. Hong, Estimating material properties of deformable objects by considering global object behavior in video streams. Multimed. Tools Appl. 74, 3361–3375 (2015)

    Article  Google Scholar 

  10. A.S. Prabuwono, S. Said, R. Sulaiman, Performance evaluation of autonomous contour following algorithms for industrial robot, in Robot Manipulators Trends and Development, ed. by A. Jimenez, B.M. Al Hadithi. InTech (2010), pp 377–398

    Google Scholar 

  11. A. Petit, V. Lippiello, B. Siciliano, Real-time tracking of 3D elastic objects with an RGB-D sensor, in Proceedings of IEEE International Conference on Intelligent Robots and Systems, Hamburg, Germany (2015), pp. 3914–3921

    Google Scholar 

  12. L. Zaidi, B. Bouzgarrou, L. Sabourin, Y. Menzouar, Modeling and analysis of 3D deformable object grasping, in Proceedings of International Conference on Robotics in Alpe-Adria-Danube Region, Smolenice Castle, Slovakia (2014), pp. 504–509

    Google Scholar 

  13. S. Hirai, T. Tsuboi, T. Wada, Robust grasping manipulation of deformable objects, in Proceedings of IEEE Symposium on Assembly and Task Planning (2001), pp 411–416

    Google Scholar 

  14. J. Stuckler, S. Behnke, Perception of deformable objects and compliant manipulation for service robots, in Soft Robotics, ed. by A. Verl et al. (Springer, 2015), pp. 69–80

    Google Scholar 

  15. C.M. Mateo, P. Gil, D. Mira, F. Torres, Analysis of shapes to measure surfaces, in Proceedings of IEEE Conference on Informatics in Control, Automation and Robotics, Colmar (2012) pp. 60–65

    Google Scholar 

  16. C. Elbrechter, R. Haschke, H. Ritter, Folding paper with anthropomorphic robot hands using real-time physics-based modeling, in Proceedings of IEEE International Conference on Humanoid Robots, Osaka (2012), pp. 210–215

    Google Scholar 

  17. Q. Pan, G. Reitmayr, T. Drummond, ProFORMA: probabilistic feature based on-line rapid model acquisition, in Proceedings of 20th British Machine Vision Conference (BMVC), London (2009)

    Google Scholar 

  18. D. Kraft, N. Pugeault, E. Baseski, M. Popovic, D. Kragic, S. Kalkan, F. Worgotter, N. Kruger, Birth of the object: detection of objectness and extraction of object shape through object action complexes. Int. J. Hum. Robots 5(2), 247–265 (2008)

    Article  Google Scholar 

  19. J. Schulman, A. Lee, J. Ho, P. Abbeel, Tracking deformable objects with point clouds, in Proceedings of IEEE International Conference on Robotics and Automation (2013)

    Google Scholar 

  20. J. Hur, H. Lim, S.C. Ahn, 3D deformable spatial pyramid for dense 3D motion flow of deformable object, in Proceedings of International Symposium on Visual Computing, Las Vegas, Nevada, US, ed. by G. Bebis et al. LNCS 8887 (2014), pp. 118–127

    Google Scholar 

  21. D. Nakhaeinia, R. Fareh, P. Payeur, R. Laganière, Trajectory planning for surface following with manipulator under RGB-D visual guidance, in Proceedings of Safety, Security and Rescue Robotics, Linkoping (2013), pp. 1–6

    Google Scholar 

  22. F.F. Khalil, P. Payeur, A.-M. Cretu, Integrated multisensory robotic hand system for deformable object manipulation, in Proceedings of IASTED International Conference Robotics and Applications, Cambridge, Massachusetts, USA (2010), pp. 159–166

    Google Scholar 

  23. D. Nakhaeinia, P. Payeur, R. Laganière, Adaptive robotic contour following from low accuracy RGB-D surface profiling and visual servoing, in Proceedings of Canadian Conference on Computer and Robot Vision (2014), pp. 48–55

    Google Scholar 

  24. M. Zollhofer, et al., Real-time non-rigid reconstruction using and RGB-D camera. ACM Trans. Graph. 33(4), 156:1–156:12 (2014)

    Google Scholar 

  25. M. Dou, J. Taylor, H. Fuchs, A. Fitzgibbon, S. Izadi, 3D scanning deformable object with a single RGBD sensor, in Proceedings of IEEE Computer Vision and Pattern Recognition (2015), pp. 493–501

    Google Scholar 

  26. Microsoft Kinect Fusion. https://msdn.microsoft.com/en-us/library/dn188670.aspx

  27. Skanect. http://skanect.occipital.com/

  28. R. Macknojia, A. Chavez-Aragon, P. Payeur, R. Laganière, Calibration of a network of Kinect sensors for robotic inspection over a large workspace, in: Proceedings of IEEE Workshop on Robot Vision (2013), pp. 184–190

    Google Scholar 

  29. D. Nehab, P. Shilane, Stratified point sampling of 3D models, in Proceedings of Eurographics Symposium Point-Based Graphics, ed. by M. Alexa, S. Rusinkiewicz (2004), pp. 49–56

    Google Scholar 

  30. H. Song, H.-Y. Feng, A point cloud simplification algorithm for mechanical part inspection, in Information technology for balanced manufacturing systems, ed. by W. Shen (Springer, 2006), pp. 461–468

    Google Scholar 

  31. A.M. Cretu, Experimental data acquisition and modeling of 3D deformable objects, Ph.D. thesis, University of Ottawa (2009)

    Google Scholar 

  32. M. Chagnon-Forget, G. Rouhafzay, A.-M. Cretu, S. Bouchard, Enhanced visual-attention model for perceptually-improved 3d object modeling in virtual environments. 3D Res. 7(4), 1–18 (2016)

    Article  Google Scholar 

  33. K. Drewing, A. Ramisch, F. Bayer, Haptic, visual and visuo-haptic softness judgements for objects with deformable surfaces, in Proceedings of Eurohaptics Conference on Haptic Interfaces for Virtual Environments and Teleoperator Systems (2009), pp 640–645

    Google Scholar 

  34. Meshmixer. http://www.meshmixer.com/

  35. Y. Shi, W.C. Karl, A real-time algorithm for the approximation of level-set-based curve evolution. IEEE Trans. Image Process. 17(5), 645–656 (2008)

    Article  MathSciNet  Google Scholar 

  36. M. Muller, Information Retrieval for Music and Motion (Springer, 2007)

    Google Scholar 

  37. J.M. Ehrich, M. Flanders, J.F. Soechting, Factors influencing haptic perception of complex shapes. IEEE Trans. Haptics 1(1), 19–26 (2008)

    Article  Google Scholar 

  38. A.-M. Cretu, P. Payeur, E.M. Petriu, Soft object deformation monitoring and learning for model-based robotic hand manipulation. IEEE Trans. Syst. Man Cybern. Part B 42(3), 740–753 (2012)

    Google Scholar 

  39. B. Tawbe, A.-M. Cretu, Data-driven representation of soft deformable eobjects based on force-torque data and 3D vision measurements, in Proceedings of 3rd International Conference on Sensors and Applications (2016) (in press)

    Google Scholar 

  40. P. Cignoni, C. Rocchini, R. Scopigno, Metro: measuring error on simplified surfaces. Comput. Graph. Forum 17(2), 167–174 (1998)

    Article  Google Scholar 

  41. V. Laparra, J. Balle, A. Berardino, E.P. Simoncelli, Perceptual image quality assessment using a normalized Laplacian pyramid, in Proceedings of Human Vision and Electronic Imaging, vol. 16 (2016)

    Google Scholar 

  42. CloudCompare—3D Point Cloud and Mesh Processing Software. http://www.danielgm.net/cc/. Accessed 1 Aug 2016

  43. M.H. Raibert, J.J. Craig, Hybrid position/force control of manipulators. ASME J. Dyn. Syst. Meas. Control 102, 126–132 (1981)

    Article  Google Scholar 

  44. N. Hogan, Stable execution of contact tasks using impedance control, in Proceedings of IEEE International Conference on Robotics and Automation (1987), pp 1047–1054

    Google Scholar 

  45. G. Tao, Adaptive Control Design and Analysis (Wiley-IEEE Press, 2003)

    Google Scholar 

Download references

Acknowledgements

The authors wish to acknowledge the contribution of numerous graduate students and collaborators to the research projects summarized in this chapter, and more specifically: A. Chavez-Aragon, F. Hui, A. Huot, F.F. Khalil, P. Laferrière, R. Macknojia, D. Nakhaeinia, E.M. Petriu, B. Tawbe, and R. Toledo. This research has been supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada Foundation for Innovation (CFI), the Ontario Innovation Trust (OIT), the Ontario Centres of Excellence (OCE), and the Fonds de recherche du Québec - Nature et Technologies (FRQNT).

Author information

Authors and Affiliations

  1. Computer Science and Engineering Department, Université du Québec en Outaouais, Gatineau, Canada

    Ana-Maria Cretu

  2. School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada

    Pierre Payeur

Authors
  1. Ana-Maria Cretu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pierre Payeur
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ana-Maria Cretu .

Editor information

Editors and Affiliations

  1. Dept. of Electrical Engineering, IIT Madras, Chennai, Tamil Nadu, India

    Boby George

  2. ECE Dept., MCKV Institute of Engineering, Liluah, West Bengal, India

    Joyanta Kumar Roy

  3. Dept. of Electrical Engineering, IIT Madras , Chennai, Tamil Nadu, India

    V. Jagadeesh Kumar

  4. Engineering, Macquarie University , Sydney, New South Wales, Australia

    Subhas Chandra Mukhopadhyay

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Cretu, AM., Payeur, P. (2017). Harnessing Vision and Touch for Compliant Robotic Interaction with Soft or Rigid Objects. In: George, B., Roy, J., Kumar, V., Mukhopadhyay, S. (eds) Advanced Interfacing Techniques for Sensors . Smart Sensors, Measurement and Instrumentation, vol 25. Springer, Cham. https://doi.org/10.1007/978-3-319-55369-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-55369-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-55368-9

  • Online ISBN: 978-3-319-55369-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation