RGB-D Based Tracking of Complex Objects

  • Alejandro Perez-Yus
  • Luis Puig
  • Gonzalo Lopez-Nicolas
  • Jose J. Guerrero
  • Dieter Fox
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10188)


Tracking the pose of objects is a relevant topic in computer vision, which potentially allows to recover meaningful information for other applications such as task supervision, robot manipulation or activity recognition. In the last years, RGB-D cameras have been widely adopted for this problem with impressive results. However, there are certain objects whose surface properties or complex shapes prevents the depth sensor from returning good depth measurements, and only color-based methods can be applied. In this work, we show how the depth information of the surroundings of the object can still be useful in the object pose tracking with RGB-D even in this situation. Specifically, we propose using the depth information to handle occlusions in a state of the art region-based object pose tracking algorithm. Experiments with recordings of humans naturally interacting with difficult objects have been performed, showing the advantages of our contribution in several image sequences.



This work was supported by Projects DPI2014-61792-EXP and DPI2015-65962-R (MINECO/FEDER, UE) and grant BES-2013-065834 (MINECO).


  1. 1.
    Asif, U., Bennamoun, M., Sohel, F.: Real-time pose estimation of rigid objects using RGB-D imagery. In: IEEE Conference on Industrial Electronics and Applications, pp. 1692–1699 (2013)Google Scholar
  2. 2.
    Azad, P., Asfour, T., Dillmann, R.: Combining appearance-based and model-based methods for real-time object recognition and 6D localization. In: IEEE International Conference on Intelligent Robots and Systems, pp. 5339–5344 (2006)Google Scholar
  3. 3.
    Azad, P., Munch, D., Asfour, T., Dillmann, R.: 6-DoF model-based tracking of arbitrarily shaped 3D objects. In: IEEE International Conference on Robotics and Automation, pp. 5204–5209 (2011)Google Scholar
  4. 4.
    Bibby, C., Reid, I.: Robust real-time visual tracking using pixel-wise posteriors. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 831–844. Springer, Heidelberg (2008). Scholar
  5. 5.
    Seo, B.-K., Park, H., Park, J.-I., Hinterstoisser, S., Ilic, S.: Optimal local searching for fast and robust textureless 3d object tracking in highly cluttered backgrounds. IEEE Trans. Visual. Comput. Graph. 20(1), 99–110 (2014)CrossRefGoogle Scholar
  6. 6.
    Choi, C., Christensen, H.I.: Robust 3D visual tracking using particle filtering on the SE(3) group. In: IEEE International Conference on Robotics and Automation, vol. 31, pp. 4384–4390 (2011)Google Scholar
  7. 7.
    Choi, C., Christensen, H.I.: 3D textureless object detection and tracking: an edge-based approach. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3877–3884 (2012)Google Scholar
  8. 8.
    Dambreville, S., Sandhu, R., Yezzi, A., Tannenbaum, A.: Robust 3D pose estimation and efficient 2D region-based segmentation from a 3D shape prior. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5303, pp. 169–182. Springer, Heidelberg (2008). Scholar
  9. 9.
    Drummond, T., Cipolla, R.: Real-time visual tracking of complex structures. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 932–946 (2002)CrossRefGoogle Scholar
  10. 10.
    Grundmann, T., Eidenberger, R., Schneider, M., Fiegert, M., Wichert, G.: Robust high precision 6D pose determination in complex environments for robotic manipulation. In: Workshop Best Practice in 3D Perception and Modeling for Mobile Manipulation at the IEEE International Conference of Robotics and Automation (2010)Google Scholar
  11. 11.
    Grundmann, T., Fiegert, M., Burgard, W.: Probabilistic rule set joint state update as approximation to the full joint state estimation applied to multi object scene analysis. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2047–2052 (2010)Google Scholar
  12. 12.
    Hinterstoisser, S., Holzer, S., Cagniart, C., Ilic, S., Konolige, K., Navab, N., Lepetit, V.: Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In: IEEE International Conference on Computer Vision, pp. 858–865 (2011)Google Scholar
  13. 13.
    Imperoli, M., Pretto, A.: D\(^{2}\)CO: fast and robust registration of 3D textureless objects using the directional chamfer distance. In: Nalpantidis, L., Krüger, V., Eklundh, J.-O., Gasteratos, A. (eds.) ICVS 2015. LNCS, vol. 9163, pp. 316–328. Springer, Cham (2015). Scholar
  14. 14.
    Kim, K., Lepetit, V., Woo, W.: Keyframe-based modeling and tracking of multiple 3D objects. In: IEEE International Symposium on Mixed and Augmented Reality, pp. 193–198. IEEE (2010)Google Scholar
  15. 15.
    Klank, U., Carton, D., Beetz, M.: Transparent object detection and reconstruction on a mobile platform. In: IEEE International Conference on Robotics and Automation, pp. 5971–5978 (2011)Google Scholar
  16. 16.
    Lepetit, V., Fua, P.: Monocular Model-Based 3D Tracking of Rigid Objects: A Survey. Now Publishers Inc., Hanover (2005)Google Scholar
  17. 17.
    Liu, M.Y., Tuzel, O., Veeraraghavan, A., Chellappa, R.: Fast directional chamfer matching. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1696–1703 (2010)Google Scholar
  18. 18.
    Lowrey, K., Kolev, S., Tassa, Y., Erez, T., Todorov, E.: Physically-consistent sensor fusion in contact-rich behaviors. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1656–1662 (2014)Google Scholar
  19. 19.
    Lysenkov, I., Eruhimov, V., Bradski, G.: Recognition and Pose Estimation of Rigid Transparent Objects with a Kinect Sensor. In: Robotics: Science and Systems (2012)Google Scholar
  20. 20.
    Phillips, C.J., Derpanis, K.G., Daniilidis, K.: A novel stereoscopic cue for figure-ground segregation of semi-transparent objects. In: IEEE International Conference on Computer Vision Workshops, vol. 1, pp. 1100–1107 (2011)Google Scholar
  21. 21.
    Prisacariu, V.A., Reid, I.D.: PWP3D: Real-time segmentation and tracking of 3D objects. Int. J. Comput. Vis. 98(3), 335–354 (2012)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Rosenhahn, B., Brox, T., Weickert, J.: Three-dimensional shape knowledge for joint image segmentation and pose tracking. Int. J. Comput. Vis. 73(3), 243–262 (2007)CrossRefGoogle Scholar
  23. 23.
    Schmidt, T., Hertkorn, K., Newcombe, R., Marton, Z., Suppa, M., Fox, D.: Depth-based tracking with physical constraints for robot manipulation. In: IEEE International Conference on Robotics and Automation., pp. 119–126 (2015)Google Scholar
  24. 24.
    Schmidt, T., Newcombe, R., Fox, D.: DART: dense articulated real-time tracking with consumer depth cameras. Auton. Robot. 39(3), 239–258 (2015)CrossRefGoogle Scholar
  25. 25.
    Schulman, J., Lee, A., Ho, J., Abbeel, P., Berkeley, U.C.: Tracking deformable objects with point clouds. In: IEEE International Conference on Robotics and Automation, pp. 1122–1129 (2013)Google Scholar
  26. 26.
    Ulrich, M., Wiedemann, C., Steger, C.: CAD-based recognition of 3D objects in monocular images. In: IEEE International Conference on Robotics and Automation, pp. 1191–1198 (2009)Google Scholar
  27. 27.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. (CSUR) 38(4) (2006)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Alejandro Perez-Yus
    • 1
  • Luis Puig
    • 2
  • Gonzalo Lopez-Nicolas
    • 1
  • Jose J. Guerrero
    • 1
  • Dieter Fox
    • 2
  1. 1.Instituto de Investigación en Ingeniería de Aragón (I3A)Universidad de ZaragozaZaragozaSpain
  2. 2.University of WashingtonSeattleUSA

Personalised recommendations