Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 6, pp 6607–6628 | Cite as

Robust and real-time pose tracking for augmented reality on mobile devices

  • Xin YangEmail author
  • Jiabin Guo
  • Tangli Xue
  • Kwang-Ting (Tim) Cheng
Article

Abstract

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.

Keywords

Feature tracking Featureless tracking Inertial sensing Sensor fusion Kalman filtering Smartphones 

Notes

Acknowledgements

This work is funded by the National Science Foundation of China (grant no. 61502188).

Supplementary material

(MP4 76.2 MB)

References

  1. 1.
    Chliveros G, Pateraki M, Trahanias P (2013) Robust multi-hypothesis 3d object pose tracking International conference on computer vision systems. Springer, pp 234–243Google Scholar
  2. 2.
    Chum O, Matas J (2005) Matching with prosac-progressive sample consensus 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1. IEEE, pp 220–226Google Scholar
  3. 3.
    Crivellaro A, Rad M, Verdie Y, Moo Yi K, Fua P, Lepetit V (2015) A novel representation of parts for accurate 3d object detection and tracking in monocular images Proceedings of the IEEE international conference on computer vision, pp 4391–4399Google Scholar
  4. 4.
    Daniel W, Dieter S (2007) Artoolkitplus for pose tracking on mobile devices Proceedings of 12th computer vision winter workshopGoogle Scholar
  5. 5.
    DiVerdi S, Hollerer T (2007) Groundcam: a tracking modality for mobile mixed reality 2007 IEEE virtual reality conference. IEEE, pp 75–82Google Scholar
  6. 6.
    Engel J, Schöps T, Cremers D (2014) Lsd-slam: large-scale direct monocular slam European conference on computer vision. Springer, pp 834–849Google Scholar
  7. 7.
    Engel J, Sturm J, Cremers D (2013) Semi-dense visual odometry for a monocular camera Proceedings of the IEEE international conference on computer vision, pp 1449–1456Google Scholar
  8. 8.
    Fiala M (2010) Designing highly reliable fiducial markers. IEEE Trans Pattern Anal Mach Intell 32(7):1317–1324CrossRefGoogle Scholar
  9. 9.
    Hallaway D, Feiner S, Höllerer T (2004) Bridging the gaps: hybrid tracking for adaptive mobile augmented reality. Appl Artif Intell 18(6):477–500CrossRefGoogle Scholar
  10. 10.
    Hare S, Saffari A, Torr PH (2012) Efficient online structured output learning for keypoint-based object tracking IEEE conference on computer vision and pattern recognition (CVPR), 2012. IEEE, pp 1894–1901Google Scholar
  11. 11.
    Ibañez AS, Figueras JP (2013) Vuforia v1. 5 sdk: analysis and evaluation of capabilities. Ph.D. thesis, Master Thesis, Universitat Politecnica De CatalunyaGoogle Scholar
  12. 12.
    Kato H, Billinghurst M (1999) Marker tracking and hmd calibration for a video-based augmented reality conferencing system 2nd IEEE and ACM international workshop on augmented reality, 1999.(IWAR’99) proceedings. IEEE, pp 85–94Google Scholar
  13. 13.
    Klein G, Murray D (2007) Parallel tracking and mapping for small ar workspaces 6th IEEE and ACM international symposium on mixed and augmented reality, 2007. ISMAR 2007. IEEE, pp 225–234Google Scholar
  14. 14.
    Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone 8th IEEE international symposium on mixed and augmented reality, 2009. ISMAR 2009. IEEE, pp 83–86Google Scholar
  15. 15.
    Klein G, Murray DW (2006) Full-3d edge tracking with a particle filter BMVC, pp 1119–1128Google Scholar
  16. 16.
    Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Trans Pattern Anal Mach Intell 28(9):1465–1479CrossRefGoogle Scholar
  17. 17.
    Li M, Kim BH, Mourikis AI (2013) Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera Robotics and automation (ICRA), 2013 IEEE international conference on. IEEE, pp 4712–4719Google Scholar
  18. 18.
    Ligorio G, Sabatini AM (2013) Extended kalman filter-based methods for pose estimation using visual, inertial and magnetic sensors: comparative analysis and performance evaluation. Sensors 13(2):1919–1941CrossRefGoogle Scholar
  19. 19.
    Morgan DL (2014) Improved orientation estimation for smart phone indoor localization. Ph.D. thesis, UNIVERSITY OF CALIFORNIA SANTA BARBARAGoogle Scholar
  20. 20.
    Obeidy WK, Arshad H, Chowdhury SA, Parhizkar B, Huang J (2013) Increasing the tracking efficiency of mobile augmented reality using a hybrid tracking technique International visual informatics conference. Springer, pp 447–457Google Scholar
  21. 21.
    Ozuysal M, Calonder M, Lepetit V, Fua P (2010) Fast keypoint recognition using random ferns. IEEE Trans Pattern Anal Mach Intell 32(3):448–461CrossRefGoogle Scholar
  22. 22.
    Payet N, Todorovic S (2011) From contours to 3d object detection and pose estimation 2011 International conference on computer vision. IEEE, pp 983–990Google Scholar
  23. 23.
    Prisacariu VA, Reid ID (2012) Pwp3d: real-time segmentation and tracking of 3d objects. Int J Comput Vis 98(3):335–354MathSciNetCrossRefGoogle Scholar
  24. 24.
    Prisacariu VA, Segal AV, Reid I (2012) Simultaneous monocular 2d segmentation, 3d pose recovery and 3d reconstruction Asian conference on computer vision. Springer, pp 593–606Google Scholar
  25. 25.
    Rolland JP, Davis L, Baillot Y (2001) A survey of tracking technology for virtual environments. Fundamentals of Wearable Computers and Augmented Reality 1 (1):67–112Google Scholar
  26. 26.
    Rosten E, Porter R, Drummond T (2010) Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119CrossRefGoogle Scholar
  27. 27.
    Rublee E, Rabaud V, Konolige K, Bradski G (2011) Orb: an efficient alternative to sift or surf 2011 International conference on computer vision. IEEE, pp 2564–2571Google Scholar
  28. 28.
    Schöps T, Engel J, Cremers D (2014) Semi-dense visual odometry for ar on a smartphone IEEE international symposium on mixed and augmented reality (ISMAR), 2014. IEEE, pp 145–150Google Scholar
  29. 29.
    Wagner D, Langlotz T, Schmalstieg D (2008) Robust and unobtrusive marker tracking on mobile phones 7th IEEE/ACM international symposium on mixed and augmented reality, 2008. ISMAR 2008. IEEE, pp 121–124Google Scholar
  30. 30.
    Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose tracking from natural features on mobile phones Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality. IEEE Computer Society, pp 125–134Google Scholar
  31. 31.
    Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2010) Real-time detection and tracking for augmented reality on mobile phones. IEEE Trans Vis Comput Graph 16(3):355–368CrossRefGoogle Scholar
  32. 32.
    White S, Feiner S, Kopylec J (2006) Virtual vouchers: prototyping a mobile augmented reality user interface for botanical species identification 3D user interfaces (3DUI’06). IEEE, pp 119– 126Google Scholar
  33. 33.
    Yang X, Cheng KTT (2014) Local difference binary for ultrafast and distinctive feature description. IEEE Trans Pattern Anal Mach Intell 36(1):188–194CrossRefGoogle Scholar
  34. 34.
    Yang X, Huang C, Cheng KTT (2014) libldb: a library for extracting ultrafast and distinctive binary feature description Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 671–674Google Scholar
  35. 35.
    Yang X, Si X, Xue T, Cheng KTT (2015) [Poster] Fusion of vision and inertial sensing for accurate and efficient pose tracking on smartphones IEEE international symposium on mixed and augmented reality (ISMAR), 2015. IEEE, pp 68–71Google Scholar
  36. 36.
    Yang X, Si X, Xue T, Zhang L, Cheng KTT (2015) Vision-inertial hybrid tracking for robust and efficient augmented reality on smartphones Proceedings of the 23rd ACM international conference on multimedia. ACM, pp 1039–1042Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Xin Yang
    • 1
    Email author
  • Jiabin Guo
    • 1
  • Tangli Xue
    • 1
  • Kwang-Ting (Tim) Cheng
    • 2
    • 3
  1. 1.School of Electronics Information and CommunicationsHuazhong University of Science and TechnologyWuhanChina
  2. 2.Department of Electrical and Computer EngineeringUniversity of CaliforniaSanta BarbaraUSA
  3. 3.Department of Computer Science and EngineeringHong Kong University of Science and TechnologyHong KongChina

Personalised recommendations