Advertisement

Virtual Volumetric Graphics on Commodity Displays Using 3D Viewer Tracking

Abstract

Three dimensional (3D) displays typically rely on stereo disparity, requiring specialized hardware to be worn or embedded in the display. We present a novel 3D graphics display system for volumetric scene visualization using only standard 2D display hardware and a pair of calibrated web cameras. Our computer vision-based system requires no worn or other special hardware. Rather than producing the depth illusion through disparity, we deliver a full volumetric 3D visualization—enabling users to interactively explore 3D scenes by varying their viewing position and angle according to the tracked 3D position of their face and eyes. We incorporate a novel wand-based calibration that allows the cameras to be placed at arbitrary positions and orientations relative to the display. The resulting system operates at real-time speeds (∼25 fps) with low latency (120–225 ms) delivering a compelling natural user interface and immersive experience for 3D viewing. In addition to objective evaluation of display stability and responsiveness, we report on user trials comparing users’ timings on a spatial orientation task.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 199

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Notes

  1. 1.

    A video demo of the system is available in the Supplementary Material.

References

  1. Alnowami, M., Alnwaimi, B., Copland, M., & Wells, K. (2011). A quantitative assessment of using the Kinect for Xbox 360 for respiratory surface motion tracking. In Proc. SPIE medical imaging.

  2. Brar, L., Sexton, I., Surman, P., Bates, R., Lee, W., Hopf, K., Neumann, F., Day, S., & Williman, E. (2010). Laser-based head-tracked 3D display research. Journal of Display Technology, 6(10), 531–543.

  3. Chen, C., Huang, Y., Chuang, S., Wu, C., Shieh, H., Mphepo, W., Hsieh, C., & Hsu, S. (2009). Liquid crystal panel for high efficiency barrier type autostereoscopic 3D displays. Applied Optics, 48(18), 3446–3454.

  4. Dang, T., Hoffmann, C., & Stiller, C. (2009). Continuous stereo self-calibration by camera parameter tracking. IEEE Transactions on Image Processing, 18(7), 1536–1549.

  5. Dodgson, N. (2004). Variation and extrema of human interpupillary distance. Proceedings of SPIE, 5291, 36–46.

  6. Ellis, S. R., Wolfram, A., & Adelstein, B. D. (2002). Three dimensional tracking in augmented environments: user performance trade-offs between system latency and update rate. Proceedings of the Human Factors and Ergonomics Society annual meeting, 46(26), 2149–2153.

  7. Erden, E., Kishore, V., Urey, H., Baghsiahi, H., Willman, E., Day, S., Selviah, D., Fernandez, F., & Surman, P. (2009). Laser scanning based autostereoscopic 3D display with pupil tracking. In Proc. IEEE photonics (pp. 10–11).

  8. Ezra, D., Woodgate, G., Omar, B., Holliman, N., Harrold, J., & Shapiro, L. (1995). New autostereoscopic display system. In Proc. SPIE Intl. Society of Optical Engineering (pp. 31–40).

  9. Free2C (2010). The free2c desktop display (Technical report). Heinrich Hertz Institute.

  10. Freund, Y., & Schapire, R. (1999). A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence, 14(7), 771–780.

  11. Lee, J. (2008). Head tracking for desktop VR displays using the WiiRemote (Technical report). Carnegie Mellon University.

  12. Malleson, C., & Collomosse, J. (2011). Volumetric 3D graphics on commodity displays using active gaze tracking. In Proc. ICCV workshop on human computer interaction.

  13. Nishimura, H., Abe, T., Yamamoto, H., Hayasaki, Y., Nagai, Y., Shimizu, Y., & Nishida, N. (2007). Development of a 140-inch autostereoscopic display by use of full-color LED panel. Proceedings of SPIE, the International Society for Optical Engineering, 6486, 64861B.

  14. OpenCV. Open source computer vision library. Accessed July 2011.

  15. Perlin, K., Poultney, C., Kollin, J., Kristjansson, D., & Paxia, S. (2001). Recent advances in the NYU autostereoscopic display. Proceedings of SPIE, the International Society for Optical Engineering, 4297, 196–203.

  16. Sandin, D., Margolis, T., Dawe, G., Leigh, J., & DeFanti, T. (2001). Varrier autostereographic display. Proceedings of SPIE, the International Society for Optical Engineering, 4297, 204–211.

  17. Schwartz, A. (1985). Head tracking stereoscopic display. In Proc. IEEE intl. conf. on display research (pp. 141–144).

  18. Sorensen, S., Hansen, P., & Sorensen, N. (2004). Method for recording and viewing stereoscopic images in color using monochrome filters. U.S. Patent 6687003.

  19. Surman, P., Sexton, I., Hopf, K., Lee, W., Buckley, E., Jones, G., & Bates, R. (2008a). European research into head tracked autostereoscopic displays. In Proc. conf on 3DTV (pp. 161–164).

  20. Surman, P., Sexton, I., Hopf, K., Lee, W., Neumann, F., Buckley, E., Jones, G., Corbett, A., Bates, R., & Talukdar, S. (2008b). European research into head tracked autostereoscopic displays. Journal of the Society for Information Display, 16, 743–753.

  21. Takaki, Y. (2006). High-density directional display for generating natural 3D images. Proceedings of the IEEE, 94(3), 654–663.

  22. Tetsutani, N., Ichinose, S., & Ishibashi, M. (1989). 3D-TV projection display system with head-tracking. In Japan Display (pp. 56–59).

  23. Tetsutani, N., Omura, K., & Kishino, F. (1994). Study on a stereoscopic display system employing eye-position tracking for multi-viewers. Proceedings of SPIE, the International Society for Optical Engineering, 2177, 135.

  24. Thacker, N. A. (1992). Online calibration of a 4 DOF stereo head. In Proc. British machine vision conference (BMVC) (pp. 528–537).

  25. Tsai, R., Tsai, C., Lee, K., Wu, C., Lin, L., Huang, K., Hsu, W., Wu, C., Lu, C., Yang, J., & Chen, Y. (2009). Challenge of 3D LCD displays. Proceedings of SPIE, the International Society for Optical Engineering, 7329, 732903.

  26. Urey, H., & Erden, E. (2011). State of the art in stereoscopic and autostereoscopic displays. Proceedings of the IEEE, 99(4), 544–555.

  27. Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. In Proc. computer vision and pattern recognition.

  28. Welch, B. L. (1947). The generalization of student’s problem when several different population variances are involved. Biometrika, 34(1–2), 28–35. doi:10.1093/biomet/34.1-2.28.

  29. Woodgate, G., Ezra, D., Harrold, J., Holliman, N., Jones, G., & Moseley, R. (1997). Observer-tracking autostereoscopic 3D display systems. Proceedings of SPIE, the International Society for Optical Engineering, 3012, 187–198.

  30. Woodgate, G., Harrold, J., Jacobs, A., Mosely, R., & Ezra, D. (2000). Flat-panel autostereoscopic displays: characterization and enhancement. Proceedings of SPIE, the International Society for Optical Engineering, 3957, 153–164.

  31. Woods, A. (2009). 3D displays in the home. Information Display, 7, 8–12.

  32. Yamamoto, H., Kouno, M., Muguruma, S., Hayasaki, Y., Nagai, Y., Shimizu, Y., & Nishida, N. (2002). Enlargement of viewing area of stereoscopic full-color LED display using parallax barrier combined with polarizer. Applied Optics, 41(32), 6907–6919.

Download references

Author information

Correspondence to John Collomosse.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(AVI 6.2 MB)

(AVI 6.2 MB)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Malleson, C., Collomosse, J. Virtual Volumetric Graphics on Commodity Displays Using 3D Viewer Tracking. Int J Comput Vis 101, 519–532 (2013) doi:10.1007/s11263-012-0533-8

Download citation

Keywords

  • Volumetric display
  • 3D graphics
  • Tracking
  • Kalman filter
  • Camera Calibration