Abstract
Integrated photography display is a commonly used technology to provide users 3D videos with horizontal and vertical parallaxes. However, 3D video contents under natural scenes are very scarce and many existing methods can only provide parallaxes in one direction using binocular cameras. In this work, we set up an end-to-end real-time virtual view synthesis system for integrated photography display, in which 64 views (4 real views and 60 virtual views) can be synthesized using only 4 cameras placed in a rectangle. This paper mainly makes two contributions. Firstly, we propose a new method of polar alignment for two-dimensional camera placement (\(2\times 2\)), which aligns horizontal and vertical polar lines at the same time and preserves the effective area of the final image as large as possible after clipping. Secondly, in the process of virtual view synthesis, we introduce an accelerated strategy by combining CPU thread-level parallelism and GPU data-level parallelism to make full use of the computing resources. Experimental results show that our system can capture 4-way 480P video sequences under natural scenes, synthesize 64-way views and render the resulting 4K video combined from 64-way views on a 3D monitor in real-time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Smolic, A., Kauff, P., Knorr, S., Hornung, A., Kunter, M., Muller, M., Lang, M.: Three-dimensional video postproduction and processing. Proc. IEEE 99(4), 607–625 (2011)
Smolic, A., Muller, K., Dix, K., Merkle, P., Kauff, P., Wiegand, T.: Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. In: 15th IEEE International Conference on Image Processing (ICIP). IEEE, San Diego, pp. 2448–2451 (2008)
Fehn, C.: Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In: Stereoscopic Displays and Virtual Reality Systems XI, San Jose, pp. 93–105, May 2004
Lang, M., Hornung, A., Wang, O., Poulakos, S., Smolic, A., Gross, M.: Nonlinear disparity mapping for stereoscopic 3D. In: ACM, Los Angeles, vol. 29, No. 4, p. 75, July 2010
Farre, M., Wang, O., Lang, M., Stefanoski, N., Hornung, A., Smolic, A.: Automatic content creation for multiview autostereoscopic displays using image domain warping. In: ICME, pp. 1–6, IEEE. Barcelona, July 2011
Yao, S.J., Wang, L.H., Lin, C.L., Zhang, M.: Real-time stereo to multi-view conversion system based on adaptive meshing. J. Real-Time Image Process. 14(2), 481–499 (2018)
Wang, R., Luo, J., Jiang, X., Wang, Z., Wang, W., Li, G., Gao, W.: Accelerating Image-Domain-Warping Virtual View Synthesis on GPGPU. IEEE Trans. Multimed. 19(6), 1392–1400 (2017)
Zheng, E., Wu, C.: Structure from motion using structure-less resection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2075–2083. IEEE, Santiago, Chile (2015)
Scarpino, M.: OpenCL in Action: How to Accelerate Graphics and Computation. Manning Publication, Shelter Island (2011)
Chaurasia, G., Sorkine, O., Drettakis, G.: Silhouette-aware warping for image-based rendering. In: Computer Graphics Forum, vol. 30, No. 4, pp. 1223–1232, June 2011
Liu, F., Gleicher, M., Jin, H., Agarwala, A.: Content-preserving warps for 3D video stabilization. In: ACM Transactions on Graphics (TOG), Vol. 28, No. 3, p. 44, New Orleans, Louisiana July, 2009
Fang, J., Varbanescu, A. L., Sips, H.: A comprehensive performance comparison of CUDA and OpenCL. In: International Conference Parallel Processing (ICPP), pp. 216–225, Taipei City, Taiwan, September, 2011
Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). https://doi.org/10.1007/11744023_34
Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., Fua, P.: BRIEF: computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 34(7), 1281–1298 (2012)
Momcilovic, S., Ilic, A., Roma, N., Sousa, L.: Dynamic load balancing for real-time video encoding on heterogeneous CPU + GPU systems. IEEE Trans. Multimed. 16(1), 108–121 (2014)
Acknowledgement
Thanks to National Natural Science Foundation of China 61672063 Shenzhen Peacock Plan, Shenzhen Research Projects of JCYJ20160506172227337 and GGFW2017041215130858, Shenzhen Key Laboratory for Intelligent Multimedia and Virtual Reality ZDSY201703031405467.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, S., Wang, Z., Zhu, M., Wang, R. (2018). An End-to-End Real-Time 3D System for Integral Photography Display. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11164. Springer, Cham. https://doi.org/10.1007/978-3-030-00776-8_23
Download citation
DOI: https://doi.org/10.1007/978-3-030-00776-8_23
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00775-1
Online ISBN: 978-3-030-00776-8
eBook Packages: Computer ScienceComputer Science (R0)