Abstract
In this paper, we present a novel view synthesis algorithm for three-dimensional video. The proposed algorithm is based on segmentation using multi-level thresholding method. Recently, numerous techniques have been suggested which use a 2-D color image and the per-pixel depth map of the scene to create virtual views of the scene from any viewing position. However, inaccuracy in the depth maps cause annoying visual artifacts in depth-based view synthesis. In the proposed method, the depth maps are first preprocessed to avoid the errors caused by wrong depth values. Then, the color images are segmented according to the depth values and the regions belonging to different segments are warped independently. To further enhance the quality of the synthesized views, a multi-level thresholding based ghost removal algorithm and a novel hole filling algorithm have been proposed. Experimental results show that the proposed methods achieve an average PSNR gain of 0.98 dB for the multi-view test sequences and also improve the subjective quality of the synthesized views.
References
Chen WY, Chang YL, Lin SF, Ding LF, Chen LG (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: ICME, pp 1314–1317
Dyer CR (2001) Volumetric scene reconstruction from multiple views. In: Foundations of image understanding, pp 469–489
Fehn C (2003) A 3D-TV system based on video plus depth information. In: Proceedings conference record of the thirty-seventh asilomar conference on signals, systems and computers, vol 2, pp 1529–1533
Feng YM, Li DX, Luo K, Zhang M (2009) Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video. In: IEEE Transactions on consumer electronics, pp 2349–2355
Gautier J, Meur OL, Guillemot C (2011) Depth-based image completion for view synthesis. In: 3DTV Conference, pp 1–4
Lim H, Kim YS, Lee S, Choi O, Kim JDK, Kim JD (2011) Bi-layer inpainting for novel view synthesis. In: IEEE International conference on image processing, pp 1089–1092
Mori Y, Fukushima N, Fujii T, Tanimoto M (2008) View generation with 3-D warping using depth information for ftv. In: 3-DTV-CON’08, pp 229–232
Muller K, Merkle P, Wiegand T (2011) 3-D video representation using depth maps.Proc IEEE 99(4):643–656
Oh K, Sehoon Y (2009) Hole-filling method using depth based in-painting for view synthesis in free viewpoint television (FTV) and 3-D video. In: Picture coding symposium (PCS), pp 1–4
Papamarkos N, Gatos B (1994) A new approach of multi level threshold selection. Graph Model Img Proc 56:357–370
Porter T, Duff T (1984) Compositing Digital Images. In: Computer graphics, pp 253–259
Scharstein D, Szeliski R, Zabih R (2001) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of the IEEE workshop on stereo and multi-baseline vision (SMBV 2001), pp 131–140
Shade J, Gortler S, He LW, Szeliski R (1998) Layered Depth Images. In: ACM SIGGRAPH, pp 213–242
Shum HY, Kang SB (1999) A survey of image-based rendering techniques. In: Videometrics, SPIE, pp 2–16
Tanimoto M, Fujii T, Suzuki K (2009) View synthesis algorithm in view synthesis reference software 3.0 (VSRS3.0). Document M16090, ISO/IEC JTC1/SC29/WG11 (MPEG)
Wang Z, Bovik AC, Sheik HR (2004) Image quality assessment: from error visibility to structural similarity. In: Proceedings of IEEE transactions on image processing, vol 13, pp 600–612
Zinger S, Do L, de With PHN (2010) Free-viewpoint depth image based rendering. In: Journal of visual communication and image representation, vol 21, pp 533–541
Zitnick CL, Kang SB, Uyttendaele M, Winder S, Szeliski R (2004) High-quality video view interpolation using a layered representation. In: ACM Transactions on graphics, pp 600–608
Zhang Z (2012) Microsoft Kinect sensor and its effect.IEEE Multimed 19(2):4–10
Zhang L, Tam WJ (2005) Stereoscopic image generation based on depth images for 3-D TV. In: IEEE Transactions on broadcasting, pp 191–199
Zhang L, Tam WJ, Wang D (2004) Stereoscopic image generation based on depth images. In: IEEE International conference on image processing, pp 2993–2996
Acknowledgments
This work was supported by the Technology Development Program for Commercializing System Semiconductor funded By the Ministry of Trade, industry & Energy (MOTIE, Korea). (No. 10041126, Title: International Collaborative R&BD Project for System Semiconductor).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Loghman, M., Kim, J. Segmentation-based view synthesis for multi-view video plus depth. Multimed Tools Appl 74, 1611–1625 (2015). https://doi.org/10.1007/s11042-013-1747-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-013-1747-7