Skip to main content
Log in

Segmentation-based view synthesis for multi-view video plus depth

Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present a novel view synthesis algorithm for three-dimensional video. The proposed algorithm is based on segmentation using multi-level thresholding method. Recently, numerous techniques have been suggested which use a 2-D color image and the per-pixel depth map of the scene to create virtual views of the scene from any viewing position. However, inaccuracy in the depth maps cause annoying visual artifacts in depth-based view synthesis. In the proposed method, the depth maps are first preprocessed to avoid the errors caused by wrong depth values. Then, the color images are segmented according to the depth values and the regions belonging to different segments are warped independently. To further enhance the quality of the synthesized views, a multi-level thresholding based ghost removal algorithm and a novel hole filling algorithm have been proposed. Experimental results show that the proposed methods achieve an average PSNR gain of 0.98 dB for the multi-view test sequences and also improve the subjective quality of the synthesized views.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Chen WY, Chang YL, Lin SF, Ding LF, Chen LG (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: ICME, pp 1314–1317

  2. Dyer CR (2001) Volumetric scene reconstruction from multiple views. In: Foundations of image understanding, pp 469–489

  3. Fehn C (2003) A 3D-TV system based on video plus depth information. In: Proceedings conference record of the thirty-seventh asilomar conference on signals, systems and computers, vol 2, pp 1529–1533

  4. Feng YM, Li DX, Luo K, Zhang M (2009) Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video. In: IEEE Transactions on consumer electronics, pp 2349–2355

  5. Gautier J, Meur OL, Guillemot C (2011) Depth-based image completion for view synthesis. In: 3DTV Conference, pp 1–4

  6. Lim H, Kim YS, Lee S, Choi O, Kim JDK, Kim JD (2011) Bi-layer inpainting for novel view synthesis. In: IEEE International conference on image processing, pp 1089–1092

  7. Mori Y, Fukushima N, Fujii T, Tanimoto M (2008) View generation with 3-D warping using depth information for ftv. In: 3-DTV-CON’08, pp 229–232

  8. Muller K, Merkle P, Wiegand T (2011) 3-D video representation using depth maps.Proc IEEE 99(4):643–656

    Article  Google Scholar 

  9. Oh K, Sehoon Y (2009) Hole-filling method using depth based in-painting for view synthesis in free viewpoint television (FTV) and 3-D video. In: Picture coding symposium (PCS), pp 1–4

  10. Papamarkos N, Gatos B (1994) A new approach of multi level threshold selection. Graph Model Img Proc 56:357–370

    Article  Google Scholar 

  11. Porter T, Duff T (1984) Compositing Digital Images. In: Computer graphics, pp 253–259

  12. Scharstein D, Szeliski R, Zabih R (2001) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of the IEEE workshop on stereo and multi-baseline vision (SMBV 2001), pp 131–140

  13. Shade J, Gortler S, He LW, Szeliski R (1998) Layered Depth Images. In: ACM SIGGRAPH, pp 213–242

  14. Shum HY, Kang SB (1999) A survey of image-based rendering techniques. In: Videometrics, SPIE, pp 2–16

  15. Tanimoto M, Fujii T, Suzuki K (2009) View synthesis algorithm in view synthesis reference software 3.0 (VSRS3.0). Document M16090, ISO/IEC JTC1/SC29/WG11 (MPEG)

  16. Wang Z, Bovik AC, Sheik HR (2004) Image quality assessment: from error visibility to structural similarity. In: Proceedings of IEEE transactions on image processing, vol 13, pp 600–612

  17. Zinger S, Do L, de With PHN (2010) Free-viewpoint depth image based rendering. In: Journal of visual communication and image representation, vol 21, pp 533–541

  18. Zitnick CL, Kang SB, Uyttendaele M, Winder S, Szeliski R (2004) High-quality video view interpolation using a layered representation. In: ACM Transactions on graphics, pp 600–608

  19. Zhang Z (2012) Microsoft Kinect sensor and its effect.IEEE Multimed 19(2):4–10

    Article  Google Scholar 

  20. Zhang L, Tam WJ (2005) Stereoscopic image generation based on depth images for 3-D TV. In: IEEE Transactions on broadcasting, pp 191–199

  21. Zhang L, Tam WJ, Wang D (2004) Stereoscopic image generation based on depth images. In: IEEE International conference on image processing, pp 2993–2996

Download references

Acknowledgments

This work was supported by the Technology Development Program for Commercializing System Semiconductor funded By the Ministry of Trade, industry & Energy (MOTIE, Korea). (No. 10041126, Title: International Collaborative R&BD Project for System Semiconductor).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Joohee Kim.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Loghman, M., Kim, J. Segmentation-based view synthesis for multi-view video plus depth. Multimed Tools Appl 74, 1611–1625 (2015). https://doi.org/10.1007/s11042-013-1747-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-013-1747-7

Keywords

Navigation