Multimedia Tools and Applications

, Volume 74, Issue 5, pp 1611–1625 | Cite as

Segmentation-based view synthesis for multi-view video plus depth

  • Maziar Loghman
  • Joohee KimEmail author


In this paper, we present a novel view synthesis algorithm for three-dimensional video. The proposed algorithm is based on segmentation using multi-level thresholding method. Recently, numerous techniques have been suggested which use a 2-D color image and the per-pixel depth map of the scene to create virtual views of the scene from any viewing position. However, inaccuracy in the depth maps cause annoying visual artifacts in depth-based view synthesis. In the proposed method, the depth maps are first preprocessed to avoid the errors caused by wrong depth values. Then, the color images are segmented according to the depth values and the regions belonging to different segments are warped independently. To further enhance the quality of the synthesized views, a multi-level thresholding based ghost removal algorithm and a novel hole filling algorithm have been proposed. Experimental results show that the proposed methods achieve an average PSNR gain of 0.98 dB for the multi-view test sequences and also improve the subjective quality of the synthesized views.


Depth image-based rendering Segmentation Hole fillings 



This work was supported by the Technology Development Program for Commercializing System Semiconductor funded By the Ministry of Trade, industry & Energy (MOTIE, Korea). (No. 10041126, Title: International Collaborative R&BD Project for System Semiconductor).


  1. 1.
    Chen WY, Chang YL, Lin SF, Ding LF, Chen LG (2005) Efficient depth image based rendering with edge dependent depth filter and interpolation. In: ICME, pp 1314–1317Google Scholar
  2. 2.
    Dyer CR (2001) Volumetric scene reconstruction from multiple views. In: Foundations of image understanding, pp 469–489Google Scholar
  3. 3.
    Fehn C (2003) A 3D-TV system based on video plus depth information. In: Proceedings conference record of the thirty-seventh asilomar conference on signals, systems and computers, vol 2, pp 1529–1533Google Scholar
  4. 4.
    Feng YM, Li DX, Luo K, Zhang M (2009) Asymmetric bidirectional view synthesis for free viewpoint and three-dimensional video. In: IEEE Transactions on consumer electronics, pp 2349–2355Google Scholar
  5. 5.
    Gautier J, Meur OL, Guillemot C (2011) Depth-based image completion for view synthesis. In: 3DTV Conference, pp 1–4Google Scholar
  6. 6.
    Lim H, Kim YS, Lee S, Choi O, Kim JDK, Kim JD (2011) Bi-layer inpainting for novel view synthesis. In: IEEE International conference on image processing, pp 1089–1092Google Scholar
  7. 7.
    Mori Y, Fukushima N, Fujii T, Tanimoto M (2008) View generation with 3-D warping using depth information for ftv. In: 3-DTV-CON’08, pp 229–232Google Scholar
  8. 8.
    Muller K, Merkle P, Wiegand T (2011) 3-D video representation using depth maps.Proc IEEE 99(4):643–656CrossRefGoogle Scholar
  9. 9.
    Oh K, Sehoon Y (2009) Hole-filling method using depth based in-painting for view synthesis in free viewpoint television (FTV) and 3-D video. In: Picture coding symposium (PCS), pp 1–4Google Scholar
  10. 10.
    Papamarkos N, Gatos B (1994) A new approach of multi level threshold selection. Graph Model Img Proc 56:357–370CrossRefGoogle Scholar
  11. 11.
    Porter T, Duff T (1984) Compositing Digital Images. In: Computer graphics, pp 253–259Google Scholar
  12. 12.
    Scharstein D, Szeliski R, Zabih R (2001) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of the IEEE workshop on stereo and multi-baseline vision (SMBV 2001), pp 131–140Google Scholar
  13. 13.
    Shade J, Gortler S, He LW, Szeliski R (1998) Layered Depth Images. In: ACM SIGGRAPH, pp 213–242Google Scholar
  14. 14.
    Shum HY, Kang SB (1999) A survey of image-based rendering techniques. In: Videometrics, SPIE, pp 2–16Google Scholar
  15. 15.
    Tanimoto M, Fujii T, Suzuki K (2009) View synthesis algorithm in view synthesis reference software 3.0 (VSRS3.0). Document M16090, ISO/IEC JTC1/SC29/WG11 (MPEG)Google Scholar
  16. 16.
    Wang Z, Bovik AC, Sheik HR (2004) Image quality assessment: from error visibility to structural similarity. In: Proceedings of IEEE transactions on image processing, vol 13, pp 600–612Google Scholar
  17. 17.
    Zinger S, Do L, de With PHN (2010) Free-viewpoint depth image based rendering. In: Journal of visual communication and image representation, vol 21, pp 533–541Google Scholar
  18. 18.
    Zitnick CL, Kang SB, Uyttendaele M, Winder S, Szeliski R (2004) High-quality video view interpolation using a layered representation. In: ACM Transactions on graphics, pp 600–608Google Scholar
  19. 19.
    Zhang Z (2012) Microsoft Kinect sensor and its effect.IEEE Multimed 19(2):4–10CrossRefGoogle Scholar
  20. 20.
    Zhang L, Tam WJ (2005) Stereoscopic image generation based on depth images for 3-D TV. In: IEEE Transactions on broadcasting, pp 191–199Google Scholar
  21. 21.
    Zhang L, Tam WJ, Wang D (2004) Stereoscopic image generation based on depth images. In: IEEE International conference on image processing, pp 2993–2996Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Department of Electrical and Computer EngineeringIllinois Institute of TechnologyChicagoUSA

Personalised recommendations