Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 16, pp 21771–21790 | Cite as

Assessing the quality of experience in viewing rendered decompressed light fields

  • Cristian Perra
Article

Abstract

A light field image is a sampling of the intensity and direction information of the light rays crossing the main lens of a digital light field camera. The light field information captured by a camera must be processed in order to render images of the scene to the final user at a given focal plane or viewpoint. In the area of light field imaging, efficient representation, processing, compression and quality evaluation techniques and methodologies are currently under research in order to foster the development of novel industrial, entertainment or scientific applications. This paper focuses on the problem of evaluating the quality of experience when viewing rendered decompressed light field images. A processing chain for coding and decoding a light field image is first defined as reference model. Then, a novel metric for quality evaluation of the rendered views is proposed. This metric measures the variation of structural similarity on a set of viewpoints extracted from the light field. Subjective evaluation is performed and the correlation between objective metrics and the subjective results is reported and discussed. The proposed objective quality metric resulted in a nearly strong correlation with the subjective assessment results.

Keywords

Light field imaging Plenoptic imaging Integral imaging 

Notes

Acknowledgments

The research activities described in this paper have been partially funded within the R&D project “DigitArch” (Top-down cluster action program, funded by the POR FESR Sardegna 2014/2020), the R&D project “Cagliari2020” (partially funded by the Italian University and Research Ministry, grant# MIUR_PON04a2_00381), and within the R&D project “CagliariPort2020” (partially funded by the MIUR, grant# SCN_00281).

References

  1. 1.
    Adelson EH, Bergen JR (1991) The plenoptic function and the elements of early vision. In: Landy MS, Movshon AJ (eds) Computational models of visual processing. MIT Press, Cambridge, pp 3–20Google Scholar
  2. 2.
    Bjontegaard G (2001) Calcuation of average PSNR differences between RD-curves. Doc. VCEG-m33 ITU-T Q6/16, Austin, TX, USA, 2–4 April 2001Google Scholar
  3. 3.
    Chan SC, Ng KT, Gan ZF, Chan KL, Shum HY (2004) The plenoptic videos: capturing, rendering and compression. In: Proceedings of the 2004 international symposium on circuits and systems, 2004. ISCAS’04Google Scholar
  4. 4.
    Chang CL, Zhu X, Ramanathan P, Girod B (2006) Light field compression using disparity-compensated lifting and shape adaptation. IEEE Trans Image Process 15(4):793–806CrossRefGoogle Scholar
  5. 5.
    Cho D, Lee M, Kim S, Tai YW (2013) Modeling the calibration pipeline of the lytro camera for high quality light-field image reconstruction. In: IEEE international conference on computer vision (ICCV), 2013, pp 3280–3287.  https://doi.org/10.1109/ICCV.2013.407
  6. 6.
    Condat L, Forster-Heinlein B, Van De Ville D (2007) H2O: reversible hexagonal-orthogonal grid conversion by 1-D filtering. In: IEEE international conference on image processing, 2007. ICIP 2007, vol 2, pp II–73–II–76.  https://doi.org/10.1109/ICIP.2007.4379095
  7. 7.
    Conti C, Nunes P, Soares L (2013) Inter-layer prediction scheme for scalable 3-d holoscopic video coding. IEEE Signal Process Lett 20(8):819–822.  https://doi.org/10.1109/LSP.2013.2267234 CrossRefGoogle Scholar
  8. 8.
    Conti C, Soares LD, Nunes P (2016) HEVC-Based 3D holoscopic video coding using self-similarity compensated prediction. Signal Process Image Commun 42:59–78CrossRefGoogle Scholar
  9. 9.
    Cui J, Liu Y, Xu Y, Zhao H, Zha H (2013) Tracking generic human motion via fusion of low-and high-dimensional approaches. IEEE Trans Syst Man Cybern Syst 43(4):996–1002CrossRefGoogle Scholar
  10. 10.
    Dansereau D, Pizarro O, Williams S (2013) Decoding, calibration and rectification for lenselet-based plenoptic cameras. In: IEEE conference on computer vision and pattern recognition (CVPR), 2013.  https://doi.org/10.1109/CVPR.2013.137
  11. 11.
    Dansereau DG, Pizarro O, Williams SB (2013) Decoding, calibration and rectification for lenselet-based plenoptic cameras. Tech. rep., Australian Centre for Field Robotics; School of Aerospace, Mechanical and Mechatronic Engineering University of Sydney, NSW, AustraliaGoogle Scholar
  12. 12.
    Dricot A, Jung J, Cagnazzo M, Pesquet B, Dufaux F (2015) Integral images compression scheme based on view extraction. In: 23rd European signal processing conference (EUSIPCO), 2015, pp 101–105.  https://doi.org/10.1109/EUSIPCO.2015.7362353
  13. 13.
    Dricot A, Jung J, Cagnazzo M, Pesquet B, Dufaux F, Kovács P, Adhikarla V (2015) Subjective evaluation of super multi-view compressed contents on high-end light-field 3D displays. Signal Process Image Commun.  https://doi.org/10.1016/j.image.2015.04.012
  14. 14.
    Gehrig N, Dragotti PL (2007) Distributed compression of multi-view images using a geometrical coding approach. In: IEEE international conference on image processing, 2007. ICIP 2007, vol 6Google Scholar
  15. 15.
    Gehrig N, Dragotti PL (2009) Geometry-driven distributed compression of the plenoptic function: Performance bounds and constructive algorithms. IEEE Trans Image Process 18(3):457–470MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Gelman A, Dragotti PL, Velisavljević V (2010) Multiview image compression using a layer-based representation. In: 17th IEEE international conference on image processing (ICIP), 2010, pp 493–496Google Scholar
  17. 17.
    Gelman A, Dragotti PL, Velisavljević V (2011) Centralized and interactive compression of multiview images. In: SPIE optical engineering+ applications, pp 81,350J–81,350JGoogle Scholar
  18. 18.
    Higa RS, Chavez RFL, Leite RB, Arthur R, Iano Y (2013) Plenoptic image compression comparison between JPEG, JPEG2000 and SPITH. Cyber Journals: JSAT 3(6):1–6Google Scholar
  19. 19.
    ITU-R BT2022 (2012) General viewing conditions for subjective assessment of quality of sdtv and hdtv television pictures on flat panel displays. International Telecommunication UnionGoogle Scholar
  20. 20.
    ITU-R BT.500-13 (2012) Methodology for the subjective assessment of the quality of television pictures. International Telecommunication UnionGoogle Scholar
  21. 21.
    Joint photographic experts group (JPEG). http://jpeg.org/items/20150320_pleno_summary.html. Accessed: 2016-08-09
  22. 22.
    Levoy M, Hanrahan P (1996) Light field rendering. In: Proceedings of the 23rd annual conference on computer graphics and interactive techniques. ACM, pp 31–42Google Scholar
  23. 23.
    Li Y, Sjöström M, Olsson R, Jennehag U (2016) Scalable coding of plenoptic images by using a sparse set and disparities. IEEE Trans Image Process 25 (1):80–91MathSciNetCrossRefGoogle Scholar
  24. 24.
    Liu Y, Cui J, Zhao H, Zha H (2012) Fusion of low-and high-dimensional approaches by trackers sampling for generic human motion tracking. In: 21st international conference on pattern recognition (ICPR), 2012. IEEE, pp 898–901Google Scholar
  25. 25.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: Recognizing complex activities from sensor data. In: IJCAI, pp 1617–1623Google Scholar
  26. 26.
    Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115CrossRefGoogle Scholar
  27. 27.
    Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In: AAAI, vol 30, pp 1266–1272Google Scholar
  28. 28.
    Lumsdaine A, Georgiev T (2009) The focused plenoptic camera. In: IEEE international conference on computational photography (ICCP), 2009.  https://doi.org/10.1109/ICCPHOT.2009.5559008
  29. 29.
    Lytro Illum specifications. https://www.lytro.com/illum/specs/. Accessed: 2016-08-09
  30. 30.
    Magnor M, Girod B (2000) Data compression for light-field rendering. IEEE Trans Circuits Syst Video Technol 10(3):338–343CrossRefGoogle Scholar
  31. 31.
    Magnor M, Girod B (2000) Model-based coding of multiviewpoint imagery. In: Visual communications and image processing 2000, pp 14–22Google Scholar
  32. 32.
    Murgia F, Fernández SB, Perra C, Giusto D (2014) Unfocused plenoptic camera calibration. In: IEEE 22nd telecommunications forum. Belgrade, SerbiaGoogle Scholar
  33. 33.
    Murgia F, Giusto D, Perra C (2015) 3D reconstruction from plenoptic image. In: IEEE 23rd telecommunications forum. Belgrade, Serbia, pp 448–451Google Scholar
  34. 34.
    Murgia F, Perra C, Giusto D (2016) 3D point cloud reconstruction from single plenoptic image. Telfor Journal 8(1):26–31CrossRefGoogle Scholar
  35. 35.
    Ng R (2006) Digital light field photography. Ph.D. Thesis, Stanford University, Stanford, CA USAGoogle Scholar
  36. 36.
    Ng R, Levoy M, Brédif M, Duval G, Horowitz M, Hanrahan P (2005) Light field photography with a hand-held plenoptic camera. Computer Science Technical Report CSTR 2(11):1–11Google Scholar
  37. 37.
    Perra C (2014) On the coding of plenoptic raw images. In: IEEE 22nd telecommunications forum. Belgrade, SerbiaGoogle Scholar
  38. 38.
    Perra C (2015) Lossless plenoptic image compression using adaptive block differential prediction. In: IEEE 40th international conference on acoustics, speech and signal processing. Brisbane, AustraliaGoogle Scholar
  39. 39.
    Perra C (2016) Light field image compression based on preprocessing and high efficiency coding. In: IEEE 24th telecommunications forum. Belgrade, SerbiaGoogle Scholar
  40. 40.
    Perra C (2016) On the quality evaluation of lossy compressed light fields. In: IEEE 24th telecommunications forum. Belgrade, SerbiaGoogle Scholar
  41. 41.
    Perra C, Assuncao P (2016) High efficiency coding of light field images based on tiling and pseudo-temporal data arrangement. In: IEEE international conference on multimedia and expo (ICME). Seattle, USAGoogle Scholar
  42. 42.
    Perra C, Giusto D (2017) JPEG 2000 Compression of unfocused light field images based on lenslet array slicing. In: IEEE international conference on consumer electronics. Las Vegas, USAGoogle Scholar
  43. 43.
    Perra C, Giusto D (2017) Light field compression on sliced lenslet array. Intern J Internet Technol Secur Trans 8(1):1–14Google Scholar
  44. 44.
    Perra C, Giusto D (2017) Raw light field image compression of sliced lenslet array. In: 2017 IEEE international symposium on broadband multimedia systems and broadcasting (BMSB), pp 1–5.  https://doi.org/10.1109/BMSB.2017.7986158
  45. 45.
    Perra C, Murgia F, Giusto D (2016) An analysis of 3D point cloud reconstruction from light field images. In: International conference on image processing theory, tools and applications. Oulu, FinlandGoogle Scholar
  46. 46.
    Rerabek M, Yuan L, Authier LA, Ebrahimi T (2015) EPFL light-field image dataset. Tech. rep. ISO/IEC JTC 1/SC 29/WG1Google Scholar
  47. 47.
    Schwiegerling J, Birch GC, Tyo JS (2012) Analysis and compression of plenoptic camera images with Zernike polynomials. In: Proc. of SPIE volGoogle Scholar
  48. 48.
    Skodras A, Christopoulos C, Ebrahimi T (2001) The JPEG 2000 still image compression standard. IEEE Signal Proc Mag 18(5):36–58.  https://doi.org/10.1109/79.952804 CrossRefzbMATHGoogle Scholar
  49. 49.
    Sullivan GJ, Ohm JR, Han WJ, Wiegand T (2012) Overview of the high efficiency video coding (HEVC) standard. IEEE Trans Circuits Syst Video Technol 22 (12):1649–1668.  https://doi.org/10.1109/TCSVT.2012.2221191 CrossRefGoogle Scholar
  50. 50.
    Taguchi Y, Agrawal A, Ramalingam S, Veeraraghavan A (2010) Axial light field for curved mirrors: Reflect your perspective, widen your view. In: IEEE Conference on computer vision and pattern recognition (CVPR), 2010, pp 499–506.  https://doi.org/10.1109/CVPR.2010.5540172
  51. 51.
    Veeraraghavan A, Raskar R, Agrawal A, Mohan A, Tumblin J (2007) Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph 26(3).  https://doi.org/10.1145/1276377.1276463
  52. 52.
    Vieira A, Duarte H, Perra C, Tavora L, Assuncao P (2015) Data formats for high efficiency coding of lytro-illum light fields. In: International conference on image processing theory, tools and applications. Orléans, franceGoogle Scholar
  53. 53.
    Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13 (4):600–612.  https://doi.org/10.1109/TIP.2003.819861 CrossRefGoogle Scholar
  54. 54.
    Wilburn B, Joshi N, Vaish V, Levoy M, Horowitz M (2004) High-speed videography using a dense camera array. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004, vol 2, pp II–294–II–301.  https://doi.org/10.1109/CVPR.2004.1315176
  55. 55.
    Yu Z, Yu J, Lumsdaine A, Georgiev T (2012) An analysis of color demosaicing in plenoptic cameras. In: IEEE Conference on computer vision and pattern recognition (CVPR), 2012, pp 901–908.  https://doi.org/10.1109/CVPR.2012.6247764
  56. 56.
    Ziv J, Lempel A (1977) A universal algorithm for sequential data compression. IEEE Trans Inf Theory 23(3):337–343MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.University of CagliariCagliariItaly

Personalised recommendations