No-reference synthetic image quality assessment with convolutional neural network and local image saliency
- 53 Downloads
Depth-image-based rendering (DIBR) is widely used in 3DTV, free-viewpoint video, and interactive 3D graphics applications. Typically, synthetic images generated by DIBR-based systems incorporate various distortions, particularly geometric distortions induced by object dis-occlusion. Ensuring the quality of synthetic images is critical to maintaining adequate system service. However, traditional 2D image quality metrics are ineffective for evaluating synthetic images as they are not sensitive to geometric distortion. In this paper, we propose a novel no-reference image quality assessment method for synthetic images based on convolutional neural networks, introducing local image saliency as prediction weights. Due to the lack of existing training data, we construct a new DIBR synthetic image dataset as part of our contribution. Experiments were conducted on both the public benchmark IRCCyN/IVC DIBR image dataset and our own dataset. Results demonstrate that our proposed metric outperforms traditional 2D image quality metrics and state-of-the-art DIBR-related metrics.
Keywordsimage quality assessment synthetic image depth-image-based rendering (DIBR) convolutional neural network local image saliency
The authors would like to thank the anonymous reviewers for their valuable comments. They would also thank Kai Wang and Jialei Li for their assistance in dataset construction and public release. The work was sponsored by the National Key R&D Program of China (No. 2017YFB1002702), and the National Natural Science Foundation of China (Nos. 61572058, 61472363).
- Smolic, A.; Mueller, K.; Merkle, P.; Kauff, P.; Wiegand, T. An overview of available and emerging 3D video formats and depth enhanced stereo as efficient generic solution. In: Proceedings of the Picture Coding Symposium, 1–4, 2009.Google Scholar
- Mark, W. Post-rendering 3D image warping: Visibility, reconstruction, and performance for depth-image warping. Technical Report. Chapel Hill, NC, USA, 1999.Google Scholar
- Zhou, Y.; Li, L.; Gu, K.; Fang, Y.; Lin, W. Quality assessment of 3D synthesized images via disoccluded region discovery. In: Proceedings of the IEEE International Conference on Image Processing, 1012–1016, 2016.Google Scholar
- Battisti, F.; Bosc, E.; Carli, M.; Le Callet, P.; Perugia, S. Objective image quality assessment of 3D synthesized views. Signal Processing: Image Communication Vol. 30, 78–88, 2015.Google Scholar
- Kang, L.; Ye, P.; Li, Y.; Doermann, D. Convolutional neural networks for no-reference image quality assessment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1733–1740, 2014.Google Scholar
- Bosse, S.; Maniry, D.; Wiegand, T.; Samek, W. A deep neural network for image quality assessment. In: Proceedings of the IEEE International Conference on Image Processing, 3773–3777, 2016.Google Scholar
- Bare, B.; Li, K.; Yan, B. An accurate deep convolutional neural networks model for no-reference image quality assessment. In: Proceedings of the IEEE International Conference on Multimedia and Expo, 1356–1361, 2017.Google Scholar
- Kim, J.; Nguyen, A.; Ahn, S.; Luo, C.; Lee, S. Multiple level feature-based universal blind image quality assessment model. In: Proceedings of the 25th IEEE International Conference on Image Processing, 291–295, 2018.Google Scholar
- Lin, K.-Y.; Wang, G. Hallucinated-IQA: No-reference image quality assessment via adversarial learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 732–741, 2018.Google Scholar
- Sheikh, H. R.; Wang, Z.; Cormack, L.; Bovik, A. C. Live image quality assessment database release 2 (2005). 2016. Available at https://doi.org/live.ece.utexas.edu/research/quality.Google Scholar
- Ponomarenko, N.; Lukin, V.; Zelensky, A.; Egiazarian, K.; Carli, M.; Battisti, F. TID2008-a database for evaluation of full-reference visual quality assessment metrics. Advances of Modern Radioelectronics Vol. 10, No. 4, 30–45, 2009.Google Scholar
- Ponomarenko, N.; Jin, L.; Ieremeiev, O.; Lukin, V.; Egiazarian, K.; Astola, J.; Vozel, B.; Chehdi, K.; Carli, M.; Battisti, F.; Jay Kuo, C.-C. Image database TID2013: Peculiarities, results and perspectives. Signal Processing: Image Communication Vol. 30, 57–77, 2015.Google Scholar
- Sandić Stanković, D.; Kukolj, D.; Le Callet, P. DIBR synthesized image quality assessment based on morphological wavelets. In: Proceedings of the 7th International Workshop on Quality of Multimedia Experience, 1–6, 2015.Google Scholar
- Heng, W.; Jiang. T. From image quality to patch quality: An image-patch model for no-reference image quality assessment. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, 1238–1242, 2017.Google Scholar
- Zhu, W.; Liang, S.; Wei, Y.; Sun, J. Saliency optimization from robust background detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2814–2821, 2014.Google Scholar
- Yang, X.; Ling, W.; Lu, Z.; Ong, E. P.; Yao, S. Just noticeable distortion model and its applications in video coding. Signal Processing: Image Communication Vol. 20, No. 7, 662–680, 2005.Google Scholar
- Kimata, H.; Kitahara, M.; Kamikura, K.; Yashima, Y. Free-viewpoint video communication using multiview video coding. NTT Technical Review Vol. 2, No. 8, 21–26, 2004.Google Scholar
- Domañski, M.; Grajek, T.; Klimaszewski, K.; Kurc, M.; Stankiewicz, O.; Stankowski, J.; Wegner, K. Poznan multiview video test sequences and camera parameters. ISO/IEC JTC1/SC29/WG11 MPEG, M17050, 2009.Google Scholar
- Mayer, N.; Ilg, E.; Hausser, P.; Fischer, P.; Cremers, D.; Dosovitskiy, A.; Brox, T. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4040–4048, 2016.Google Scholar
- Hirschmuller, H.; Scharstein, D. Evaluation of cost functions for stereo matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2007.Google Scholar
- Bosc, E.; Pépion, R.; Le Callet, P.; Köppel, M.; Ndjiki-Nya, P.; Morin, L.; Pressigout, M. Perceived quality of DIBR-based synthesized views. In: Proceedings of SPIE 8135, Applications of Digital Image Processing XXXIV, 81350I, 2011.Google Scholar
- Song, R.; Ko, H.; Jay Kuo, C.-C. MCL-3D: A database for stereoscopic image quality assessment using 2D-image-plus-depth source.Journal of Information Science and Engineering Vol. 31, 1593–1611, 2015.Google Scholar
- I. T. Union. ITU-R BT.910. In: Subjective video quality assessment methods for multimedia applications. 1999.Google Scholar
- I. T. Union. ITU-R BT.500-12. In: Recommendation: Methodology for the subjective assessment of the quality of television pictures. 1993.Google Scholar
- Liu, L.; Liu, B.; Huang, H.; Bovik, A. C. No-reference image quality assessment based on spatial and spectral entropies. Signal Processing: Image Communication Vol. 29, No. 8, 856–863, 2014.Google Scholar
- Bao, P.; Gourlay, D. Low bandwidth remote rendering using 3D image warping. In: Proceedings of the International Conference on Visual Information Engineering. Ideas, Applications, Experience, 61–64, 2003.Google Scholar
- Shi, S.; Nahrstedt, K.; Campbell, R. A real-time remote rendering system for interactive mobile graphics. ACM Transactions on Multimedia Computing, Communications, and Applications Vol. 8, No. 3s, Article No. 46, 2012.Google Scholar
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit https://doi.org/creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from https://doi.org/www.springer.com/journal/41095. To submit a manuscript, please go to https://doi.org/www.editorialmanager.com/cvmj.