Abstract
360-degree video has attracted more and more attention in recent years. However, it is a highly challenging task to transmit the high-resolution video within the limited bandwidth. In this paper, we first propose to unequally compress the cubemaps in each frame of the 360-degree video to reduce the total bitrate of the transmitted data. Specifically, a Group of Pictures (GOP) is used as a unit to alternately transmit different versions of the video. Each version consists of 3 high-quality cubemaps and 3 low-quality cubemaps. Then, the convolutional neural network (CNN) is introduced to enhance the low-quality cubemaps with the high-quality cubemaps by exploring the inter-frame similarities. It is shown in the experiment that a single CNN model can be used for various videos. The experimental results also show that the proposed method has an excellent quality enhancement compared with the benchmark in terms of PSNR, especially for videos with slow motion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Virtual reality augmented reality 2017 report. http://cdn.instantmagazine.com/upload/4666/bom_vrar_2017reportpdf.68ec9bc00f1c.pdf. Accessed 25 May 2018
Oculus Rift: http://www.oculus.com/rift/. Accessed 25 May 2018
Samsung Gear VR: http://www.samsung.com/global/microsite/gearvr. Accessed 25 May 2018
Google Cardboard. https://www.google.com/get/cardboard/. Accessed 25 May 2018
Equirectangular. https://en.wikipedia.org/wiki/Equirectangular_projection. Accessed 25 May 2018
Ng, K.-T., Chan, S.-C., Shum, H.-Y.: Data compression and transmission aspects of panoramic videos. IEEE Trans. Circuits Syst. Video Technol. 15(1), 82–95 (2005)
Kuzyakov, E., Pio, D.: Next-generation video encoding techniques for 360 video and vr. 2016-07-15) [2017-11-07] (2016). https://code.facebook.com/posts/1126354007399553/next-generationvideo-encoding-techniques-for-360-video-and-vr/. 15 July 2016
Fu, C.-W., Wan, L., Wong, T.-T., Leung, C.-S.: The rhombic dodecahedron map: an efficient scheme for encoding panoramic video. IEEE Trans. Multimedia 11(4), 634–644 (2009)
Sreedhar, K.K., Aminlou, A., Hannuksela, M.M., Gabbouj, M.: Viewport-adaptive encoding and streaming of 360-degree video for virtual reality applications. In: 2016 IEEE International Symposium on Multimedia (ISM), pp. 583–586. IEEE (2016)
Sullivan, G.J., Ohm, J.-R., Han, W.-J., Wiegand, T., et al.: Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. Circuits Syst. Video Technol. 22(12), 1649–1668 (2012)
Wiegand, T., Sullivan, G.J., Bjntegaard, G., Luthra, A.: Overview of the H.264/AVC video coding standard. IEEE Trans. Circuits Syst. Video Technol. 13(7), 560–576 (2003)
Corbillon, X., Simon, G., Devlic, A., Chakareski, J.: Viewport-adaptive navigable 360-degree video delivery. In: 2017 IEEE International Conference on Communications (ICC), pp. 1–7. IEEE (2017)
Zare, A., Aminlou, A., Hannuksela, M.M., Gabbouj, M.: Hevc-compliant tilebased streaming of panoramic video for virtual reality applications. In: Proceedings of the 2016 ACM on Multimedia Conference, pp. 601–605. ACM (2016)
Ozcinar, C., De Abreu, A., Smolic, A.: Viewport-aware adaptive 360° video streaming using tiles for virtual reality. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 2174–2178. IEEE (2017)
Ju, R., He, J., Sun, F., Li, J., Li, F., Zhu, J., Han, L.: Ultra wide view based panoramic vr streaming. In: Proceedings of the Workshop on Virtual Reality and Augmented Reality Network, pp. 19–23. ACM (2017)
Sanchez, Y., Skupin, R., Hellge, C., Schierl, T.: Spatio-temporal activity based tiling for panorama streaming. In: Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video, pp. 61–66. ACM (2017)
Gu, B., Sheng, V.S., Tay, K.Y., Romano, W., Li, S.: Incremental support vector learning for ordinal regression. IEEE Trans. Neural Netw. Learn. Syst. 26(7), 1403–1416 (2015)
Wen, X., Shao, L., Xue, Y., Fang, W.: A rapid learning algorithm for vehicle classification. Inf. Sci. 295(C), 395–406 (2015)
Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13
Lim, B., Son, S., Kim, H., Nah, S., Lee, K.M.: Enhanced deep residual networks for single image super-resolution. IEEE Conf. Comput. Vis. Pattern Recogn. Workshops 1, 4 (2017)
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: CVPR, pp. 4681–4690 (2017)
Dong, C., Deng, Y., Chen, C.L., Tang, X.: Compression artifacts reduction by a deep convolutional network. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 576–584 (2015)
Yu, L., Tillo, T., Xiao, J., Grangetto, M.: Convolutional neural network for intermediate view enhancement in multiview streaming. IEEE Trans. Multimed. 20(1), 15–28 (2018)
Dong, Y., Lin, C., Zhao, Y., Yao, C.: Depth map upsampling using joint edgeguided convolutional neural network for virtual view synthesizing. J. Electron. Imaging 26(4), 043004 (2017)
Wang, Y., Wang, R.: VRM, reference test model of immersive video content coding IEEE, 11857.9 subgroup (2017)
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant (61772066), by the Fundamental Research Funds for the Central Universities (2018JBM011).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Y., Yu, L., Lin, C., Zhao, Y., Gabbouj, M. (2018). Convolutional Neural Network Based Inter-Frame Enhancement for 360-Degree Video Streaming. In: Hong, R., Cheng, WH., Yamasaki, T., Wang, M., Ngo, CW. (eds) Advances in Multimedia Information Processing – PCM 2018. PCM 2018. Lecture Notes in Computer Science(), vol 11165. Springer, Cham. https://doi.org/10.1007/978-3-030-00767-6_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-00767-6_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00766-9
Online ISBN: 978-3-030-00767-6
eBook Packages: Computer ScienceComputer Science (R0)