Advertisement

B4MultiSR: A Benchmark for Multiple-Image Super-Resolution Reconstruction

  • Daniel KostrzewaEmail author
  • Łukasz Skonieczny
  • Paweł Benecki
  • Michał Kawulok
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 928)

Abstract

Super-resolution reconstruction (SRR) methods consist in processing single or multiple images to increase their spatial resolution. Deployment of such techniques is particularly important, when high resolution image acquisition is associated with high cost or risk, like for medical or satellite imaging. Unfortunately, the existing SRR techniques are not sufficiently robust to be deployed in real-world scenarios, and no real-life benchmark to validate multiple-image SRR has been published so far. As gathering a set of images presenting the same scene at different spatial resolution is not a trivial task, the SRR methods are evaluated based on different assumptions, employing various metrics and datasets, often without using any ground-truth data. In this paper, we introduce a new multi-layer benchmark dataset for systematic evaluation of multiple-image SRR techniques with particular reference to satellite imaging. We hope that the new benchmark will help the researchers to improve the state of the art in SRR, making it suitable for real-world applications.

Keywords

Super-resolution reconstruction Image processing Benchmark Dataset 

Notes

Acknowledgements

The reported work is a part of the SISPARE project run by Future Processing and funded by European Space Agency. In addition, the authors were partially supported by Statutory Research funds of Institute of Informatics, Silesian University of Technology, Gliwice, Poland (grants no. BKM-509/RAu2/2017 (DK) and BK-230/RAu2/2017 (MK)).

References

  1. 1.
    Aymaz, S., Köse, C.: A novel image decomposition-based hybrid technique with super-resolution method for multi-focus image fusion. Inf. Fusion 45, 113–127 (2019)CrossRefGoogle Scholar
  2. 2.
    Berkeley Segmentation Dataset. https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/. Accessed 27 Nov 2017
  3. 3.
    Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: Proceedings of 15th International Conference on Pattern Recognition, vol. 1, pp. 600–605. IEEE (2000)Google Scholar
  4. 4.
    Chang, K., Ding, P.L.K., Li, B.: Single image super-resolution using collaborative representation and non-local self-similarity. Sig. Process. 149, 49–61 (2018)CrossRefGoogle Scholar
  5. 5.
    Chavez-Roman, H., Ponomaryov, V.: Super resolution image generation using wavelet domain interpolation with edge extraction via a sparse representation. IEEE Geosci. Remote Sens. Lett. 11(10), 1777–1781 (2014)CrossRefGoogle Scholar
  6. 6.
    Common Objects in Context (COCO). http://cocodataset.org. Accessed 14 June 2018
  7. 7.
    Del Gallego, N.P., Ilao, J.: Multiple-image super-resolution on mobile devices: an image warping approach. EURASIP J. Image Video Process. 2017(1), 8 (2017)CrossRefGoogle Scholar
  8. 8.
    Demirel, H., Anbarjafari, G.: Image resolution enhancement by using discrete and stationary wavelet decomposition. IEEE Trans. Image Process. 20(5), 1458–1460 (2011)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 248–255. IEEE (2009)Google Scholar
  10. 10.
    Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2016)CrossRefGoogle Scholar
  11. 11.
    Farsiu, S., Robinson, M.D., Elad, M., Milanfar, P.: Fast and robust multiframe super resolution. IEEE Trans. Image Process. 13(10), 1327–1344 (2004)CrossRefGoogle Scholar
  12. 12.
    Fattal, R.: Image upsampling via imposed edge statistics. In: ACM Transactions on Graphics (TOG), vol. 26, p. 95. ACM (2007)Google Scholar
  13. 13.
    Hagita, K., Higuchi, T., Jinnai, H.: Super-resolution for asymmetric resolution of FIB-SEM 3D imaging using ai with deep learning. Sci. Rep. 8(1), 5877 (2018)CrossRefGoogle Scholar
  14. 14.
    Heinrich, L., Bogovic, J.A., Saalfeld, S.: Deep learning for isotropic super-resolution from non-isotropic 3D electron microscopy. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 135–143. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66185-8_16CrossRefGoogle Scholar
  15. 15.
    Huang, J.B., Singh, A., Ahuja, N.: Single image super-resolution from transformed self-exemplars. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5197–5206 (2015)Google Scholar
  16. 16.
    ImageNet. http://www.image-net.org/. Accessed 14 June 2018
  17. 17.
    Irani, M., Peleg, S.: Improving resolution by image registration. CVGIP Graph. Models Image Process. 53(3), 231–239 (1991)CrossRefGoogle Scholar
  18. 18.
    Jiang, J., Hu, R., Wang, Z., Han, Z.: Face super-resolution via multilayer locality-constrained iterative neighbor embedding and intermediate dictionary learning. IEEE Trans. Image Process. 23(10), 4220–4231 (2014)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Kawulok, M., Benecki, P., Kostrzewa, D., Skonieczny, L.: Towards evolutionary super-resolution. In: Sim, K., Kaufmann, P. (eds.) EvoApplications 2018. LNCS, vol. 10784, pp. 480–496. Springer, Cham (2018).  https://doi.org/10.1007/978-3-319-77538-8_33CrossRefGoogle Scholar
  20. 20.
    Köhler, T., Haase, S., Bauer, S., Wasza, J., Kilgus, T., Maier-Hein, L., Feußner, H., Hornegger, J.: ToF meets RGB: novel multi-sensor super-resolution for hybrid 3-D endoscopy. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8149, pp. 139–146. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-40811-3_18CrossRefGoogle Scholar
  21. 21.
    Li, L., Zhang, Y., Tian, Q.: Multi-face location on embedded dsp image processing system. In: Image and Signal Processing, 2008. CISP’08. Congress on. vol. 4, pp. 124–128. IEEE (2008)Google Scholar
  22. 22.
    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  23. 23.
    Liu, D., Wang, Z., Wen, B., Yang, J., Han, W., Huang, T.S.: Robust single image super-resolution via deep networks with sparse prior. IEEE Trans. Image Process. 25(7), 3194–3207 (2016)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Lukinavičius, G., Umezawa, K., Olivier, N., Honigmann, A., Yang, G., Plass, T., Mueller, V., Reymond, L., Corrêa Jr., I.R., Luo, Z.G.: A near-infrared fluorophore for live-cell super-resolution microscopy of cellular proteins. Nat. Chem. 5(2), 132–139 (2013)CrossRefGoogle Scholar
  25. 25.
    Martin, D., Fowlkes, C., Tal, D., Malik, J.: A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of 8th International Conference on Computer Vision, vol. 2, pp. 416–423, July 2001Google Scholar
  26. 26.
    Multi-Sensor Super-Resolution Datasets. https://www5.cs.fau.de/research/data/multi-sensor-super-resolution-datasets/. Accessed 02 Jan 2018
  27. 27.
    Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)CrossRefGoogle Scholar
  28. 28.
    Pickup, L.C., Roberts, S.J., Zisserman, A.: Optimizing and learning for super-resolution. In: Proceedings of the British Machine Vision Conference (2006)Google Scholar
  29. 29.
    Qian, S.E., Chen, G.: Enhancing spatial resolution of hyperspectral imagery using sensor’s intrinsic keystone distortion. IEEE Trans. Geosci. Remote Sens. 50(12), 5033–5048 (2012)CrossRefGoogle Scholar
  30. 30.
    Qu, C., Luo, D., Monari, E., Schuchert, T., Beyerer, J.: Capturing ground truth super-resolution data. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 2812–2816 (2016)Google Scholar
  31. 31.
    Schultz, R.R., Stevenson, R.L.: Extraction of high-resolution frames from video sequences. IEEE Trans. Image Process. 5(6), 996–1011 (1996)CrossRefGoogle Scholar
  32. 32.
    Sun, L., Hays, J.: Super-resolution from internet-scale scene matching. In: IEEE International Conference on Computational Photography (ICCP), pp. 1–12 (2012)Google Scholar
  33. 33.
    Super-Resolution dataset, Audiovisual Communications Laboratory, Ecole Polytechnique Federale de Lausanne, Switzerland. http://lcav.epfl.ch/software/superresolution. Accessed 02 Jan 2018
  34. 34.
    Vandewalle, P., Süsstrunk, S., Vetterli, M.: A frequency domain approach to registration of aliased images with application to super-resolution. EURASIP J. Adv. Sig. Process. 2006(1), 071459 (2006)CrossRefGoogle Scholar
  35. 35.
    Visual Geometry Group, Department of Engineering Science, University of Oxford, UK. http://www.robots.ox.ac.uk/~vgg/research/SR/index.html. Accessed 02 Jan 2018
  36. 36.
    Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)CrossRefGoogle Scholar
  37. 37.
    Wu, S., Ren, J., Chen, Z., Jin, W., Liu, X., Li, H., Pan, H., Guo, W.: Influence of reconstruction scale, spatial resolution and pixel spatial relationships on the sub-pixel mapping accuracy of a double-calculated spatial attraction model. Remote Sens. Environ. 210, 345–361 (2018)CrossRefGoogle Scholar
  38. 38.
    Yang, F., Chen, Y., Wang, R., Zhang, Q.: Super-resolution microwave imaging: time-domain tomography using highly accurate evolutionary optimization method. In: 2015 9th European Conference on Antennas and Propagation (EuCAP), pp. 1–4. IEEE (2015)Google Scholar
  39. 39.
    Yue, L., Shen, H., Li, J., Yuan, Q., Zhang, H., Zhang, L.: Image super-resolution: the techniques, applications, and future. Sig. Process. 128, 389–408 (2016)CrossRefGoogle Scholar
  40. 40.
    Zhong, Y., Zhang, L.: Remote sensing image subpixel mapping based on adaptive differential evolution. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 42(5), 1306–1329 (2012)CrossRefGoogle Scholar
  41. 41.
    Zhu, H., Song, W., Tan, H., Wang, J., Jia, D.: Super resolution reconstruction based on adaptive detail enhancement for ZY-3 satellite images. In: Proceedings of ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, pp. 213–217 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Daniel Kostrzewa
    • 1
    • 2
    Email author
  • Łukasz Skonieczny
    • 1
  • Paweł Benecki
    • 1
    • 2
  • Michał Kawulok
    • 1
    • 2
  1. 1.Future ProcessingGliwicePoland
  2. 2.Institute of InformaticsSilesian University of TechnologyGliwicePoland

Personalised recommendations