Denoising and Inpainting of Sea Surface Temperature Image with Adversarial Physical Model Loss

  • Nobuyuki HiraharaEmail author
  • Motoharu Sonogashira
  • Hidekazu Kasahara
  • Masaaki Iiyama
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12046)


This paper proposes a new approach for meteorology; estimating sea surface temperatures (SSTs) by using deep learning. SSTs are essential information for ocean-related industries but are hard to measure. Although multi-spectral imaging sensors on meteorological satellites are used for measuring SSTs over a wide area, they cannot measure sea temperature in regions covered by clouds, so most of the temperature data will be partially occluded. In meteorology, data assimilation with physics-based simulation is used for interpolating occluded SSTs, and can generate physically-correct SSTs that match observations by satellites, but it requires huge computational cost. We propose a low-cost learning-based method using pre-computed data-assimilation SSTs. Our restoration model employs adversarial physical model loss that evaluates physical correctness of generated SST images, and restores SST images in real time. Experimental results with satellite images show that the proposed method can reconstruct physically-correct SST images without occlusions.


Image inpainting Sea surface temperature Adversarial loss 


  1. 1.
    Agostinelli, F., Anderson, M.R., Lee, H.: Adaptive multi-column deep neural networks with application to robust image denoising. In: Advances in Neural Information Processing Systems, pp. 1493–1501 (2013)Google Scholar
  2. 2.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 417–424 (2000)Google Scholar
  3. 3.
    Cai, N., Su, Z., Lin, Z., Wang, H., Yang, Z., Ling, B.W.K.: Blind inpainting using the fully convolutional neural network. Vis. Comput. 33(2), 249–261 (2017)CrossRefGoogle Scholar
  4. 4.
    Criminisi, A., Perez, P., Toyama, K.: Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13(9), 1200–1212 (2004)CrossRefGoogle Scholar
  5. 5.
    Demir, U., Unal, G.: Patch-based image inpainting with generative adversarial networks. arXiv preprint arXiv:1803.07422 (2018)
  6. 6.
    Ditri, A., Minnett, P., Liu, Y., Kilpatrick, K., Kumar, A.: The accuracies of himawari-8 and mtsat-2 sea-surface temperatures in the tropical Western Pacific ocean. Remote Sens. 10(2), 212 (2018). Scholar
  7. 7.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (Proc. of SIGGRAPH 2017) 36(4), 107:1–107:14 (2017)Google Scholar
  9. 9.
    Ishida, H., Nakajima, T.Y.: Development of an unbiased cloud detection algorithm for a spaceborne multispectral imager. J. Geophys. Res. (Atmos.) 114 (2009) Google Scholar
  10. 10.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Conference on Computer Vision and Pattern Recognition, pp. 5967–5976 (2017)Google Scholar
  11. 11.
    Mao, X.J., Shen, C., Yang, Y.B.: Image denoising using very deep fully convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing Systems, pp. 2810–2818 (2016)Google Scholar
  12. 12.
    McNally, A., Watts, P.: A cloud detection algorithm for high-spectral-resolution infrared sounders. Q. J. R. Meteorol. Soc. 129(595), 3411–3423 (2003)CrossRefGoogle Scholar
  13. 13.
    Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: feature learning by inpainting. In: Conference on Computer Vision and Pattern Recognition, pp. 2536–2544 (2016)Google Scholar
  14. 14.
    Shibata, S., Iiyama, M., Hashimoto, A., Minoh, M.: Restoration of sea surface temperature satellite images using a partially occluded training set. In: International Conference on Pattern Recognition (2018)Google Scholar
  15. 15.
    Song, Y., Yang, C., Lin, Z., Li, H., Huang, Q., Kuo, C.J.: Image inpainting using multi-scale feature image translation. CoRR abs/1711.08590 (2017).
  16. 16.
    Usui, N., Ishizaki, S., Fujii, Y., Tsujino, H., Yasuda, T., Kamachi, M.: Meteorological research institute multivariate ocean variational estimaion (move) system: some early results. Adv. Space Res. 37(4), 806–822 (2006)CrossRefGoogle Scholar
  17. 17.
    Xie, J., Xu, L., Chen, E.: Image denoising and inpainting with deep neural networks. In: Advances in Neural Information Processing Systems, pp. 341–349 (2012)Google Scholar
  18. 18.
    Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H.: High-resolution image inpainting using multi-scale neural patch synthesis. In: Conference on Computer Vision and Pattern Recognition, pp. 4076–4084 (2017)Google Scholar
  19. 19.
    Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Generative image inpainting with contextual attention. CoRR abs/1801.07892 (2018).

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Graduate School of InformaticsKyoto UniversityKyotoJapan
  2. 2.Academic Center for Computing and Media StudiesKyoto UniversityKyotoJapan

Personalised recommendations