Journal of Visualization

, Volume 22, Issue 1, pp 95–108 | Cite as

A novel in situ compression method for CFD data based on generative adversarial network

  • Yang LiuEmail author
  • Yueqing Wang
  • Liang Deng
  • Fang Wang
  • Fang Liu
  • Yutong Lu
  • Sikun Li
Regular Paper


As one of the main technologies of in situ visualization, data compression plays a key role in solving I/O bottleneck and has been intensively studied. However, existing methods take too much compression time to meet the requirement of in situ processing on computational fluid dynamics (CFD) flow field data. To address this problem, we introduce deep learning into CFD data compression and propose a novel in situ compression method based on generative adversarial network (GAN) in this paper. In specific, the proposed method samples small patches from CFD data and trains a GAN which includes two convolutional neural networks: the discriminative network and the generative network. The discriminative network is responsible for compressing data on compute nodes, while the generative network is used to reconstruct data on visualization nodes. Compared with the existing CFD data compression methods, our method has great advantages in compression time and manages to adjust compression ratio according to acceptable reconstruction effect, showing its applicability for loosely coupled in situ visualization. Extensive experimental results demonstrate the good generalization of the proposed method on many datasets.

Graphical Abstract


In situ visualization Data compression CFD flow field GAN 



The authors wish to thank Dr. Dong Sun for his guidance in CFD. This work was supported in part by the National Key Research and Development Program of China (#2016YFB0200701, #2018YFB0203904 and #2016YFB1000302), National Nature Science Foundation of China (#U1611261) and the Program for Guangdong Introducing Innovative and Enterpreneurial Teams (#2016ZT06D211).


  1. Agustsson E, Tschannen M, Mentzer F, Timofte R, Van Gool L (2018) Generative adversarial networks for extreme learned image compression. arXiv:1804.02958
  2. Alexander L, Jrg S (2006) Variable quality compression of fluid dynamical data sets using a 3d dct technique. Geochem Geophys Geosyst 7(1):1–13. Google Scholar
  3. Ballé J, Laparra V, Simoncelli EP (2016) End-to-end optimized image compression. arXiv:1611.01704
  4. Bauer AC, Abbasi H, Ahrens J, Childs H, Geveci B, Klasky S, Moreland K, O’Leary P, Vishwanath V, Whitlock B, Bethel EW (2016) In situ methods, infrastructures, and applications on high performance computing platforms. Comput Gr Forum 35(3):577–597. CrossRefGoogle Scholar
  5. Bengio Y (2009) Learning deep architectures for ai. Found Trends Mach Learn 2(1):1–127. MathSciNetCrossRefzbMATHGoogle Scholar
  6. Cox D (2016) Syntactically informed text compression with recurrent neural networks. arXiv:1608.02893
  7. Guo X, Li W, Iorio F (2016) Convolutional neural networks for steady flow approximation. In: Proceedings of the 22Nd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’16. ACM, New York, NY, USA, pp 481–490.
  8. Guo Y, Liu F, Cai Z, Xiao N, Zhao Z (2018) Edge-based efficient search over encrypted data mobile cloud storage. Sensors 18(4):
  9. Jiang F, Tao W, Liu S, Ren J, Guo X, Zhao D (2017) An end-to-end compression framework based on convolutional neural networks. IEEE Trans Circ Syst Video Technol, 1–13.
  10. Kress J, Klasky S, Podhorszki N, Choi J, Childs H, Pugmire D (2015) Loosely coupled in situ visualization: a perspective on why it’s here to stay. In: Proceedings of the first workshop on in situ infrastructures for enabling extreme-scale analysis and visualization, ISAV2015. ACM, New York, NY, USA, pp. 1–6.
  11. Lakshminarasimhan S, Shah N, Ethier S, Klasky S, Latham R, Ross R, Samatova NF (2011) Compressing the incompressible with ISABELA: in-situ reduction of spatio-temporal data. In: Jeannot E, Namyst R, Roman J (eds) Euro-Par 2011 parallel processing. Springer, Berlin, pp 366–379CrossRefGoogle Scholar
  12. Lee S, You D (2017) Prediction of laminar vortex shedding over a cylinder using deep learning. arXiv:1712.07854
  13. Li M, Zuo W, Gu S, Zhao D, Zhang D (2017) Learning convolutional networks for content-weighted image compression. arXiv:1703.10553
  14. Li S, Gruchalla K, Potter K, Clyne J, Childs H (2015) Evaluating the efficacy of wavelet configurations on turbulent-flow data. In: 2015 IEEE 5th symposium on large data analysis and visualization (LDAV), pp. 81–89.
  15. Lindstrom P (2014) Fixed-rate compressed floating-point arrays. IEEE Trans Vis Comput Gr 20(12):2674–2683. CrossRefGoogle Scholar
  16. Lindstrom P, Isenburg M (2006) Fast and efficient compression of floating-point data. IEEE Trans Vis Comput Gr 12(5):1245–1250. CrossRefGoogle Scholar
  17. Ling J, Kurzawski A, Templeton J (2016) Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J Fluid Mech 807:155–166. MathSciNetCrossRefzbMATHGoogle Scholar
  18. Liu F, Li T (2018) A clustering k-anonymity privacy-preserving method for wearable iot devices. Sec Commun Netw 2018:1–8. Google Scholar
  19. Sakai R, Sasaki D, Nakahashi K (2013) Parallel implementation of large-scale cfd data compression toward aeroacoustic analysis. Comput Fluids 80:116–127. (selected contributions of the 23rd International Conference on Parallel Fluid Dynamics ParCFD2011)CrossRefGoogle Scholar
  20. Salloum M, Fabian N, Hensinger DM, Templeton JA (2015) Compressed sensing and reconstruction of unstructured mesh datasets. arXiv:1508.06314
  21. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117. CrossRefGoogle Scholar
  22. Schmidhuber J, Heil S (1996) Sequential neural text compression. IEEE Trans Neural Netw 7(1):142–146. CrossRefGoogle Scholar
  23. Theis L, Bethge M (2015) Generative image modeling using spatial lstms. In: Cortes C, Lawrence ND, Lee DD, Sugiyama M, Garnett R (eds) Advances in neural information processing systems, vol 28. Curran Associates, Inc., Red Hook, pp 1927–1935Google Scholar
  24. Theis L, Shi W, Cunningham A, Huszár F (2017) Lossy Image compression with compressive autoencoders. arXiv:1703.00395
  25. Toderici G, O’Malley SM, Hwang SJ, Vincent D, Minnen D, Baluja S, Covell M, Sukthankar R (2015) Variable rate image compression with recurrent neural networks. arXiv:1511.06085
  26. van den Oord A, Kalchbrenner N, Kavukcuoglu K (2016) Pixel recurrent neural networks. arXiv:1601.06759
  27. Vogler P, Rist U (2017) Wavelet-based compression of volumetric cfd data sets. In: Resch MM, Bez W, Focht E, Gienger M, Kobayashi H (eds) Sustained simulation performance 2017. Springer International Publishing, Cham, pp 123–136CrossRefGoogle Scholar
  28. Welch TA (1984) A technique for high-performance data compression. Computer 17(6):8–19. CrossRefGoogle Scholar
  29. Zeyen M, Ahrens J, Hagen H, Heitmann K, Habib S (2017) Cosmological particle data compression in practice. In: Proceedings of the in situ infrastructures on enabling extreme-scale analysis and visualization, ISAV’17. ACM, New York, NY, USA, pp. 12–16.

Copyright information

© The Visualization Society of Japan 2018

Authors and Affiliations

  1. 1.College of Computer at National University of Defense TechnologyChangshaChina
  2. 2.Computational Aerodynamics Institute at China Aerodynamics Research and Development CenterMianyangChina
  3. 3.School of Data and Computer Science at Sun Yat-Sen UniversityGuangzhouChina

Personalised recommendations