Advertisement

Soft Computing

, Volume 23, Issue 24, pp 13393–13408 | Cite as

Analysis of remote sensing imagery for disaster assessment using deep learning: a case study of flooding event

  • Liping YangEmail author
  • Guido Cervone
Methodologies and Application

Abstract

This paper proposes a methodology that integrates deep learning and machine learning for automatically assessing damage with limited human input in hundreds of thousands of aerial images. The goal is to develop a system that can help automatically identifying damaged areas in massive amount of data. The main difficulty consists in damaged infrastructure looking very different from when undamaged, likely resulting in an incorrect classification because of their different appearance, and the fact that deep learning and machine learning training sets normally only include undamaged infrastructures. In the proposed method, a deep learning algorithm is firstly used to automatically extract the presence of critical infrastructure from imagery, such as bridges, roads, or houses. However, because damaged infrastructure looks very different from when undamaged, the set of features identified can contain errors. A small portion of the images are then manually labeled if they include damaged areas, or not. Multiple machine learning algorithms are used to learn attribute–value relationships on the labeled data to capture the characteristic features associated with damaged areas. Finally, the trained classifiers are combined to construct an ensemble max-voting classifier. The selected max-voting model is then applied to the remaining unlabeled data to automatically identify images including damaged infrastructure. Evaluation results (85.6% accuracy and 89.09% F1 score) demonstrated the effectiveness of combining deep learning and an ensemble max-voting classifier of multiple machine learning models to analyze aerial images for damage assessment.

Keywords

Spatiotemporal data Image classification TensorFlow Machine learning Deep learning Damage assessment 

Abbreviations

ML

Machine learning

DL

Deep learning

CAP

Civilian Air Patrol

AI

Artificial intelligence

CNN

Convolutional neural network

RNN

Recurrent neural network

MLP

Multilayer perceptron

SVM

Support vector machine

RBF

Radial basis function

DT

Decision tree

NB

Naive Bayes

KNN

k-Nearest neighbors

RF

Random forest

GB

Gradient boosting

GBC

Gradient boosting classifier

LR

Logistic regression

LDA

Linear discriminant analysis

NN

Neural networks

USGS

United States Geological Survey

USGS HDDS

USGS Hazards Data Distribution System

Notes

Acknowledgements

This work was partially supported by the Office of Naval Research (ONR) award no. N00014-16-1-2543 (PSU no. 171570) and by the NVIDIA Corporation. We acknowledge Dr. Davide Del Vento from NCAR CISL and Dr. Chuck Pavloski at the Penn State Institute for CyberScience (ICS). The authors wish to thank Elena Sava for useful discussions and for providing the data and initial results relative to the Texas flood event.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283Google Scholar
  2. Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185MathSciNetGoogle Scholar
  3. Amancio DR, Comin CH, Casanova D, Travieso G, Bruno OM, Rodrigues FA, da Fontoura Costa L (2014) A systematic comparison of supervised classifiers. PLoS ONE 9(4):e94–137CrossRefGoogle Scholar
  4. Ben-Hur A, Horn D, Siegelmann HT, Vapnik V (2001) Support vector clustering. J Mach Learn Res 2(Dec):125–137zbMATHGoogle Scholar
  5. Bird S, Klein E, Loper E (2009) Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., NewtonzbMATHGoogle Scholar
  6. Bishop MC (2006) Pattern recognition and machine learning. Springer, New YorkzbMATHGoogle Scholar
  7. Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140zbMATHGoogle Scholar
  8. Breiman L (2001) Random forests. Mach Learn 45(1):5–32zbMATHCrossRefGoogle Scholar
  9. Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  10. Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2(2):121–167CrossRefGoogle Scholar
  11. Caruana R, Niculescu-Mizil A (2006) An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd international conference on machine learning. ACM, pp 161–168Google Scholar
  12. Cervone G, Sava E, Huang Q, Schnebele E, Harrison J, Waters N (2016) Using Twitter for tasking remote-sensing data collection and damage assessment: 2013 Boulder flood case study. Int J Remote Sens 37(1):100–124CrossRefGoogle Scholar
  13. Daelemans W, Van den Bosch A (2005) Memory-based language processing. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  14. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. CVPR 2009, IEEE, pp 248–255Google Scholar
  15. Domingos P (2012) A few useful things to know about machine learning. Commun ACM 55(10):78–87CrossRefGoogle Scholar
  16. Domingos P (2015) The master algorithm: how the quest for the ultimate learning machine will remake our world. Basic Books, New YorkGoogle Scholar
  17. Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29(2–3):103–130zbMATHCrossRefGoogle Scholar
  18. Dubitzky W, Granzow M, Berrar DP (2007) Fundamentals of data mining in genomics and proteomics. Springer, BerlinzbMATHCrossRefGoogle Scholar
  19. Elman JL (1990) Finding structure in time. Cognit Sci 14(2):179–211CrossRefGoogle Scholar
  20. Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139MathSciNetzbMATHCrossRefGoogle Scholar
  21. Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29:1189–1232MathSciNetzbMATHCrossRefGoogle Scholar
  22. Gislason PO, Benediktsson JA, Sveinsson JR (2006) Random forests for land cover classification. Pattern Recognit Lett 27(4):294–300CrossRefGoogle Scholar
  23. Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J et al (2017) Recent advances in convolutional neural networks. Pattern Recognit 77:354CrossRefGoogle Scholar
  24. Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier, AmsterdamzbMATHGoogle Scholar
  25. Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New YorkzbMATHCrossRefGoogle Scholar
  26. Kamiński B, Jakubczyk M, Szufel P (2018) A framework for sensitivity analysis of decision trees. Cent Eur J Oper Res 26(1):135–159MathSciNetzbMATHCrossRefGoogle Scholar
  27. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  28. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRefGoogle Scholar
  29. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRefGoogle Scholar
  30. Li Z, Wang C, Emrich CT, Guo D (2018) A novel approach to leveraging social media for rapid flood mapping: a case study of the 2015 South Carolina floods. Cartogr Geogr Inf Sci 45(2):97–110CrossRefGoogle Scholar
  31. Liong CY, Foo SF (2013) Comparison of linear discriminant analysis and logistic regression for data classification. In: AIP conference proceedings, AIP, vol 1522, pp 1159–1165Google Scholar
  32. Murthy SK (1998) Automatic construction of decision trees from data: a multi-disciplinary survey. Data Mining Knowl Discov 2(4):345–389CrossRefGoogle Scholar
  33. Ng AY, Jordan MI (2002) On discriminative vs. generative classifiers: a comparison of logistic regression and Naive Bayes. In: Advances in neural information processing systems, pp 841–848Google Scholar
  34. Opitz DW, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res (JAIR) 11:169–198zbMATHCrossRefGoogle Scholar
  35. Panteras G, Cervone G (2018) Enhancing the temporal resolution of satellite-based flood extent generation using crowdsourced data for disaster monitoring. Int J Remote Sens 39(5):1459–1474CrossRefGoogle Scholar
  36. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12(Oct):2825–2830MathSciNetzbMATHGoogle Scholar
  37. Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Syst Mag 6(3):21–45CrossRefGoogle Scholar
  38. Press SJ, Wilson S (1978) Choosing between logistic regression and discriminant analysis. J Am Stat Assoc 73(364):699–705zbMATHCrossRefGoogle Scholar
  39. Provost F, Kohavi R (1998) Glossary of terms. J Mach Learn 30(2–3):271–274Google Scholar
  40. Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106Google Scholar
  41. Rokach L (2010) Ensemble-based classifiers. Artif Intell Rev 33(1–2):1–39CrossRefGoogle Scholar
  42. Russell SJ, Norvig P, Canny JF, Malik JM, Edwards DD (2003) Artificial intelligence: a modern approach, vol 2. Prentice Hall, Upper Saddle RiverGoogle Scholar
  43. Salzberg SL (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Mining Knowl Discov 1(3):317–328CrossRefGoogle Scholar
  44. Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  45. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
  46. Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427–437CrossRefGoogle Scholar
  47. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A, et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
  48. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826Google Scholar
  49. Wainer J (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets. arXiv preprint arXiv:1606.00930
  50. Weiss GM, Provost F (2003) Learning when training data are costly: the effect of class distribution on tree induction. J Artif Intell Res 19:315–354zbMATHCrossRefGoogle Scholar
  51. Witten IH, Frank E, Hall MA (2011) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, BurlingtonGoogle Scholar
  52. Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8(7):1341–1390CrossRefGoogle Scholar
  53. Xiao T, Xia T, Yang Y, Huang C, Wang X (2015) Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2691–2699Google Scholar
  54. Yang L, MacEachren AM, Mitra P, Onorati T (2018) Visually-enabled active deep learning for (geo) text and image classification: a review. ISPRS Int J Geo-Inf 7(2):65CrossRefGoogle Scholar
  55. Zhu XX, Tuia D, Mou L, Xia GS, Zhang L, Xu F, Fraundorfer F (2017) Deep learning in remote sensing: a review. arXiv preprint arXiv:1710.03959

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Geography and Institute for CyberScienceThe Pennsylvania State UniversityUniversity ParkUSA

Personalised recommendations