# Analysis of remote sensing imagery for disaster assessment using deep learning: a case study of flooding event

## Abstract

This paper proposes a methodology that integrates deep learning and machine learning for automatically assessing damage with limited human input in hundreds of thousands of aerial images. The goal is to develop a system that can help automatically identifying damaged areas in massive amount of data. The main difficulty consists in damaged infrastructure looking very different from when undamaged, likely resulting in an incorrect classification because of their different appearance, and the fact that deep learning and machine learning training sets normally only include undamaged infrastructures. In the proposed method, a deep learning algorithm is firstly used to automatically extract the presence of critical infrastructure from imagery, such as bridges, roads, or houses. However, because damaged infrastructure looks very different from when undamaged, the set of features identified can contain errors. A small portion of the images are then manually labeled if they include damaged areas, or not. Multiple machine learning algorithms are used to learn attribute–value relationships on the labeled data to capture the characteristic features associated with damaged areas. Finally, the trained classifiers are combined to construct an ensemble max-voting classifier. The selected max-voting model is then applied to the remaining unlabeled data to automatically identify images including damaged infrastructure. Evaluation results (85.6% accuracy and 89.09% F1 score) demonstrated the effectiveness of combining deep learning and an ensemble max-voting classifier of multiple machine learning models to analyze aerial images for damage assessment.

## Keywords

Spatiotemporal data Image classification TensorFlow Machine learning Deep learning Damage assessment## Abbreviations

- ML
Machine learning

- DL
Deep learning

- CAP
Civilian Air Patrol

- AI
Artificial intelligence

- CNN
Convolutional neural network

- RNN
Recurrent neural network

- MLP
Multilayer perceptron

- SVM
Support vector machine

- RBF
Radial basis function

- DT
Decision tree

- NB
Naive Bayes

- KNN
k-Nearest neighbors

- RF
Random forest

- GB
Gradient boosting

- GBC
Gradient boosting classifier

- LR
Logistic regression

- LDA
Linear discriminant analysis

- NN
Neural networks

- USGS
United States Geological Survey

- USGS HDDS
USGS Hazards Data Distribution System

## Notes

### Acknowledgements

This work was partially supported by the Office of Naval Research (ONR) award no. N00014-16-1-2543 (PSU no. 171570) and by the NVIDIA Corporation. We acknowledge Dr. Davide Del Vento from NCAR CISL and Dr. Chuck Pavloski at the Penn State Institute for CyberScience (ICS). The authors wish to thank Elena Sava for useful discussions and for providing the data and initial results relative to the Texas flood event.

### Compliance with ethical standards

### Conflict of interest

The authors declare that they have no conflict of interest.

### Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

## References

- Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283Google Scholar
- Altman NS (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46(3):175–185MathSciNetGoogle Scholar
- Amancio DR, Comin CH, Casanova D, Travieso G, Bruno OM, Rodrigues FA, da Fontoura Costa L (2014) A systematic comparison of supervised classifiers. PLoS ONE 9(4):e94–137CrossRefGoogle Scholar
- Ben-Hur A, Horn D, Siegelmann HT, Vapnik V (2001) Support vector clustering. J Mach Learn Res 2(Dec):125–137zbMATHGoogle Scholar
- Bird S, Klein E, Loper E (2009) Natural language processing with Python: analyzing text with the natural language toolkit. O’Reilly Media, Inc., NewtonzbMATHGoogle Scholar
- Bishop MC (2006) Pattern recognition and machine learning. Springer, New YorkzbMATHGoogle Scholar
- Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140zbMATHGoogle Scholar
- Breiman L (2001) Random forests. Mach Learn 45(1):5–32zbMATHCrossRefGoogle Scholar
- Buhmann MD (2003) Radial basis functions: theory and implementations, vol 12. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
- Burges CJ (1998) A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov 2(2):121–167CrossRefGoogle Scholar
- Caruana R, Niculescu-Mizil A (2006) An empirical comparison of supervised learning algorithms. In: Proceedings of the 23rd international conference on machine learning. ACM, pp 161–168Google Scholar
- Cervone G, Sava E, Huang Q, Schnebele E, Harrison J, Waters N (2016) Using Twitter for tasking remote-sensing data collection and damage assessment: 2013 Boulder flood case study. Int J Remote Sens 37(1):100–124CrossRefGoogle Scholar
- Daelemans W, Van den Bosch A (2005) Memory-based language processing. Cambridge University Press, CambridgeCrossRefGoogle Scholar
- Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. CVPR 2009, IEEE, pp 248–255Google Scholar
- Domingos P (2012) A few useful things to know about machine learning. Commun ACM 55(10):78–87CrossRefGoogle Scholar
- Domingos P (2015) The master algorithm: how the quest for the ultimate learning machine will remake our world. Basic Books, New YorkGoogle Scholar
- Domingos P, Pazzani M (1997) On the optimality of the simple Bayesian classifier under zero-one loss. Mach Learn 29(2–3):103–130zbMATHCrossRefGoogle Scholar
- Dubitzky W, Granzow M, Berrar DP (2007) Fundamentals of data mining in genomics and proteomics. Springer, BerlinzbMATHCrossRefGoogle Scholar
- Elman JL (1990) Finding structure in time. Cognit Sci 14(2):179–211CrossRefGoogle Scholar
- Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55(1):119–139MathSciNetzbMATHCrossRefGoogle Scholar
- Friedman JH (2001) Greedy function approximation: a gradient boosting machine. Ann Stat 29:1189–1232MathSciNetzbMATHCrossRefGoogle Scholar
- Gislason PO, Benediktsson JA, Sveinsson JR (2006) Random forests for land cover classification. Pattern Recognit Lett 27(4):294–300CrossRefGoogle Scholar
- Gu J, Wang Z, Kuen J, Ma L, Shahroudy A, Shuai B, Liu T, Wang X, Wang G, Cai J et al (2017) Recent advances in convolutional neural networks. Pattern Recognit 77:354CrossRefGoogle Scholar
- Han J, Pei J, Kamber M (2011) Data mining: concepts and techniques. Elsevier, AmsterdamzbMATHGoogle Scholar
- Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New YorkzbMATHCrossRefGoogle Scholar
- Kamiński B, Jakubczyk M, Szufel P (2018) A framework for sensitivity analysis of decision trees. Cent Eur J Oper Res 26(1):135–159MathSciNetzbMATHCrossRefGoogle Scholar
- Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
- LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRefGoogle Scholar
- LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRefGoogle Scholar
- Li Z, Wang C, Emrich CT, Guo D (2018) A novel approach to leveraging social media for rapid flood mapping: a case study of the 2015 South Carolina floods. Cartogr Geogr Inf Sci 45(2):97–110CrossRefGoogle Scholar
- Liong CY, Foo SF (2013) Comparison of linear discriminant analysis and logistic regression for data classification. In: AIP conference proceedings, AIP, vol 1522, pp 1159–1165Google Scholar
- Murthy SK (1998) Automatic construction of decision trees from data: a multi-disciplinary survey. Data Mining Knowl Discov 2(4):345–389CrossRefGoogle Scholar
- Ng AY, Jordan MI (2002) On discriminative vs. generative classifiers: a comparison of logistic regression and Naive Bayes. In: Advances in neural information processing systems, pp 841–848Google Scholar
- Opitz DW, Maclin R (1999) Popular ensemble methods: an empirical study. J Artif Intell Res (JAIR) 11:169–198zbMATHCrossRefGoogle Scholar
- Panteras G, Cervone G (2018) Enhancing the temporal resolution of satellite-based flood extent generation using crowdsourced data for disaster monitoring. Int J Remote Sens 39(5):1459–1474CrossRefGoogle Scholar
- Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12(Oct):2825–2830MathSciNetzbMATHGoogle Scholar
- Polikar R (2006) Ensemble based systems in decision making. IEEE Circuits Syst Mag 6(3):21–45CrossRefGoogle Scholar
- Press SJ, Wilson S (1978) Choosing between logistic regression and discriminant analysis. J Am Stat Assoc 73(364):699–705zbMATHCrossRefGoogle Scholar
- Provost F, Kohavi R (1998) Glossary of terms. J Mach Learn 30(2–3):271–274Google Scholar
- Quinlan JR (1986) Induction of decision trees. Mach Learn 1(1):81–106Google Scholar
- Rokach L (2010) Ensemble-based classifiers. Artif Intell Rev 33(1–2):1–39CrossRefGoogle Scholar
- Russell SJ, Norvig P, Canny JF, Malik JM, Edwards DD (2003) Artificial intelligence: a modern approach, vol 2. Prentice Hall, Upper Saddle RiverGoogle Scholar
- Salzberg SL (1997) On comparing classifiers: pitfalls to avoid and a recommended approach. Data Mining Knowl Discov 1(3):317–328CrossRefGoogle Scholar
- Shawe-Taylor J, Cristianini N (2004) Kernel methods for pattern analysis. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
- Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
- Sokolova M, Lapalme G (2009) A systematic analysis of performance measures for classification tasks. Inf Process Manag 45(4):427–437CrossRefGoogle Scholar
- Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A, et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
- Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826Google Scholar
- Wainer J (2016) Comparison of 14 different families of classification algorithms on 115 binary datasets. arXiv preprint arXiv:1606.00930
- Weiss GM, Provost F (2003) Learning when training data are costly: the effect of class distribution on tree induction. J Artif Intell Res 19:315–354zbMATHCrossRefGoogle Scholar
- Witten IH, Frank E, Hall MA (2011) Data mining: practical machine learning tools and techniques, 3rd edn. Morgan Kaufmann, BurlingtonGoogle Scholar
- Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Comput 8(7):1341–1390CrossRefGoogle Scholar
- Xiao T, Xia T, Yang Y, Huang C, Wang X (2015) Learning from massive noisy labeled data for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2691–2699Google Scholar
- Yang L, MacEachren AM, Mitra P, Onorati T (2018) Visually-enabled active deep learning for (geo) text and image classification: a review. ISPRS Int J Geo-Inf 7(2):65CrossRefGoogle Scholar
- Zhu XX, Tuia D, Mou L, Xia GS, Zhang L, Xu F, Fraundorfer F (2017) Deep learning in remote sensing: a review. arXiv preprint arXiv:1710.03959