Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 3, pp 2837–2875 | Cite as

Social media and satellites

Disaster event detection, linking and summarization
  • Kashif AhmadEmail author
  • Konstantin Pogorelov
  • Michael Riegler
  • Nicola Conci
  • Pål Halvorsen
Article

Abstract

Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time. In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data. To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.

Keywords

Information retrieval Event detection Natural disaster Social media 

References

  1. 1.
    Ahmad K, De Natale F, Boato G, Rosani A (2016) A hierarchical approach to event discovery from single images using mil framework. In: 2016 IEEE global conference on Signal and information processing (globalSIP). IEEE, pp 1223–1227Google Scholar
  2. 2.
    Ahmad K, Konstantin P, Riegler M, Conci N, Halvorsen p. (2017) CNN and GAN based satellite and social media data fusion for disaster detection. In: Working notes proceedings of the Mediaeval workshop, pp 2Google Scholar
  3. 3.
    Ahmad K, Riegler M, Pogorelov K, Conci N, Halvorsen P, De Natale F (2017) Jord: a system for collecting information and monitoring natural disasters by linking social media with satellite imagery. In: Proceedings of the 15th International Workshop on Content-Based Multimedia Indexing. ACM, p 12Google Scholar
  4. 4.
    Ahmad S, Ahmad K, Ahmad N, Conci N (2017) Convolutional neural networks for disaster images retrieval. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  5. 5.
    Ahmad K, Conci N, De Natale F (2018) A saliency-based approach to event recognition. Signal Process Image Commun 60:42–51CrossRefGoogle Scholar
  6. 6.
    Amit SNKB, Shiraishi S, Inoshita T, Aoki Y (2016) Analysis of satellite images for disaster detection. In: 2016 IEEE international Geoscience and remote sensing symposium (IGARSS). IEEE, pp 5189–5192Google Scholar
  7. 7.
    Atefeh F, Khreich W (2015) A survey of techniques for event detection in twitter. Comput Intell 31(1):132–164MathSciNetCrossRefGoogle Scholar
  8. 8.
    Bakillah M, Li RY, Liang SH (2015) Geo-located community detection in twitter with enhanced fast-greedy optimization of modularity: the case study of typhoon haiyan. Int J Geogr Inf Sci 29(2):258– 279CrossRefGoogle Scholar
  9. 9.
    Bischke B, Borth D, Schulze C, Dengel A (2016) Contextual enrichment of remote-sensed events with social media streams. Proceedings of the 2016 ACM on Multimedia Conference. ACM, pp 1077–1081Google Scholar
  10. 10.
    Bischke B, Bhardwaj P, Gautam A, Helber P, Borth D, Dengel A (2017) Detection of flooding events in social multimedia and satellite imagery using deep neural networksGoogle Scholar
  11. 11.
    Bischke B, Helber P, Schulze C, Venkat S, Dengel A, Borth D (2017) The multimedia satellite task at mediaeval 2017: Emergence response for flooding events. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  12. 12.
    Campbell JB, Wynne RH (2011) Introduction to remote sensing. Guilford Press, New YorkGoogle Scholar
  13. 13.
    Chang SF (2016) New frontiers of large scale multimedia information retrieval. In: Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. ACM, pp. 5–5Google Scholar
  14. 14.
    Chavez P, Sides SC, Anderson JA (1991) Comparison of three different methods to merge multiresolution and multispectral data- landsat tm and spot panchromatic. Photogramm Eng Remote Sens 57(3):295–303Google Scholar
  15. 15.
    Chen T, Borth D, Darrell T, Chang SF (2014) Deepsentibank: Visual sentiment concept classification with deep convolutional neural networks. arXiv:1410.8586
  16. 16.
    Cheong M, Lee V (2010) Twittering for earth: a study on the impact of microblogging activism on earth hour 2009 in australia. In: Springer ACIIDS. Springer, pp. 114–123Google Scholar
  17. 17.
    Crooks A, Croitoru A, Stefanidis A, Radzikowski J (2013) # earthquake: Twitter as a distributed sensor system. Trans GIS 17(1):124–147CrossRefGoogle Scholar
  18. 18.
    Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: 2009. CVPR 2009. IEEE conference on Computer vision and pattern recognition. IEEE, pp. 248–255Google Scholar
  19. 19.
    Du R, Yu Z, Mei T, Wang Z, Wang Z, Guo B (2014) Predicting activity attendance in event-based social networks: Content, context and social influence. In: Proceedings of the 2014 ACM international joint conference on pervasive and ubiquitous computing. ACM, pp. 425–434Google Scholar
  20. 20.
    Earle PS, Bowden DC, Guy M (2012) Twitter earthquake detection: earthquake monitoring in a social world. Annals of Geophysics 54(6):708–715Google Scholar
  21. 21.
    Erol B, Hull JJ, Lee DS (2003) Linking multimedia presentations with their symbolic source documents: algorithm and applications. In: Proceedings of ACM MM, pp 498–507Google Scholar
  22. 22.
    Fisher A, Flood N, Danaher T (2016) Comparing landsat water index methods for automated water classification in eastern australia. IJRSE 175:167–182Google Scholar
  23. 23.
    Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680Google Scholar
  24. 24.
    Guha-Sapir D, Below R, Hoyois P (2015) Em-dat: International disaster database. Catholic University of Louvain, BrusselsGoogle Scholar
  25. 25.
    Guille A, Favre C (2015) Event detection, tracking, and visualization in twitter: a mention-anomaly-based approach. Soc Netw Anal Min 5(1):1–18CrossRefGoogle Scholar
  26. 26.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  27. 27.
    Joyce KE, Belliss SE, Samsonov SV, McNeill SJ, Glassey PJ (2009) A review of the status of satellite remote sensing and image processing techniques for mapping natural hazards and disasters. Progress in Physical GeographyGoogle Scholar
  28. 28.
    Jung M, Henkel K, Herold M, Churkina G (2006) Exploiting synergies of global land cover products for carbon cycle modeling. IJRSE 101(4):534–553Google Scholar
  29. 29.
    Kamilaris A, Prenafeta-Boldú FX Disaster monitoring using unmanned aerial vehicles and deep learningGoogle Scholar
  30. 30.
    Kansas J, Vargas J, Skatter HG, Balicki B, McCullum K (2016) Using landsat imagery to backcast fire and post-fire residuals in the boreal shield of saskatchewan: implications for woodland caribou management. IJWF 25(5):597–607Google Scholar
  31. 31.
    Keiller N, Samuel F, Ícaro D, Rafael W, Javier M, Otávio P, Rodrigo C (2017) Data-driven flood detection using neural networks. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  32. 32.
    Konstantinos A, Anastasia M, Andreadis S, Emmanouil M, Ilias G, Stefanos V, Ioannis K (2017) Visual and textual analysis of social media and satellite images for flood detection @ multimedia satellite task mediaeval 2017. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  33. 33.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  34. 34.
    Laura LF, Joost W, Marc B, Harald S (2017) Multi-modal deep learning approach for flood detection. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  35. 35.
    Li C, Sun A, Datta A (2012) Twevent: segment-based event detection from tweets. In: Proceedings of ACM IKM. ACM, pp 155–164Google Scholar
  36. 36.
    Liu Y, Wu L (2016) Geological disaster recognition on optical remote sensing images using deep learning. Proced Comput Sci 91:566–575CrossRefGoogle Scholar
  37. 37.
    Lux M, Riegler M, Halvorsen P, Pogorelov K, Anagnostopoulos N (2016) Lire: open source visual information retrieval. In: Proceedings of ACM MMSysGoogle Scholar
  38. 38.
    Manjunath T, Hegadi RS, Ravikumar G (2010) A survey on multimedia data mining and its relevance today. IJCSNS 10(11):165–170Google Scholar
  39. 39.
    Mathioudakis M, Koudas N (2010) Twittermonitor: trend detection over the twitter stream. In: Proceedings of ACM SIGMOD, pp 1155–1158Google Scholar
  40. 40.
    Meladianos P, Nikolentzos G, Rousseau F, Stavrakas Y, Vazirgiannis M (2015) Degeneracy-based real-time sub-event detection in twitter stream. In: Ninth international AAAI conference on web and social media, pp 248–257Google Scholar
  41. 41.
    Minh-Son D, Quang-Nhat-Minh P, Duc-Tien DN (2017) A domain-based late-fusion for disaster image retrieval from social media. In: Proceedings of the MediaEval. Workshop. DublinGoogle Scholar
  42. 42.
    Muhammad H, Muhammad A, Mahrukh K, Mohammad R (2017) Flood detection using social media data and spectral regression based kernel discriminant analysis. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  43. 43.
    Nataliya T, Arkaitz Z, Procter R (2017) Wisc at mediaeval 2017: Multimedia satellite task. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  44. 44.
    Nurwidyantoro A, Winarko E (2013) Event detection in social media: a survey. In: 2013 international conference on ICT For smart society (ICISS). IEEE, pp 1–5Google Scholar
  45. 45.
    Paul F, Andreassen LM (2009) A new glacier inventory for the svartisen region, norway, from landsat etm+ data: challenges and change assessment. J Glaciol 55(192):607–618CrossRefGoogle Scholar
  46. 46.
    Pogorelov K, Randel KR, Griwodz C, Eskeland SL, de Lange T, Johansen D, Spampinato C, Dang-Nguyen DT, Lux M, Schmidt PT, Riegler M, Halvorsen P (2017) Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In: Proceedings of the 8th ACM on Multimedia Systems Conference, MMSys’17. ACM, New York, pp 164–169.  https://doi.org/10.1145/3083187.3083212
  47. 47.
    Pogorelov K, Riegler M, Eskeland SL, de Lange T, Johansen D, Griwodz C, Schmidt PT, Halvorsen P (2017) Efficient disease detection in gastrointestinal videos–global features versus neural networks. Multimedia Tools and Applications 76:1–33Google Scholar
  48. 48.
    Popescu AM, Pennacchiotti M (2010) Detecting controversial events from twitter. In: Proceedings of ACM IKM. ACM, pp 1873–1876Google Scholar
  49. 49.
    Riegler M, Gaddam VR, Larson M, Eg R, Halvorsen P, Griwodz C (2016) Crowdsourcing as self-fulfilling prophecy: Influence of discarding workers in subjective assessment tasks. In: 2016 14th International Workshop on Content-Based Multimedia Indexing (CBMI), pp 1–6.  https://doi.org/10.1109/CBMI.2016.7500256
  50. 50.
    Sakaki T, Okazaki M, Matsuo Y (2010) Earthquake shakes twitter users: real-time event detection by social sensors. In: Proceedings of the 19th international conference on World wide web. ACM, pp 851–860Google Scholar
  51. 51.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  52. 52.
    Son J, Park SJ, Jung KH (2017) Retinal vessel segmentation in fundoscopic images with generative adversarial networks. arXiv:1706.09318
  53. 53.
    Stelter B, Cohen N (2008) Citizen journalists provided glimpses of mumbai attacks. The New York Times:30Google Scholar
  54. 54.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
  55. 55.
    Takahashi B, Tandoc EC, Carmichael C (2015) Communicating on twitter during a disaster: an analysis of tweets during typhoon haiyan in the philippines. Comput Hum Behav 50:392–398CrossRefGoogle Scholar
  56. 56.
    Team P (2016) Planet application program interface: In space for life on earth. san francisco, caGoogle Scholar
  57. 57.
    Tzelepis C, Ma Z, Mezaris V, Ionescu B, Kompatsiaris I, Boato G, Sebe N, Yan S (2016) Event-based media processing and analysis: A survey of the literature. Image and Vision ComputingGoogle Scholar
  58. 58.
    Wood H (2002) The use of earth observing satellites for hazard support: Assessments and scenarios. Final Report of the CEOS Disaster Management Support Group. Available from < http://disaster.ceos.org/legal. cfm
  59. 59.
    Xu Z, Zhang H, Sugumaran V, Choo KKR, Mei L, Zhu Y (2016) Participatory sensing-based semantic and spatial analysis of urban emergency events using mobile social media. EURASIP J Wirel Commun Netw 2016(1):44CrossRefGoogle Scholar
  60. 60.
    Yan C, Xie H, Liu S, Yin J, Zhang Y, Dai Q (2017) Effective uyghur language text detection in complex background images for traffic prompt identification. IEEE transactions on intelligent transportation systemsGoogle Scholar
  61. 61.
    Yan C, Xie H, Yang D, Yin J, Zhang Y, Dai Q (2017) Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE transactions on intelligent transportation systemsGoogle Scholar
  62. 62.
    Yin J, Lampert A, Cameron M, Robinson B, Power R (2012) Using social media to enhance emergency situation awareness. IJIS 27(6):52–59Google Scholar
  63. 63.
    Zhengyu Z, Larson M (2017) Retrieving social flooding images based on multimodal information. In: Proceedings of the MediaEval. Workshop, DublinGoogle Scholar
  64. 64.
    Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database. In: Proceedings of the NIPS, pp 487–495Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Kashif Ahmad
    • 1
    Email author
  • Konstantin Pogorelov
    • 1
  • Michael Riegler
    • 1
  • Nicola Conci
    • 1
  • Pål Halvorsen
    • 1
  1. 1.DISI-University of TrentoTrentoItaly

Personalised recommendations