Skip to main content
SpringerLink
Log in

ST-Sem: A Multimodal Method for Points-of-Interest Classification Using Street-Level Imagery

  • Conference paper
  • First Online:
Book cover Web Engineering (ICWE 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11496))

Included in the following conference series:

Abstract

Street-level imagery contains a variety of visual information about the facades of Points of Interest (POIs). In addition to general morphological features, signs on the facades of, primarily, business-related POIs could be a valuable source of information about the type and identity of a POI. Recent advancements in computer vision could leverage visual information from street-level imagery, and contribute to the classification of POIs. However, there is currently a gap in existing literature regarding the use of visual labels contained in street-level imagery, where their value as indicators of POI categories is assessed. This paper presents Scene-Text Semantics (ST-Sem), a novel method that leverages visual labels (e.g., texts, logos) from street-level imagery as complementary information for the categorization of business-related POIs. Contrary to existing methods that fuse visual and textual information at a feature-level, we propose a late fusion approach that combines visual and textual cues after resolving issues of incorrect digitization and semantic ambiguity of the retrieved textual components. Experiments on two existing and a newly-created datasets show that ST-Sem can outperform visual-only approaches by 80% and related multimodal approaches by 4%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/CSAILVision/places365/.

  2. 2.

    http://rrc.cvc.uab.es/?ch=4.

  3. 3.

    http://www.image-net.org/synset.

  4. 4.

    https://github.com/barrust/pyspellchecker.

  5. 5.

    https://github.com/aboSamoor/polyglot.

  6. 6.

    https://fasttext.cc/docs/en/crawl-vectors.html.

  7. 7.

    https://developers.google.com/streetview/.

  8. 8.

    http://commoncrawl.org/the-data/.

  9. 9.

    https://github.com/shahinsharifi/ST-Sem.

  10. 10.

    https://github.com/taotaoorange/words-matter-scene-text-for-image-classification.

References

  1. Alcantarilla, P.F., Stent, S., Ros, G., Arroyo, R., Gherardi, R.: Street-view change detection with deconvolutional networks. Auton. Robots 42(7), 1301–1322 (2018)

    Article  Google Scholar 

  2. Balduini, M., Bozzon, A., Della Valle, E., Huang, Y., Houben, G.J.: Recommending venues using continuous predictive social media analytics. IEEE Internet Comput. 18(5), 28–35 (2014)

    Article  Google Scholar 

  3. Bocconi, S., Bozzon, A., Psyllidis, A., Titos Bolivar, C., Houben, G.J.: Social glass: a platform for urban analytics and decision-making through heterogeneous social data. In: Proceedings of the 24th International Conference on World Wide Web, pp. 175–178. WWW 2015 Companion. ACM, New York (2015)

    Google Scholar 

  4. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  5. Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4) (2012)

    Google Scholar 

  6. Falcone, D., Mascolo, C., Comito, C., Talia, D., Crowcroft, J.: What is this place? inferring place categories through user patterns identification in geo-tagged tweets. In: 2014 6th International Conference on Mobile Computing, Applications and Services (MobiCASE), pp. 10–19. IEEE (2014)

    Google Scholar 

  7. Fu, K., Chen, Z., Lu, C.T.: Streetnet: preference learning with convolutional neural network on urban crime perception. In: Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pp. 269–278. ACM (2018)

    Google Scholar 

  8. Gebru, T., Krause, J., Wang, Y., Chen, D., Deng, J., Aiden, E.L., Fei-Fei, L.: Using deep learning and google street view to estimate the demographic makeup of the us. arXiv preprint arXiv:1702.06683 (2017)

  9. Goel, R., et al.: Estimating city-level travel patterns using street imagery: a case study of using Google street view in britain. PloS One 13(5), e0196521 (2018)

    Article  Google Scholar 

  10. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  11. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. Int. J. Comput. Vis. 116(1), 1–20 (2016)

    Article  MathSciNet  Google Scholar 

  12. Jia, Y., et al.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)

    Google Scholar 

  13. Karaoglu, S., Tao, R., van Gemert, J.C., Gevers, T.: Con-text: text detection for fine-grained object classification. IEEE Trans. Image Proc. 26(8), 3965–3980 (2017)

    Article  MathSciNet  Google Scholar 

  14. Karaoglu, S., Tao, R., Gevers, T., Smeulders, A.W.: Words matter: scene text for image classification and retrieval. IEEE Trans. Multimed. 19(5), 1063–1076 (2017)

    Article  Google Scholar 

  15. Karatzas, D., et al.: ICDAR 2015 competition on robust reading. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1156–1160. IEEE (2015)

    Google Scholar 

  16. Li, X., Ratti, C., Seiferling, I.: Mapping urban landscapes along streets using Google street view. In: Peterson, M.P. (ed.) ICACI 2017. LNGC, pp. 341–356. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57336-6_24

    Chapter  Google Scholar 

  17. Liao, M., Shi, B., Bai, X.: Textboxes++: a single-shot oriented scene text detector. IEEE Trans. Image Proc. 27(8), 3676–3690 (2018)

    Article  MathSciNet  Google Scholar 

  18. Lofi, C.: Measuring semantic similarity and relatedness with distributional and knowledge-based approaches. Inf. Media Technol. 10(3), 493–501 (2015)

    Google Scholar 

  19. Luo, C., Jin, L., Sun, Z.: Moran: A multi-object rectified attention network for scene text recognition. Pattern Recognition (2019)

    Google Scholar 

  20. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  21. Movshovitz-Attias, Y., Yu, Q., Stumpe, M.C., Shet, V., Arnoud, S., Yatziv, L.: Ontological supervision for fine grained classification of street view storefronts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1693–1702 (2015)

    Google Scholar 

  22. Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. In: BMVC, vol. 1, p. 6 (2015)

    Google Scholar 

  23. Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  24. Quy Phan, T., Shivakumara, P., Tian, S., Lim Tan, C.: Recognizing text with perspective distortion in natural scenes. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 569–576 (2013)

    Google Scholar 

  25. Smith, S.L., Turban, D.H., Hamblin, S., Hammerla, N.Y.: Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859 (2017)

  26. Srivastava, S., Vargas Muñoz, J.E., Lobry, S., Tuia, D.: Fine-grained landuse characterization using ground-based pictures: a deep learning solution based on globally available data. Int. J. Geogr. Inf. Sci. 1–20 (2018)

    Google Scholar 

  27. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  28. Yan, B., Janowicz, K., Mai, G., Zhu, R.: xnet+sc: Classifying places based on images by incorporating spatial contexts. In: 10th International Conference on Geographic Information Science (GIScience 2018). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2018)

    Google Scholar 

  29. Yang, D., Li, B., Cudré-Mauroux, P.: Poisketch: semantic place labeling over user activity streams. Technical Report, Université de Fribourg (2016)

    Google Scholar 

  30. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: a 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2018)

    Article  Google Scholar 

  31. Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Advances in Neural Information Processing Systems, pp. 487–495 (2014)

    Google Scholar 

  32. Zhu, Y., Deng, X., Newsam, S.: Fine-grained land use classification at the city scale using ground-level images. IEEE Trans. Multimed. (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Delft University of Technology, Delft, The Netherlands

    Shahin Sharifi Noorian, Achilleas Psyllidis & Alessandro Bozzon

Authors
  1. Shahin Sharifi Noorian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Achilleas Psyllidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessandro Bozzon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shahin Sharifi Noorian .

Editor information

Editors and Affiliations

  1. Novosibirsk State Technical University, Novosibirsk, Russia

    Maxim Bakaev

  2. Erasmus University Rotterdam, Rotterdam, The Netherlands

    Flavius Frasincar

  3. Korea Advanced Institute of Science and Technology, Daejeon, Korea (Republic of)

    In-Young Ko

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Noorian, S.S., Psyllidis, A., Bozzon, A. (2019). ST-Sem: A Multimodal Method for Points-of-Interest Classification Using Street-Level Imagery. In: Bakaev, M., Frasincar, F., Ko, IY. (eds) Web Engineering. ICWE 2019. Lecture Notes in Computer Science(), vol 11496. Springer, Cham. https://doi.org/10.1007/978-3-030-19274-7_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-19274-7_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-19273-0

  • Online ISBN: 978-3-030-19274-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation