Skip to main content
SpringerLink
Log in

Multimedia and geographic data integration for cultural heritage information retrieval

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper a system providing an efficient integration between Content-Based Image Retrieval (CBIR) and Geographic Information Retrieval (GIR) is presented. Over the years, many CBIR systems have been proposed to give a solution for an efficient use of multimedia/visual contents and other issues as performance, quality of retrieval, data heterogeneity, and multimodal information integration. The aim of the proposed approach is to prove that the use of geographic data can improve the results obtained by an image matching system based only on visual data. Our framework is composed of three parts, each of them described in detail in this paper: the first part is dedicated to CBIR, with an experimental comparison of a large number of different multimedia features to choose the one to use in the system implementation; in the second part the methodology to integrate geographic and multimedia data is showed; in the last part is presented a GIR system implementation using a “points of interest” search. An Android application has been developed for the client-side using Apache Solr as server side provider for the information retrieval functionalities. An experimental evaluation is carried out to demonstrate the effective improvement given by the combination of geographic and multimedia data. Our results have been obtained using a real dataset composed of artworks located in Naples’s museums.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://resourcemagonline.com/2014/12/infographic-there-will-be-one-trillion-photos-taken-in-2015/45332/ Last seen March 28, 2018.

  2. http://www.opengeospatial.org. Last seen March 28, 2018.

  3. https://www.google.com/maps. Last seen March 28, 2018.

  4. https://www.bing.com/maps. Last seen March 28, 2018.

  5. https://smartify.org. Last seen March 28, 2018.

  6. https://www.travel.getcoo.com. Last seen March 28, 2018.

References

  1. Adams B, McKenzie G, Gahegan M (2015) Frankenplace: interactive thematic mapping for ad hoc exploratory search. In: Proceedings of the 24th international conference on world wide web. ACM, pp 12–22

  2. Agnello F, Corsale R, Franco V, Lo Brutto M, Midulla P, Orlando P, Villa B (2003) Cultural heritage and information systems, an investigation into a dedicated hypertext. Int Archives Photogrammetry Remote Sens Spatial Inform Sci 34(5/W12):7–12

    Google Scholar 

  3. Ai L-F, Jun-Qing Y, He Y-F, Guan T (2013) High-dimensional indexing technologies for large scale content-based image retrieval: a review. J Zhejiang University Sci C 14(7):505–520

    Article  Google Scholar 

  4. Akgül CB, Rubin DL, Napel S, Beaulieu CF, Greenspan H, Acar B (2011) Content-based image retrieval in radiology: current status and future directions. J Digit Imag 24(2):208–222

    Article  Google Scholar 

  5. Basanth Kumar HB (2015) An overview on content-based image retrieval

  6. Bhatti A, Butt SM, Butt MM (2014) Visual feature extraction for content-based image retrieval. Sci Int 26(1):1–5

    Google Scholar 

  7. Bosch A, Zisserman A, Munoz X (2007) Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM international conference on image and video retrieval. ACM, pp 401–408

  8. Carson C, Belongie S, Greenspan H, Malik J (2002) Blobworld: image segmentation using expectation-maximization and its application to image querying. IEEE Trans Pattern Anal Mach Intell 24(8):1026–1038

    Article  Google Scholar 

  9. Chang S-F, Sikora T, Purl A (2001) Overview of the mpeg-7 standard. IEEE Trans Circuits Syst Video Technol 11(6):688–695

    Article  Google Scholar 

  10. Chatzichristofis SA, Boutalis YS (2008) Cedd: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval. In: International conference on computer vision systems. Springer, pp 312–322

  11. Chatzichristofis SA, Boutalis YS (2010) Content based radiology image retrieval using a fuzzy rule based scalable composite descriptor. Multimed Tool Appl 46(2-3):493–519

    Article  Google Scholar 

  12. Chatzichristofis SA, Fcth YSB (2008) Fuzzy color and texture histogram-a low level feature for accurate image retrieval. In: Ninth international workshop on Image analysis for multimedia interactive services, 2008. WIAMIS’08. IEEE, pp 191–196

  13. Chatzichristofis S, Boutalis Y, Lux M (2009) Selection of the proper compact composite descriptor for improving content based image retrieval. In: Proceedings of the 6th IASTED international conference, vol 134643, p 064

  14. Chen C-C, Wactlar HD, Wang JZ, Kiernan K (2005) Digital imagery for significant cultural and historical materials. Int J Digit Libr 5(4):275–286

    Article  Google Scholar 

  15. Datta R, Li J, Wang JZ (2005) Content-based image retrieval: approaches and trends of the new age. In: Proceedings of the 7th ACM SIGMM international workshop on multimedia information retrieval. ACM, pp 253–262

  16. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Computing Surveys (Csur) 40(2):5

    Article  Google Scholar 

  17. de Vries AP, Westerveld T (2004) A comparison of continuous vs. discrete image models for probabilistic image and video retrieval. In: 2004 international conference on Image processing, 2004. ICIP’04, vol 4. IEEE, pp 2387–2390

  18. Deselaers T, Keysers D, Ney H (2004) Fire–flexible image retrieval engine: imageclef 2004 evaluation. In: Workshop of the cross-language evaluation forum for european languages. Springer, pp 688–698

  19. Deselaers T, Keysers D, Ney H (2008) Features for image retrieval: an experimental comparison. Inform Retrieval 11(2):77–107

    Article  Google Scholar 

  20. Frederix G, Caenen G, Pauwels EJ (2000) Panoramic, adaptive and reconfigurable interface for similarity search. In: 2000 international conference on Image processing, 2000. Proceedings, vol 3. IEEE, pp 222–225

  21. Giuca A-M, Seitz KA, Furst J, Raicu D (2012) Expanding diagnostically labeled datasets using content-based image retrieval. In: 2012 19th IEEE international conference on Image processing (ICIP). IEEE, pp 2397–2400

  22. Golubovic N, Krintz C, Wolski R, Lafia S, Hervey T, Kuhn W (2016) Extracting spatial information from social media in support of agricultural management decisions. In: Proceedings of the 10th workshop on geographic information retrieval. ACM, p 4

  23. Hariharan R, Hore B, Li C, Mehrotra S (2007) Processing spatial-keyword (sk) queries in geographic information retrieval (gir) systems. In: 19th international conference on Scientific and statistical database management, 2007. SSBDM’07. IEEE, pp 16–16

  24. Huang J, Ravi Kumar S, Mitra M, Zhu W-J, Zabih R (1997) Image indexing using color correlograms. In: 1997 Proceedings of the IEEE computer society conference on Computer vision and pattern recognition, 1997. IEEE, pp 762–768

  25. Iqbal Q, Aggarwal JK (2002) Cires: a system for content-based retrieval in digital image libraries. In: 7th international conference on Control, automation, robotics and vision, 2002. ICARCV 2002, vol 1. IEEE, pp 205–210

  26. Irtaza A, Jaffar AM, Aleisa E, Choi T-S (2014) Embedding neural networks for semantic association in content based image retrieval. Multimed Tool Appl 72(2):1911–1931

    Article  Google Scholar 

  27. Irtaza A, Jaffar MA, Muhammad MS (2015) Content based image retrieval in a web 3.0 environment. Multimed Tool Appl 74(14):5055–5072

    Article  Google Scholar 

  28. Ivanova K, Dobreva M, Stanchev P, Totkov G (2012) Access to digital cultural heritage: innovative applications of automated metadata generation plovdiv university publishing house paisii hilendarski

  29. Ji Wx, Wang D, Hoi SCH, Pengcheng W, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval A comprehensive study. In: Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 157–166

  30. Karamti H, Tmar M, Visani M, Urruty T, Gargouri F (2017) Vector space model adaptation and pseudo relevance feedback for content-based image retrieval. Multimed Tool Appl:1–27

  31. Kim J, Vasardani M, Winter S (2017) Similarity matching for integrating spatial information extracted from place descriptions. Int J Geogr Inf Sci 31(1):56–80

    Article  Google Scholar 

  32. Koolen M, Kamps J, de Keijzer V (2009) Information retrieval in cultural heritage. Interdisc Sci Rev 34(2-3):268–284

    Article  Google Scholar 

  33. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  34. Kumar A, Kim J, Cai W, Fulham M, Feng D (2013) Content-based medical image retrieval: a survey of applications to multidimensional and multimodality data. J Digital Imag 26(6):1025–1039

    Article  Google Scholar 

  35. Larson RR (1996) Geographic information retrieval and spatial browsing. Geographic information systems and libraries: patrons, maps, and spatial information [papers presented at the 1995 Clinic on Library Applications of Data Processing, April 10-12, 1995]

  36. Lehmann TM, Güld MO, Deselaers T, Keysers D, Schubert H, Spitzer K, Ney H, Wein BB (2005) Automatic categorization of medical images for content-based retrieval and data mining. Comput Med Imaging Graph 29(2):143–155

    Article  Google Scholar 

  37. Liew CL (2005) Online cultural heritage exhibitions: a survey of information retrieval features. Program 39(1):4–24

    Article  MathSciNet  Google Scholar 

  38. Longley P (2005) Geographic information systems and science. Wiley, New York

    Google Scholar 

  39. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  40. Maliene V, Grigonis V, Paleviċius V, Griffiths S (2011) Geographic information system: old principles with new capabilities. Urban Des Int 16(1):1–6

    Article  Google Scholar 

  41. Mark DM (2003) Geographic information science: defining the field. Found Geographic Inform Sci 1:3–18

    Article  Google Scholar 

  42. Müller H, Müller W, Squire DM, Marchand-Maillet S, Pun T (2001) Performance evaluation in content-based image retrieval: overview and proposals. Pattern Recogn Lett 22(5):593–601

    Article  MATH  Google Scholar 

  43. Ng JY-H, Yang F, Davis LS (2015) Exploiting local features from deep networks for image retrieval. arXiv:1504.05133

  44. Ohm J-R (2001) The mpeg-7 visual description framework—concepts, accuracy, and applications. In: Computer analysis of images and patterns. Springer, pp 2–10

  45. Rasuli B, Asadi S (2017) Web geographic information retrieval: Literature review and conceptual model

  46. Rinaldi AM (2011) Automatic web pages hierarchical classification using dynamic domain ontologies. Int J Knowl Web Intell 2(4):231–256

    Article  Google Scholar 

  47. Rinaldi AM (2014) A multimedia ontology model based on linguistic properties and audio-visual features. Inf Sci 277:234–246

    Article  Google Scholar 

  48. Ruigang F, Li B, Gao Y, Wang P (2016) Content-based image retrieval based on cnn and svm. In: 2016 2nd IEEE International Conference on Computer and Communications (ICCC). IEEE, pp 638–642

  49. Shete DS, Chavan MS, Kolhapur K (2012) Content based image retrieval. Int J Emerging Technol Advan Eng 2(9):85–90

    Google Scholar 

  50. Shyu C-R, Klaric M, Scott GJ, Barb AS, Davis CH, Geoiris KP (2007) Geospatial information retrieval and indexing system—content mining, semantics modeling, and complex queries. IEEE Trans Geoscience Remote Sens 45(4):839–852

    Article  Google Scholar 

  51. Siggelkow S, Schael Mx, Burkhardt H (2001) Simba—search images by appearance. In: Joint pattern recognition symposium. Springer, pp 9–16

  52. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380

    Article  Google Scholar 

  53. Squire D, Muller W, Muller H, Raki J (1998) Content-based query of image databases, inspirations from text retrieval: inverted files frequency-based weights and relevance feedback

  54. Tangelder JWH, Veltkamp RC (2008) A survey of content based 3d shape retrieval methods. Multimed Tool Appl 39(3):441

    Article  Google Scholar 

  55. Tsai C-F (2007) A review of image retrieval methods for digital cultural heritage resources. Online Inf Rev 31(2):185–198

    Article  Google Scholar 

  56. Wang JZ, Li J, Wiederhold G (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans Pattern Anal Mach Intell 23(9):947–963

    Article  Google Scholar 

  57. Zaila YL (2017) GEIR: a full-fledged geographically enhanced information retrieval solution. PhD thesis alma

  58. Zheng Y-T, Zha Z-J, Chua T-S (2011) Research and applications on georeferenced multimedia: a survey. Multimed Tool Appl 51(1):77–98

    Article  Google Scholar 

  59. Zheng L, Yi Y, Tian Q (2017) Sift meets cnn: a decade survey of instance retrieval. IEEE Trans Pattern Anal Mach Intell

  60. Zhou XS, Huang TS (2003) Relevance feedback in image retrieval: A comprehensive review. Multimed Syst 8(6):536–544

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Information Technologies, University of Naples Federico II, Napoli, Italy

    Erasmo Purificato & Antonio M. Rinaldi

Authors
  1. Erasmo Purificato
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Antonio M. Rinaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonio M. Rinaldi.

Appendix A: Dataset

Appendix A: Dataset

The used dataset is show below togheter with the place of interest, the artwork name and the author.

  • Museo nazionale di Capodimonte

    1. 1.

      Atalanta e Ippomene - Guido Reni;

    2. 2.

      Bottega del macellaio - Joachim Beuckelaer;

    3. 3.

      Carlo di Borbone visita il Papa Benedetto XIV nella coffee-house del Quirinale - Giovanni Paolo Pannini;

    4. 4.

      Crocifissione - Masaccio;

    5. 5.

      Danae - Tiziano Vecellio;

    6. 6.

      Ercole al bivio - Annibale Carracci;

    7. 7.

      Flagellazione di Cristo - Caravaggio;

    8. 8.

      Fondazione di Santa Maria Maggiore - Masolino da Panicale;

    9. 9.

      Giuditta che decapita Oloferne - Artemisia Gentileschi;

    10. 10.

      La Zingarella - Correggio;

    11. 11.

      Lucrezia romana - Parmigianino;

    12. 12.

      Madonna col Bambino e due angeli - Botticelli;

    13. 13.

      Napoleone I, imperatore - François Gerard;

    14. 14.

      Partenza di Carlo per la Spagna vista da terra - Antonio Joli;

    15. 15.

      Partenza di Carlo per la Spagna vista dal mare - Antonio Joli;

    16. 16.

      Pietà - Annibale Carracci;

    17. 17.

      Ritratto di Ferdinando IV - Anton Raphael Mengs;

    18. 18.

      Ritratto di Galeazzo Sanvitale - Parmigianino;

    19. 19.

      Ritratto di Paolo III - Tiziano Vecellio;

    20. 20.

      Trasfigurazione di Cristo - Giovanni Bellini.

  • Gallerie di palazzo Zevallos

    1. 1.

      A Posillipo - Anton Sminck van Pitloo;

    2. 2.

      Adorazione dei Magi - Maestro dell’Annuncio ai pastori;

    3. 3.

      Agar e Ismaele nel deserto confortati dall’angelo - Francesco Solimena;

    4. 4.

      Autoritratto - Francesco Paolo Michetti;

    5. 5.

      Coppia di popolani in costume partenopeo - Anton Sminck van Pitloo;

    6. 6.

      Giuditta che decapita Oloferne - Louis Finson;

    7. 7.

      Il concerto - Gaspare Traversi;

    8. 8.

      Il coro della chiesa di Santa Maria Donnaregina Nuova - Domenico Battaglia;

    9. 9.

      La Zingara - Vincenzo Gemito;

    10. 10.

      Martirio di Sant’Orsola - Caravaggio;

    11. 11.

      Museo - Paolo Vetri;

    12. 12.

      Napoli via Toledo, impressioni di pioggia - Carlo Brancaccio;

    13. 13.

      Profeta - Vincenzo Gemito;

    14. 14.

      Ratto di Elena - Luca Gioradano;

    15. 15.

      Sansone e Dalila - Artemisia Gentileschi;

    16. 16.

      Tentazione - Gaetano Esposito;

    17. 17.

      Veduta di Napoli con largo di palazzo - Gaspar van Wittel;

    18. 18.

      Veduta di Roma con piazza del Popolo - Gaspar van Wittel;

    19. 19.

      Veduta di Roma con piazza Navona - Gaspar van Wittel;

    20. 20.

      Vitello - Anton Smick van Pitloo.

  • Museo nazionale di San Martino

    1. 1.

      Da Frisio a Santa Lucia - Eduardo Dalbono;

    2. 2.

      Darsena di Napoli - Gaspar van Wittel;

    3. 3.

      La grotta di Posillipo - Gaspar van Wittel;

    4. 4.

      La principessa di Sant’Antimo - Francesco Hayez;

    5. 5.

      Napoli dal mare - Gaspar van Wittel;

    6. 6.

      Presunto ritratto di Masaniello - Onofrio Palumbo;

    7. 7.

      Prevetariello - Antonio Mancini;

    8. 8.

      Ritratto di Eduardo Scarpetta nelle vesti di Felice Sciosciammocca - Augusto Emilio Fabri;

    9. 9.

      Ritratto di Masaniello - Onofrio Palumbo;

    10. 10.

      San Girolamo - Jusepe de Ribeira;

    11. 11.

      Tavola Strozzi - unknown;

    12. 12.

      Tribunale della vicaria - Carlo Coppola.

    13. 13.

      La Giumenta - Giuseppe Renda;

    14. 14.

      Madonna col Bambino e San Giovannino - Pietro Bernini;

    15. 15.

      Malatiello - Vincenzo Gemito;

    16. 16.

      San Martino divide il mantello con il povero - Pietro Bernini.

  • Cappella Sansevero

    1. 1.

      Ritratto di Vincenzo di Sangro - Carlo Amalfi.

    2. 2.

      Altare Maggiore (La Deposizione) - Francesco Celebrano;

    3. 3.

      Amor divino - unknown;

    4. 4.

      Cristo Velato - Giuseppe Sanmartino;

    5. 5.

      Decoro - Antonio Corradini;

    6. 6.

      Disinganno - Francesco Queirolo;

    7. 7.

      Dominio di se stessi - Francesco Celebrano;

    8. 8.

      Educazione - Francesco Queirolo;

    9. 9.

      Liberalità - Francesco Queirolo;

    10. 10.

      Macchine Anatomiche - Giuseppe Salerno;

    11. 11.

      Pudicizia velata - Antonio Corradini;

    12. 12.

      Santa Rosalia - Francesco Queirolo;

    13. 13.

      Sincerità - Francesco Queirolo;

    14. 14.

      Soavità del giogo coniugale - Paolo Persico;

    15. 15.

      Zelo della religione - Fortunato Onelli.

  • Museo archeologico nazionale di Napoli

    1. 1.

      Diana - unknown;

    2. 2.

      Flora - unknown;

    3. 3.

      Leda - unknown;

    4. 4.

      Medea - unknown;

    5. 5.

      Ritratto di Paquio Proculo - unknown;

    6. 6.

      Venditrice di amorini - unknown.

    7. 7.

      Afrodite Sosandra - Calamide;

    8. 8.

      Atlante Farnese - unknown;

    9. 9.

      Battaglia di Isso (mosaic) - unknown;

    10. 10.

      Cave canem (mosaic) - unknown;

    11. 11.

      Doriforo - Policleto;

    12. 12.

      Ercole Farnese - Glycon ateniese;

    13. 13.

      Hermes in riposo - unknown;

    14. 14.

      Testa di cavallo (Testa Carafa) - Donatello;

    15. 15.

      Venere di Capua - unknown.

  • Museo civico di Castel Nuovo

    1. 1.

      Abramo e i tre angeli - Pacecco De Rosa;

    2. 2.

      Adorazione dei Magi - Marco Cardisco;

    3. 3.

      Cesare Mormile e la rivolta napoletana del 1547 - Vincenzo Marinelli;

    4. 4.

      Ingresso di Garibaldi a Napoli - Franz Wenzel Schwarz;

    5. 5.

      Le stragi di Altamura - Michele Cammarano;

    6. 6.

      Madonna con Bambino e San Mauro Abate - Francesco Solimena;

    7. 7.

      Morte di San Giuseppe - Paolo De Matteis;

    8. 8.

      San Nicola in Gloria - Luca Giordano;

    9. 9.

      Veduta della nuova strada della Riviera di Chiaia - Carlo Vanvitelli.

    10. 10.

      Carlotta d’Asburgo a Miramare - Francesco Jerace;

    11. 11.

      Guappetiello - Francesco Jerace;

    12. 12.

      Il pescatorello - Vincenzo Gemito;

    13. 13.

      Testa di fanciullo - Vincenzo Gemito.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Purificato, E., Rinaldi, A.M. Multimedia and geographic data integration for cultural heritage information retrieval. Multimed Tools Appl 77, 27447–27469 (2018). https://doi.org/10.1007/s11042-018-5931-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5931-7

Keywords

Search

Navigation