Skip to main content
Log in

A rank aggregation framework for video multimodal geocoding

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a rank aggregation framework for video multimodal geocoding. Textual and visual descriptions associated with videos are used to define ranked lists. These ranked lists are later combined, and the resulting ranked list is used to define appropriate locations for videos. An architecture that implements the proposed framework is designed. In this architecture, there are specific modules for each modality (e.g, textual and visual) that can be developed and evolved independently. Another component is a data fusion module responsible for combining seamlessly the ranked lists defined for each modality. We have validated the proposed framework in the context of the MediaEval 2012 Placing Task, whose objective is to automatically assign geographical coordinates to videos. Obtained results show how our multimodal approach improves the geocoding results when compared to methods that rely on a single modality (either textual or visual descriptors). We also show that the proposed multimodal approach yields comparable results to the best submissions to the Placing Task in 2012 using no extra information besides the available development/training data. Another contribution of this work is related to the proposal of a new effectiveness evaluation measure. The proposed measure is based on distance scores that summarize how effective a designed/tested approach is, considering its overall result for a test dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://maps.google.com/ (As of Apr. 2013).

  2. http://www.google.com/earth/ (As of Apr. 2013).

  3. http://www.geonames.org (As of Apr. 2012).

  4. http://www.flickr.com/ (As of Apr. 2012).

  5. http://lucene.apache.org/solr/ (As of Apr. 2012).

  6. http://github.com/tow/sunburnt (As of Apr. 2012).

  7. http://lucene.apache.org/core/ (As of Apr. 2012).

References

  1. Almeida J, Leite NJ, Torres R da S (2011) Comparison of video sequences with histograms of motion patterns. In: International conference on image processing, pp 3673–3676

  2. Andrade FSP, Almeida J, Pedrini H, Torres R da S (2012) Fusion of local and global descriptors for content-based image and video retrieval. In: Iberoamerican congress on pattern recognition (CIARP’S), pp 845–853

  3. Boureau YL, Bach F, LeCun Y, Ponce J (2010) Learning mid-level features for recognition. In: Conference on computer vision and pattern recognition, pp 2559–2566. doi:10.1109/CVPR.2010.5539963

  4. Candeias R, Martins B (2011) Associating relevant photos to georeferenced textual documents through rank aggregation. In: Terra Cognita 2011 workshop. In conjunction with 10th international semantic web conference

  5. Choi J, Ekambaram VN, Friedland G, Ramchandran K (2012) The 2012 ICSI/Berkeley video location estimation system. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  6. Choi J, Lei H, Friedland G (2011) The 2011 ICSI video location estimation system. In: Working notes proceedings of the MediaEval workshop, vol 807

  7. Clinchant S, Ah-Pine J, Csurka G (2011) Semantic combination of textual and visual information in multimedia retrieval. In: International conference on multimedia retrieval, pp 44:1–44:8

  8. Coppersmith D, Fleischer LK, Rurda A (2010) Ordering by weighted number of wins gives a good ranking for weighted tournaments. ACM Trans Algorithm 6(3):55:1–55:13

    Article  MathSciNet  Google Scholar 

  9. Cormack GV, Clarke CLA, Buettcher S (2009) Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In: ACM SIGIR conference on research and development in information retrieval, pp 758–759

  10. Croft WB (2002) Combining approaches to information retrieval. In: Croft WB, Croft WB (eds) Advances in information retrieval, the information retrieval, vol 7. Springer US, pp 1–36

  11. Ding D, Zhang B (2007) Probabilistic model supported rank aggregation for the semantic concept detection in video. In: Proceedings of the 6th ACM international Conference on Image and Video Retrieval, CIVR ’07, pp 587–594. doi:10.1145/1282280.1282364. http://doi.acm.org/10.1145/1282280.1282364

  12. Faria FA, Veloso A, de Almeida HM, Valle E, Torres R da S, Gonçalves MA, Jr WM (2010) Learning to rank for content-based image retrieval. In: International conference on multimedia information retrieval, pp 285–294

  13. Fishburn PC (1988) Nonlinear preference and utility theory/Peter C. Fishburn. Johns Hopkins University Press, Baltimore

    Google Scholar 

  14. Fox EA, Shaw JA (1994) Combination of multiple searches. In: Text REtrieval Conference (TREC-2), vol 500–215, pp 243–252

  15. Friendly M (2002) Corrgrams: exploratory displays for correlation matrices. Am Stat 56(4):316–324

    Article  MathSciNet  Google Scholar 

  16. Hauff C, Houben GJ (2011) WISTUD at MediaEval 2011: placing task. In: Working notes proceedings of the MediaEval workshop, vol 807

  17. Hays J, Efros AA (2008) im2gps: estimating geographic information from a single image. In: Conference on computer vision and pattern recognition

  18. Jones CB, Purves RS (2008) Geographical information retrieval. Int J Geogr Inf Sci 22(3):219–228

    Article  Google Scholar 

  19. Kalantidis Y, Tolias G, Avrithis Y, Phinikettos M, Spyrou E, Mylonas P, Kollias S (2011) Viral: visual image retrieval and localization. Multimed Tools Appl 51:555–592

    Article  Google Scholar 

  20. Kelm P, Schmiedeke S, Sikora T (2011) A hierarchical, multi-modal approach for placing videos on the map using millions of flickr photographs. In: Workshop on Social and Behavioural Networked Media Access, SBNMA ’11, pp 15–20

  21. Kelm P, Schmiedeke S, Sikora T (2011) Multi-modal, multi-resource methods for placing Flickr videos on the map. In: International conference on multimedia retrieval

  22. Kelm P, Schmiedeke S, Sikora T (2012) How spatial segmentation improves the multimodal geo-tagging. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  23. Kelm P, Schmiedeke S, Sikora T (2012) Multimodal geo-tagging in social media websites using hierarchical spatial segmentation. In: LBSN ’12, pp 32–39. ACM, New York, NY, doi:10.1145/2442796.2442805. http://doi.acm.org/10.1145/2442796.2442805

    Google Scholar 

  24. Khudyak KA, Kurland O (2011) Cluster-based fusion of retrieved lists. In: Proceedings of the 34th international ACM SIGIR conference on research and development in information retrieval, SIGIR ’11, pp 893–902

  25. Klementiev A, Roth D, Small K (2008) A framework for unsupervised rank aggregation. In: Proc. of the ACM SIGIR conference (SIGIR) workshop on learning to rank for information retrieval, pp 32–39. http://cogcomp.cs.illinois.edu/papers/KlementievRoSm08a.pdf

  26. Kludas J, Bruno E, Marchand-Maillet S (2008) Information fusion in multimedia information retrieval. In: Boujemaa N, Detyniecki M, Nürnberger A (eds) Adaptive multimedial retrieval: retrieval, user, and semantics. Springer, New York, pp 147–159

    Chapter  Google Scholar 

  27. Kokar MM, Tomasik JA, Weyman J (2004) Formalizing classes of information fusion systems. Inform Fusion 5(3):189–202

    Article  Google Scholar 

  28. Laere OV, Schockaert S, Dhoedt B (2011) Ghent university at the 2011 placing task. In: Working notes proceedings of the MediaEval workshop, vol 807

  29. Laere OV, Schockaert S, Quinn JA, Langbein FC, Dhoedt B (2012) Ghent and cardiff university at the 2012 placing task. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  30. Larson M, Soleymani M, Serdyukov P, Rudinac S, Wartena C, Murdock V, Friedland G, Ordelman R, Jones GJF (2011) Automatic tagging and geotagging in video collections and communities. In: International conference on multimedia retrieval, pp 51:1–51:8

  31. Larson RR (2009) Geographic information retrieval and digital libraries. In: European conference on research and advanced technology for digital libraries, vol 5714, pp 461–464

  32. Li LT, Almeida J, Pedronette DCG, Penatti OAB, Torres R da S (2012) A multimodal approach for video geocoding. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  33. Li LT, Almeida J, Torres R da S (2011) RECOD working notes for placing task MediaEval 2011. In: Working notes proceedings of the MediaEval workshop, vol 807

  34. Li LT, Pedronette DCG, Almeida J, Penatti OAB, Calumby RT, Torres R da S (2012) Multimedia multimodal geocoding. In: ACM SIGSPATIAL international conference on advances in geographic information systems, pp 474–477

  35. Li X, Hauff C, Larson M, Hanjalic A (2012) Preliminary exploration of the use of geographical information for content-based geo-tagging of social video. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  36. Luo J, Joshi D, Yu J, Gallagher A (2011) Geotagging in multimedia and computer vision–a survey. Multimed Tools Appl 51:187–211

    Article  Google Scholar 

  37. Manning CD, Raghavan P, Schtze H (2008) Introduction to information retrieval. Cambridge University Press, New York, NY

    Book  MATH  Google Scholar 

  38. Montague M, Aslam JA (2002) Condorcet fusion for improved retrieval. In: Proceedings of the 11th international Conference on Information and Knowledge Management, CIKM ’02, pp 538–548. doi:10.1145/584792.584881. http://doi.acm.org/10.1145/584792.584881

  39. Olligschlaeger AM, Hauptmann AG (1999) Multimodal information systems and GIS: the informedia digital video library. In: 1999 ESRI user conference. http://www.informedia.cs.cmu.edu/documents/ESRI99.html

  40. Pedronette DCG (2012) Exploiting contextual information for image re-ranking and rank aggregation in image retrieval tasks. Ph.D. thesis, University of Campinas (UNICAMP), Campinas, SP, Brazil

  41. Pedronette DCG, Torres R da S (2011) Exploiting clustering approaches for image re-ranking. J Vis Lang Comput 22(6):453–466

    Article  Google Scholar 

  42. Pedronette DCG, Torres R da S, Calumby RT (2012) Using contextual spaces for image re-ranking and rank aggregation. Multimed Tools Appl :1–28. doi:10.1007/s11042-012-1115-z

  43. Penatti OAB, Li LT, Almeida J, Torres R da S (2012) A visual approach for video geocoding using bag-of-scenes. In: International conference on multimedia retrieval

  44. Poh N, Bengio S (2005) How do correlation and variance of base-experts affect fusion in biometric authentication tasks? IEEE Trans Signal Proces 53(11):4384–4396

    Article  MathSciNet  Google Scholar 

  45. Popescu A, Ballas N (2012) CEA LIST’s participation at mediaeval 2012 placing task. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  46. Rae A, Kelm P (2012) Working notes for the placing task at mediaeval 2012. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  47. Schalekamp F, Zuylen A (1998) Rank aggregation: together were strong. In: Workshop on Algorithm Engineering and Experiments (ALENEX), pp 38–51

  48. Sculley D (2007) Rank aggregation for similar items. In: SIAM international conference on Data Mining (SDM 2007), pp 587–592

  49. Serdyukov P, Murdock V, van Zwol R (2009) Placing flickr photos on a map. In: ACM SIGIR, pp 484–491. doi:10.1145/1571941.1572025

  50. Trevisiol M, Delhumeau J, Jégou H, Gravier G (2012) How INRIA/IRISA identifies geographic location of a video. In: Larson MA, Schmiedeke S, Kelm P, Rae A, Mezaris V, Piatrik T, Soleymani M, Metze F, Jones GJF (eds) Working notes proceedings of the MediaEval 2012 workshop, Santa Croce in Fossabanda, Pisa, Italy, 4–5 October, 2012, CEUR Workshop Proceedings, vol. 927. CEUR-WS.org

  51. Trevisiol M, Jégou H, Delhumeau J, Gravier G (2013) Retrieving geo-location of videos with a divide & conquer hierarchical multimodal approach. In: International conference on multimedia retrieval

  52. van Gemert JC, Veenman CJ, Smeulders AWM, Geusebroek JM (2010) Visual word ambiguity. IEEE Trans Pattern Anal Mach Intell 32:1271–1283

    Article  Google Scholar 

  53. Van Laere O, Schockaert S, Dhoedt B (2011) Finding locations of flickr resources using language models and similarity search. In: International conference on multimedia retrieval, pp 48:1–48:8. doi:10.1145/1991996.1992044

  54. Young HP (1974) An axiomatization of borda’s rule. J Econ Theory 9(1):43–52

    Article  Google Scholar 

  55. Zhang H, Jiang L, Su J (2005) Augmenting naive bayes for ranking. In: International conference on machine learning, pp 1020–1027

  56. Zhou X, Depeursinge A, Müller H (2010) Information fusion for combining visual and textual image retrieval in imageclef@icpr. In: Proceedings of the 20th International Conference on Recognizing Patterns in signals, speech, images, and videos, ICPR ’10. Springer-Verlag, Berlin, Heidelberg, pp 129–137. http://portal.acm.org/citation.cfm?id=1939170.1939189

    Chapter  Google Scholar 

Download references

Acknowledgements

The authors thank CAPES (Brazilian Federal Agency for Support and Evaluation of Graduate Education), FAPESP (São Paulo Research Foundation) grants 2011/11171-5 and 2009/10554-8, and CNPq (National Council for Scientific and Technological Development) grants 306580/2012-8 and 484254/2012-0, as well as CPqD Foundation (Telecommunications Research and Development Center) for their support. Additionally we would like to thank for the suggestions and questions arisen by the anonymous reviewers that gave us the chance to improve our paper.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lin Tzy Li.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Li, L.T., Pedronette, D.C.G., Almeida, J. et al. A rank aggregation framework for video multimodal geocoding. Multimed Tools Appl 73, 1323–1359 (2014). https://doi.org/10.1007/s11042-013-1588-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-013-1588-4

Keywords

Navigation