Semantics Extraction From Multimedia Data: An Ontology-Based Machine Learning Approach

  • Sergios PetridisEmail author
  • Stavros J. Perantonis
Part of the Springer Series in Cognitive and Neural Systems book series (SSCNS)


It is often the case that related pieces of information lie in adjacent but different types of data sources. Besides extracting such information from each particular type of source, an important issue raised is how to put together all the pieces of information extracted by each source, or, more generally, what is the optimal way to collectively extract information, considering all media sources together. This chapter presents a machine learning method for extracting complex semantics stemming from multimedia sources. The method is based on transforming the inference problem into a graph expansion problem, expressing graph expansion operators as a combination of elementary ones and optimally seeking elementary graph operators. The latter issue is then reduced to learn a set of soft classifiers, based on features each one corresponding to a unique graph path. The advantages of the method are demonstrated on an athletics web-pages corpus, comprising images and text.


Multimedia Data Elementary Operator Multimedia Document Graph Expansion Approximate Inference 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This study is partly supported by the research projects “BOEMIE, Bootstrapping Ontology Evolution with Multimedia Information Extraction”. FP6-027538/STREP, 2006–2009, and “CASAM, Computer-Aided Semantic Annotation of Multimedia”. ICT-217061/STREP, 2008,


  1. P. Aarabi and B.V. Dasarathy. Robust speech processing using multi-sensor multi-source information fusion – an overview of the state of the art. Information Fusion, 5(2):77–80, 2004.CrossRefGoogle Scholar
  2. A. Goshtasby and S. Nikolov. Image fusion: Advances in the state of the art. Information Fusion, 8(2):114–118, 2007.CrossRefGoogle Scholar
  3. N. Friedman, L. Getoor, D. Koller, and A. Pfeffer. Learning probabilistic relational models. In International Joint Conference on Artificial Intelligence, volume 16, pages 1300–1309. Citeseer, 1999.Google Scholar
  4. K. Kersting, L. De Raedt, and T. Raiko. Logical hidden markov models. Journal of Artificial Intelligence Research, 25(1):425–456, 2006.Google Scholar
  5. B.D. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. In International joint conference on artificial intelligence, volume 3, pages 674–679. Citeseer, 1981.Google Scholar
  6. T. Lukasiewicz. Probabilistic description logic programs. International Journal of Approximate Reasoning, 45(2):288–307, 2007.CrossRefGoogle Scholar
  7. A.V. Nefian, L. Liang, X. Pi, X. Liu, and K. Murphy. Dynamic Bayesian Networks for Audio-Visual Speech Recognition. EURASIP Journal on Applied Signal Processing, 2002(11): 1274–1288, 2002.CrossRefGoogle Scholar
  8. J. Neville and D. Jensen. Relational dependency networks. The Journal of Machine Learning Research, 8:692, 2007.Google Scholar
  9. S.E. Peraldi, A. Kaya, S. Melzer, R. Moller, and M. Wessel. Multimedia interpretation as abduction. In Proc. DL-2007: International Workshop on Description Logics, 2007.Google Scholar
  10. S. Petridis and N. Tsapatsoulis. Semantics Extraction from Multimedia Content: The BOEMIE Architecture. In Proceeding of the first international conference on Semantics and digital Media Technology (SAMT 2006), pages 6–8, 2006.Google Scholar
  11. M. Richardson and P. Domingos. Markov logic networks. Machine Learning, 62(1):107–136, 2006.CrossRefGoogle Scholar
  12. S. Rudolph, T. Tserendorj, and P. Hitzler. What Is Approximate Reasoning? In Proceedings of the 2nd International Conference on Web Reasoning and Rule Systems, pages 150–164. Springer, 2008.Google Scholar
  13. U. Straccia. Reasoning within Fuzzy Description Logics. Journal of Artificial Intelligence Research, 14:137–166, 2001.Google Scholar
  14. E. Zavitsanos, G. Paliouras, G.A. Vouros, and S. Petridis. Discovering subsumption hierarchies of ontology concepts from text corpora. In Proceedings of the IEEE/WIC/ACM International Conference on Web Intelligence, pages 402–408. IEEE Computer Society Washington, DC, USA, 2007.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Institute of Informatics and TelecommunicationsNCSR “Demokritos”AttikiGreece

Personalised recommendations