Skip to main content
SpringerLink
Log in

Interpretation, Evaluation and the Semantic Gap ... What if We Were on a Side-Track?

  • Conference paper
  • First Online:
Graphics Recognition. Current Trends and Challenges (GREC 2013)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 8746))

Included in the following conference series:

Abstract

A significant amount of research in Document Image Analysis, and Machine Perception in general, relies on the extraction and analysis of signal cues with the goal of interpreting them into higher level information. This paper gives an overview on how this interpretation process is usually considered, and how the research communities proceed in evaluating existing approaches and methods developed for realizing these processes. Evaluation being an essential part to measuring the quality of research and assessing the progress of the state-of-the art, our work aims at showing that classical evaluation methods are not necessarily well suited for interpretation problems, or, at least, that they introduce a strong bias, not necessarily visible at first sight, and that new ways of comparing methods and measuring performance are necessary. It also shows that the infamous Semantic Gap seems to be an inherent and unavoidable part of the general interpretation process, especially when considered within the framework of traditional evaluation. The use of Formal Concept Analysis is put forward to leverage these limitations into a new tool to the analysis and comparison of interpretation contexts.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.icdar2013.org/program/competitions

  2. 2.

    http://www.image-net.org/challenges/LSVRC/2012/

  3. 3.

    This is a somewhat strong statement, and in many cases it can be helpful to use these functions anyway, as an instance of common practice in experimental research: “If we cannot immediately solve the global problem, let’s try and solve a more manageable sub-problem.”

  4. 4.

    We are making the implicit assumption that interpretations are mutually exclusive. Although this may seem restrictive, it is not. In cases where multiple interpretations are acceptable, one can simply replace \(\mathcal I\) by \(\left\{ 0,1\right\} ^{\left| I\right| }\).

  5. 5.

    This fuzzy distinction between syntax, semiosis and semantics is actually what troubled interpretation of hieroglyphs [25].

  6. 6.

    Results by Z. Jiang, M.Eng. student at Mines Nancy, France.

  7. 7.

    http://sourceforge.net/projects/lattice-miner/

References

  1. Lamiroy, B., Lopresti, D.: An open architecture for end-to-end document analysis benchmarking. In: 11th International Conference on Document Analysis and Recognition - ICDAR 2011, Beijing, China, pp. 42–47. IEEE Computer Society (2011)

    Google Scholar 

  2. Popper, K.R.: The Logic of Scientific Discovery, Reprint edn. Routledge, New York (1992) (Original edition, 1934 “Logik der Forschung”)

    Google Scholar 

  3. Lamiroy, B., Lopresti, D.: A platform for storing, visualizing, and interpreting collections of noisy documents. In: Fourth Workshop on Analytics for Noisy Unstructured Text Data - AND’10, Toronto, Canada. ACM International Conference Proceeding Series. ACM (2010)

    Google Scholar 

  4. Hu, J., Kashi, R., Lopresti, D., Nagy, G., Wilfong, G.: Why table ground-truthing is hard. In: Proceedings of the Sixth International Conference on Document Analysis and Recognition, Seattle, WA, pp. 129–133, September 2001

    Google Scholar 

  5. Lopresti, D., Nagy, G., Smith, E.B.: Document analysis issues in reading optical scan ballots. In: Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, Boston, MA, USA, pp. 105–112. ACM (2010)

    Google Scholar 

  6. Smith, E.H.B.: An analysis of binarization ground truthing. In: Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, Boston, MA, USA, pp. 27–34. ACM (2010)

    Google Scholar 

  7. Clavelli, A., Karatzas, D., Lladós, J.: A framework for the assessment of text extraction algorithms on complex colour images. In: Proceedings of the 8th IAPR International Workshop on Document Analysis Systems, Boston, MA, USA, pp. 19–26. ACM (2010)

    Google Scholar 

  8. Lopresti, D., Nagy, G.: Issues in ground-truthing graphic documents. In: Proceedings of the Fourth IAPR International Workshop on Graphics Recognition, Kingston, Ontario, Canada, pp. 59–72, September 2001

    Google Scholar 

  9. Eco, U.: The Limits of Interpretation. Indiana University Press, Bloomington (1990)

    Google Scholar 

  10. Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)

    Article  Google Scholar 

  11. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL Visual Object Classes Challenge 2011 (VOC2011) Results (2011). http://www.pascal-network.org/challenges/VOC/voc2011/workshop/index.html

  12. Voorhees, E., Harman, D., et al.: TREC: Experiment and Evaluation in Information Retrieval, vol. 63. MIT press, Cambridge (2005)

    Google Scholar 

  13. Mller, H., Clough, P., Deselaers, T., Caputo, B.: ImageCLEF: Experimental Evaluation in Visual Information Retrieval, 1st edn. Springer, Heidelberg (2010)

    Book  Google Scholar 

  14. Valveny, E., Dosch, P., Fornés, A., Escalera, S.: Report on the third contest on symbol recognition. In: Liu, W., Lladós, J., Ogier, J.-M. (eds.) GREC 2007. LNCS, vol. 5046, pp. 321–328. Springer, Heidelberg (2008). (French Techno-Vision program (Ministry of Research) Spanish project TIN2006-15694-C02-02 Spanish research program Consolider Ingenio 2010:MIPRCV (CSD2007-00018))

    Google Scholar 

  15. Carlotto, M.J.: Effect of errors in ground truth on classification accuracy. Int. J. Remote Sens. 30(18), 4831–4849 (2009)

    Article  Google Scholar 

  16. Lopresti, D.P., Nagy, G.: Adapting the turing test for declaring document analysis problems solved. In: Blumenstein, M., Pal, U., Uchida, S., eds.: Document Analysis Systems, pp. 1–5. IEEE, New York (2012)

    Google Scholar 

  17. Lamiroy, B., Sun, T.: Computing precision and recall with missing or uncertain ground truth. In: Kwon, Y.-B., Ogier, J.-M. (eds.) GREC 2011. LNCS, vol. 7423, pp. 149–162. Springer, Heidelberg (2013)

    Google Scholar 

  18. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  19. Winn, J., Everingham, M.: The PASCAL visual object classes challenge 2007 (VOC2007) annotation guidelines (2007). http://PASCALlin.ecs.soton.ac.uk/challenges/VOC/VOC2007/guidelines.html

  20. Tarantola, A.: Inverse Problem Theory and Methods for Model Parameter Estimation. Society for Industrial Mathematics, Philadelphia (2005)

    Book  MATH  Google Scholar 

  21. Heidegger, M.: Being and Time. Library of Philosophy and Theology. Blackwell, Oxford (1967)

    Google Scholar 

  22. Peirce, C.S.: Syllabus: Nomenclature and Division of Triadic Relations, as far as they are determined. MS [R] 540 (1903)

    Google Scholar 

  23. Eco, U., Collini, S., Culler, J., Rorty, R., Brooke-Rose, C.: Interpretation and Overinterpretation. Tanner Lectures in Human Values. Cambridge University Press, Cambridge (1992)

    Book  Google Scholar 

  24. Eco, U.: Dall’albero al labirinto: studi storici sul segno e l’interpretazione. Bompiani (2007)

    Google Scholar 

  25. Champollion, J.: Précis du système hiéroglyphique des anciens égyptiens, ou recherches sur les élémens premiers de cette écriture sacrée, sur leurs diverses combinaisons, et sur les rapports de ce système avec les autres méthodes graphiques égyptiennes. Imprimerie royale (1828)

    Google Scholar 

  26. Ganter, B., Wille, R.: Formal Concept Analysis - Mathematical Foundations. Springer, Heidelberg (1999)

    Book  MATH  Google Scholar 

  27. Ganter, B., Stumme, G., Wille, R. (eds.): Formal Concept Analysis. LNCS (LNAI), vol. 3626. Springer, Heidelberg (2005)

    Google Scholar 

  28. Coustaty, M., Bertet, K., Visani, M., Ogier, J.M.: A new adaptive structural signature for symbol recognition by using a Galois lattice as a classifier. IEEE Trans. Syst. Man Cybern. B 41(4), 1136–1148 (2011)

    Article  Google Scholar 

  29. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    Article  MathSciNet  Google Scholar 

  30. Niblack, W.: An Introduction to Digital Image Processing. Strandberg Publishing Company, Birkeroed (1985)

    Google Scholar 

  31. Sauvola, J., Pietikäinen, M.: Adaptive document image binarization. Pattern Recogn. 33(2), 225–236 (2000)

    Article  Google Scholar 

  32. Wolf, C., Jolion, J.M., Chassaing, F.: Text localization, enhancement and binarization in multimedia documents. In: Proceedings of the International Conference on Pattern Recognition, vol. 2, pp. 1037–1040 (2002)

    Google Scholar 

  33. Lahcen, B., Kwudia, L.K.: Lattice miner: a tool for concept lattice construction and exploration. In: Supplementary Proceeding of International Conference on Formal concept analysis (ICFCA’10) (2010)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LORIA (UMR 7503), Campus Scientifique, Université de Lorraine, BP 239, 54506, Vandoeuvre-lès-Nancy CEDEX, France

    Bart Lamiroy

Authors
  1. Bart Lamiroy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bart Lamiroy .

Editor information

Editors and Affiliations

  1. Université de Lorraine, Vandoeuvre-lès-Nancy, France

    Bart Lamiroy

  2. Université de La Rochelle, La Rochelle, France

    Jean-Marc Ogier

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lamiroy, B. (2014). Interpretation, Evaluation and the Semantic Gap ... What if We Were on a Side-Track?. In: Lamiroy, B., Ogier, JM. (eds) Graphics Recognition. Current Trends and Challenges. GREC 2013. Lecture Notes in Computer Science(), vol 8746. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-44854-0_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-44854-0_17

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-44853-3

  • Online ISBN: 978-3-662-44854-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation