Automated Semantic Annotation of Species Names in Handwritten Texts

  • Lise StorkEmail author
  • Andreas Weber
  • Jaap van den Herik
  • Aske Plaat
  • Fons Verbeek
  • Katherine Wolstencroft
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11437)


In this paper, scientific species names from images of handwritten species observations are automatically recognised and annotated with semantic concepts, so that they can be used for document retrieval and faceted search. Until now, automated semantic annotation of such named entities was only applied to printed or digital text. We employ a two-step approach. First, word images are classified, identifying elements of scientific species names; Genus, species, author, using (i) visual structural features, (ii) position, and (iii) context. Second, the identified species names are semantically annotated according to the NHC-Ontology, an ontology that describes species observations. Internationalised Resource Identifiers (IRIs) are assigned to the elements so that they can be linked and disambiguated at a later stage by individual researchers. For the identification of scientific species names, we achieve an average F1 score of 0.86. Moreover, we discuss how our method will function in a semi-automated annotation process, with a fruitful dialogue between system and user as the main objective.


Deep learning Ontologies Taxonomy Scientific names Semantic annotation Historical biodiversity research 



This work is supported by the Netherlands Organisation for Scientific Research (NWO), grant 652.001.001, and Brill publishers.


  1. 1.
    Adak, C., Chaudhuri, B.B., Blumenstein, M.: Named entity recognition from unstructured handwritten document images. In: 12th IAPR Workshop on Document Analysis Systems (DAS), pp. 375–380. IEEE (2016)Google Scholar
  2. 2.
    Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw. 106, 249–259 (2018)CrossRefGoogle Scholar
  3. 3.
    Bulacu, M., van Koert, R., Schomaker, L., van der Zant, T.: Layout analysis of handwritten historical documents for searching the archive of the cabinet of the Dutch queen. In: Proceedings of the Ninth International Conference on Document Analysis and Recognition, vol. 1, 2, pp. 357–361. IEEE (2007)Google Scholar
  4. 4.
    Canfield, M.R.: Field Notes on Science & Nature. Harvard University Press, Cambridge (2011)CrossRefGoogle Scholar
  5. 5.
    Carbonell, M., Villegas, M., Fornés, A., Lladós, J.: Joint recognition of handwritten text and named entities with a neural end-to-end model. arXiv preprint arXiv:1803.06252 (2018)
  6. 6.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the 2009 Computer Vision and Pattern Recognition, CVPR, pp. 248–255. IEEE (2009)Google Scholar
  7. 7.
    Drew, J.A., Moreau, C.S., Stiassny, M.L.: Digitization of museum collections holds the potential to enhance researcher diversity. Nature Ecol. Evol. 1(12), 1789–1790 (2017)CrossRefGoogle Scholar
  8. 8.
    Drinkwater, R.E., Cubey, R.W., Haston, E.M.: The use of optical character recognition (OCR) in the digitisation of herbarium specimen labels. PhytoKeys 38, 15–30 (2014)CrossRefGoogle Scholar
  9. 9.
    Gwinn, N.E., Rinaldo, C.: The biodiversity heritage library: sharing biodiversity literature with the world. IFLA J. 35(1), 25–34 (2009)CrossRefGoogle Scholar
  10. 10.
    Heidorn, P.B., Wei, Q.: Automatic metadata extraction from museum specimen labels. In: Proceedings of the 2008 International Conference on Dublin Core and Metadata Applications, pp. 57–68 (2008)Google Scholar
  11. 11.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  12. 12.
    Kennedy, J.B., Kukla, R., Paterson, T.: Scientific names are ambiguous as identifiers for biological taxa: their context and definition are required for accurate data integration. In: Ludäscher, B., Raschid, L. (eds.) DILS 2005. LNCS, vol. 3615, pp. 80–95. Springer, Heidelberg (2005). Scholar
  13. 13.
    Koning, D., Sarkar, I.N., Moritz, T.: Taxongrab: extracting taxonomic names from text. Biodivers. Inf. 2, 79–82 (2005)Google Scholar
  14. 14.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  15. 15.
    MacGregor, A. (ed.): Naturalists in the Field. Brill, Leiden (2018)Google Scholar
  16. 16.
    Chollet, F., et al.: Keras (2015).
  17. 17.
    McCallum, A., Li, W.: Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In: Proceedings of the Seventh Conference on Natural Language Learning, vol. 4, pp. 188–191. Association for Computational Linguistics (2003)Google Scholar
  18. 18.
    Miracle, M.E.G.: On whose authority? Temminck’s debates on zoological classification and nomenclature: 1820–1850. J. Hist. Biol. 44(3), 445–481 (2011)CrossRefGoogle Scholar
  19. 19.
    Müller-Wille, S.: Names and numbers: “data” in classical natural history, 1758–1859. Osiris 32(1), 109–128 (2017)CrossRefGoogle Scholar
  20. 20.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)Google Scholar
  21. 21.
    Page, L.M., MacFadden, B.J., Fortes, J.A., Soltis, P.S., Riccardi, G.: Digitization of biodiversity collections reveals biggest data on biodiversity. BioScience 65(9), 841–842 (2015)CrossRefGoogle Scholar
  22. 22.
    Sarkar, I.N.: Biodiversity informatics: organizing and linking information across the spectrum of life. Briefings Bioinform. 8(5), 347–357 (2007)CrossRefGoogle Scholar
  23. 23.
    Schomaker, L.: Design considerations for a large-scale image-based text search engine in historical manuscript collections. It - Inf. Technol. 58(2), 80–88 (2016)Google Scholar
  24. 24.
    Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)CrossRefGoogle Scholar
  25. 25.
    Shi, Z.: Datefinder: detecting date regions on handwritten document images based on positional expectancy. Master’s thesis, University of Groningen, Groningen, the Netherlands (2016)Google Scholar
  26. 26.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556 (2014)Google Scholar
  27. 27.
    Stork, L., et al.: Semantic annotation of natural history collections. Web Semant. Sci. Serv. Agents World Wide Web (2018).
  28. 28.
    Toledo, J.I., Sudholt, S., Fornés, A., Cucurull, J., Fink, G.A., Lladós, J.: Handwritten word image categorization with convolutional neural networks and spatial pyramid pooling. In: Robles-Kelly, A., Loog, M., Biggio, B., Escolano, F., Wilson, R. (eds.) S+SSPR 2016. LNCS, vol. 10029, pp. 543–552. Springer, Cham (2016). Scholar
  29. 29.
    Van der Zant, T., Schomaker, L., Haak, K.: Handwritten-word spotting using biologically inspired features. IEEE Trans. Pattern Anal. Mach. Intell. 30(11), 1945–1957 (2008)CrossRefGoogle Scholar
  30. 30.
    van Oosten, J.-P., Schomaker, L.: Separability versus prototypicality in handwritten word-image retrieval. Pattern Recogn. 47(3), 1031–1038 (2014)CrossRefGoogle Scholar
  31. 31.
    Weber, A., Ameryan, M., Wolstencroft, K., Stork, L., Heerlien, M., Schomaker, L.: Towards a digital infrastructure for illustrated handwritten archives. In: Ioannides, M. (ed.) Digital Cultural Heritage. LNCS, vol. 10605, pp. 155–166. Springer, Cham (2018). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Leiden Institute of Advanced Computer ScienceLeidenthe Netherlands
  2. 2.Leiden Centre of Data ScienceLeidenthe Netherlands
  3. 3.University of TwenteEnschedethe Netherlands

Personalised recommendations