Advertisement

Earth Science Informatics

, Volume 12, Issue 4, pp 565–579 | Cite as

BiLSTM-CRF for geological named entity recognition from the geoscience literature

  • Qinjun Qiu
  • Zhong Xie
  • Liang WuEmail author
  • Liufeng Tao
  • Wenjia Li
Research Article
  • 100 Downloads

Abstract

Many detailed geoscience reports lie unused, offering both challenges and opportunities for information extraction. In geoscience research, geological named entity recognition (GNER) is an important task in the field of geoscience information extraction. Regarding numerical geoscience data, research on information extraction remains limited. Most conventional NER approaches are heavily dependent on feature engineering, and such sentence-level-based methods suffer from the tagging inconsistency problem. Based on the above observations, this paper proposes a neural network approach, namely, attention-based bidirectional long short-term memory with a conditional random field layer (Att-BiLSTM-CRF), for name entity recognition to extract information entities describing geoscience information from geoscience reports. This approach leverages global information learned from an attention mechanism to enforce tagging consistency across multiple instances of the same token in a document. Experiments on the constructed dataset show that our method achieves comparable performance to that of other state-of-the-art systems. Additionally, our method achieved an average F1 score of 91.47% in the NER extraction task.

Keywords

Geological name entity recognition Attention mechanism Prior knowledge Word embedding 

Notes

Acknowledgments

We would like to thank the Kai Ma and anonymous reviewers for carefully reading this paper and their very useful comments. This study was financially supported by the National Key Research and Development Program (2018YFB0505500, 2018YFB0505504, 2017YFB0503600, 2017YFC0602204), the National Natural Science Foundation of China (41871311, 41671400, 41871305).

Compliance with ethical standards

Conflict of interest

The authors declare no conflict of interest.

References

  1. Arnab A, Jayasumana S, Zheng S, & Torr, PH (2016) Higher order conditional random fields in deep neural networks. European Conference on Computer Vision 524–540Google Scholar
  2. Babaie HA, Davarpanah A (2018) Semantic modeling of plastic deformation of polycrystalline rock. Comput Geosci 111:213–222CrossRefGoogle Scholar
  3. Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166CrossRefGoogle Scholar
  4. Cernuzzi L, Pane J (2014) Toward open government in Paraguay[J]. It Professional 16(5):62–64CrossRefGoogle Scholar
  5. Chen X, Shi Z, Qiu X, et al (2017) Adversarial multi-criteria learning for Chinese word segmentation. arXiv, arXiv:1193–1203Google Scholar
  6. Chiu J P C, Nichols E (2016) Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics 4(1):357–370CrossRefGoogle Scholar
  7. Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa PP (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537Google Scholar
  8. Cracknell MJ, Reading AM (2014) Geological mapping using remote sensing data: a comparison of five machine learning algorithms, their response to variations in the spatial distribution of training data and the use of explicit spatial information. Comput Geosci 63(1):22–33CrossRefGoogle Scholar
  9. Elman JL (1990) Finding structure in time. Cogn Sci 14(2):179–211CrossRefGoogle Scholar
  10. Eltyeb S Salim N (2014) Chemical named entities recognition: a review on approaches and applications. J Cheminform 6:17Google Scholar
  11. Finkel J, Dingare S, Nguyen H et al (2004) Exploiting context for biomedical entity recognition: from syntax to the web. In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications. Association for Computational Linguistics 88–91Google Scholar
  12. Gao J, Li M, Huang CN, Wu A (2005) Chinese word segmentation and named entity recognition: a pragmatic approach. Computational Linguistics 31(4):531–574CrossRefGoogle Scholar
  13. Habibi M, Weber L, Neves M, Wiegandt DL, Leser U (2017) Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14):i37–i48CrossRefGoogle Scholar
  14. He H, Sun X (2017) A unified model for cross-domain and semi-supervised named entity recognition in Chinese social media. In Thirty-First AAAI Conference on Artificial Intelligence 3216–3222Google Scholar
  15. He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  16. Hettne KM et al (2009) A dictionary to identify small molecules and drugs in free text. Bioinformatics 25:2983–2991CrossRefGoogle Scholar
  17. Hinton G, Deng L, Yu D, Dahl G, Mohamed AR, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath T, Kingsbury B (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97CrossRefGoogle Scholar
  18. Huang L, Du Y, Chen G (2015) GeoSegmenter: a statistically learned Chinese word segmenter for the geoscience domain. Comput Geosci 76:11–17CrossRefGoogle Scholar
  19. Li L, Liu Y, Zhu H, Ying S, Luo Q, Luo H, Kuai X, Xia H, Shen H (2017) A bibliometric and visual analysis of global geo-ontology research. Comput Geosci 99:1–8CrossRefGoogle Scholar
  20. Lima LA, Görnitz N, Varella LE, Vellasco M, Müller KR, Nakajima S (2017) Porosity estimation by semi-supervised learning with sparsely available labeled samples. Comput Geosci 106:33–48CrossRefGoogle Scholar
  21. Liu S et al (2015) Drug name recognition: approaches and resources. Information 6:790–810CrossRefGoogle Scholar
  22. Ma X, Fox P (2013) Recent progress on geologic time ontologies and considerations for future works. Earth Sci Inf 6(1):31–46CrossRefGoogle Scholar
  23. Ma X, Hovy E (2016) End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354Google Scholar
  24. Ma X, Carranza EJM, Wu C, van der Meer FD (2012) Ontology-aided annotation, visualization, and generalization of geological time-scale information from online geological map services. Comput Geosci 40:107–119CrossRefGoogle Scholar
  25. Ma X, Hummer D, Golden JJ et al (2017) Using visual exploratory data analysis to facilitate collaboration and hypothesis generation in cross-disciplinary research. ISPRS Int J Geo Inf 6(11):368CrossRefGoogle Scholar
  26. Mastella LS, Abel M, De Ros LF et al (2007) Event ordering reasoning ontology applied to petrology and geological modelling. In: Theoretical Advances and applications of fuzzy logic and soft computing. Springer, Berlin, pp 465–475Google Scholar
  27. Mikolov T, Kombrink S, Burget L, et al (2011) Extensions of recurrent neural network language model. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 5528–5531Google Scholar
  28. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781Google Scholar
  29. Nawroth C, Schmedding M, Brocks H, Kaufmann M, Fuchs M, Hemmje M (2015) Towards cloud-based knowledge capturing based on natural language processing. Procedia Computer Science 68:206–216CrossRefGoogle Scholar
  30. Pennington J, Socher R, Glove MC (2014) Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543Google Scholar
  31. Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230CrossRefGoogle Scholar
  32. Rei M, Crichton G K, Pyysalo S (2016) Attending to characters in neural sequence labeling models. In: International Conference on Computational Linguistics, pp 309–318Google Scholar
  33. Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323:533–536CrossRefGoogle Scholar
  34. Santos R, Flores PM, Calado P et al (2017) Toponym matching through deep neural networks. Int J Geogr Inf Sci (3):1–25Google Scholar
  35. Sarkar K, Shaw SK (2017) A memory-based learning approach for named entity recognition in Hindi. J Intell Syst 26(2):301–321Google Scholar
  36. Shen Y, Yun H, Lipton ZC et al (2017) Deep Active learning for named entity recognition. arXiv preprint arXiv:1707.05928Google Scholar
  37. Sutskever I, Vinyals O, Le Q V (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112Google Scholar
  38. Tsochantaridis I, Joachims T, Hofmann T et al (2005) Large margin methods for structured and interdependent output variables. J Mach Learn Res 6(2):1453–1484Google Scholar
  39. Unanue IJ, Borzeshi EZ, Piccardi M (2017) Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition. J Biomed Inform 76:102–109CrossRefGoogle Scholar
  40. Viterbi A (1967) Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transinformattheory 13(2):260–269Google Scholar
  41. Wang C, Chen J, Xiao F (2016a) Application of empirical model decomposition and independent component analysis to magnetic anomalies separation: a case study for Gobi Desert coverage in eastern tian Shan, China. In: Geostatistical and geospatial approaches for the characterization of natural resources in the environment. Springer, Cham, pp 593–598CrossRefGoogle Scholar
  42. Wang C, Chen J, Xiao F, Fode T, Li L (2016b) Radioelement distributions and analysis of micro topographical influences in a shallow covered area, Inner Mongolia, China: implications for mineral exploration. J Appl Geophys 133:62–69CrossRefGoogle Scholar
  43. Wang C, Ma X, Chen J (2018a) Ontology-driven data integration and visualization for exploring regional geologic time and paleontological information. Comput Geosci 115:12–19CrossRefGoogle Scholar
  44. Wang C, Ma X, Chen J, Chen J (2018b) Information extraction and knowledge graph construction from geoscience literature. Comput Geosci 112:112–120CrossRefGoogle Scholar
  45. Werbos PJ (1988) Generalization of backpropagation with application to a recurrent gas market model. Neural Netw 1(4):339–356CrossRefGoogle Scholar
  46. Wu L, Xue L, Li C, Lv X, Chen Z, Jiang B, Guo M, Xie Z (2017) A knowledge-driven geospatially enabled framework for geological big data. ISPRS Int J Geo Inf 6(6):166CrossRefGoogle Scholar
  47. Xiao F, Chen Z, Chen J, Zhou Y (2016) A batch sliding window method for local singularity mapping and its application for geochemical anomaly identification. Comput Geosci 90(PA):189–201CrossRefGoogle Scholar
  48. Xie S, Girshick R, Dollár P et al (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5987–5995Google Scholar
  49. Yang Z, Salakhutdinov R, Cohen WW (2017) Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint arXiv:1703.06345Google Scholar
  50. Zeng D, Sun C, Lin L et al (2017) Lstm-crf for drug-named entity recognition. Entropy 19(6):283CrossRefGoogle Scholar
  51. Zhong J, Aydina A, McGuinness DL (2009) Ontology of fractures. J Struct Geol 31(3):251–259CrossRefGoogle Scholar
  52. Zhu Q, Li X, Conesa A et al (2017) GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics:btx815Google Scholar
  53. Zhu Q, Li X, Conesa A, Pereira C (2018) GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics 34(9):1547–1554CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Qinjun Qiu
    • 1
    • 2
  • Zhong Xie
    • 1
    • 2
  • Liang Wu
    • 1
    • 2
    Email author
  • Liufeng Tao
    • 1
    • 2
  • Wenjia Li
    • 1
    • 2
  1. 1.School of Geography and Information EngineeringChina University of GeosciencesWuhanChina
  2. 2.National Engineering Research Center of Geographic Information SystemWuhanChina

Personalised recommendations