Skip to main content

Advertisement

SpringerLink
Log in

BiLSTM-CRF for geological named entity recognition from the geoscience literature

  • Research Article
  • Published:
Earth Science Informatics Aims and scope Submit manuscript

Abstract

Many detailed geoscience reports lie unused, offering both challenges and opportunities for information extraction. In geoscience research, geological named entity recognition (GNER) is an important task in the field of geoscience information extraction. Regarding numerical geoscience data, research on information extraction remains limited. Most conventional NER approaches are heavily dependent on feature engineering, and such sentence-level-based methods suffer from the tagging inconsistency problem. Based on the above observations, this paper proposes a neural network approach, namely, attention-based bidirectional long short-term memory with a conditional random field layer (Att-BiLSTM-CRF), for name entity recognition to extract information entities describing geoscience information from geoscience reports. This approach leverages global information learned from an attention mechanism to enforce tagging consistency across multiple instances of the same token in a document. Experiments on the constructed dataset show that our method achieves comparable performance to that of other state-of-the-art systems. Additionally, our method achieved an average F1 score of 91.47% in the NER extraction task.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Arnab A, Jayasumana S, Zheng S, & Torr, PH (2016) Higher order conditional random fields in deep neural networks. European Conference on Computer Vision 524–540

  • Babaie HA, Davarpanah A (2018) Semantic modeling of plastic deformation of polycrystalline rock. Comput Geosci 111:213–222

    Article  Google Scholar 

  • Bengio Y, Simard P, Frasconi P (1994) Learning long-term dependencies with gradient descent is difficult. IEEE Trans Neural Netw 5(2):157–166

    Article  Google Scholar 

  • Cernuzzi L, Pane J (2014) Toward open government in Paraguay[J]. It Professional 16(5):62–64

    Article  Google Scholar 

  • Chen X, Shi Z, Qiu X, et al (2017) Adversarial multi-criteria learning for Chinese word segmentation. arXiv, arXiv:1193–1203

  • Chiu J P C, Nichols E (2016) Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics 4(1):357–370

    Article  Google Scholar 

  • Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa PP (2011) Natural language processing (almost) from scratch. J Mach Learn Res 12:2493–2537

    Google Scholar 

  • Cracknell MJ, Reading AM (2014) Geological mapping using remote sensing data: a comparison of five machine learning algorithms, their response to variations in the spatial distribution of training data and the use of explicit spatial information. Comput Geosci 63(1):22–33

    Article  Google Scholar 

  • Elman JL (1990) Finding structure in time. Cogn Sci 14(2):179–211

    Article  Google Scholar 

  • Eltyeb S Salim N (2014) Chemical named entities recognition: a review on approaches and applications. J Cheminform 6:17

  • Finkel J, Dingare S, Nguyen H et al (2004) Exploiting context for biomedical entity recognition: from syntax to the web. In: Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications. Association for Computational Linguistics 88–91

  • Gao J, Li M, Huang CN, Wu A (2005) Chinese word segmentation and named entity recognition: a pragmatic approach. Computational Linguistics 31(4):531–574

    Article  Google Scholar 

  • Habibi M, Weber L, Neves M, Wiegandt DL, Leser U (2017) Deep learning with word embeddings improves biomedical named entity recognition. Bioinformatics 33(14):i37–i48

    Article  Google Scholar 

  • He H, Sun X (2017) A unified model for cross-domain and semi-supervised named entity recognition in Chinese social media. In Thirty-First AAAI Conference on Artificial Intelligence 3216–3222

  • He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • Hettne KM et al (2009) A dictionary to identify small molecules and drugs in free text. Bioinformatics 25:2983–2991

    Article  Google Scholar 

  • Hinton G, Deng L, Yu D, Dahl G, Mohamed AR, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath T, Kingsbury B (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  • Huang L, Du Y, Chen G (2015) GeoSegmenter: a statistically learned Chinese word segmenter for the geoscience domain. Comput Geosci 76:11–17

    Article  Google Scholar 

  • Li L, Liu Y, Zhu H, Ying S, Luo Q, Luo H, Kuai X, Xia H, Shen H (2017) A bibliometric and visual analysis of global geo-ontology research. Comput Geosci 99:1–8

    Article  Google Scholar 

  • Lima LA, Görnitz N, Varella LE, Vellasco M, Müller KR, Nakajima S (2017) Porosity estimation by semi-supervised learning with sparsely available labeled samples. Comput Geosci 106:33–48

    Article  Google Scholar 

  • Liu S et al (2015) Drug name recognition: approaches and resources. Information 6:790–810

    Article  Google Scholar 

  • Ma X, Fox P (2013) Recent progress on geologic time ontologies and considerations for future works. Earth Sci Inf 6(1):31–46

    Article  Google Scholar 

  • Ma X, Hovy E (2016) End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354

  • Ma X, Carranza EJM, Wu C, van der Meer FD (2012) Ontology-aided annotation, visualization, and generalization of geological time-scale information from online geological map services. Comput Geosci 40:107–119

    Article  Google Scholar 

  • Ma X, Hummer D, Golden JJ et al (2017) Using visual exploratory data analysis to facilitate collaboration and hypothesis generation in cross-disciplinary research. ISPRS Int J Geo Inf 6(11):368

    Article  Google Scholar 

  • Mastella LS, Abel M, De Ros LF et al (2007) Event ordering reasoning ontology applied to petrology and geological modelling. In: Theoretical Advances and applications of fuzzy logic and soft computing. Springer, Berlin, pp 465–475

    Google Scholar 

  • Mikolov T, Kombrink S, Burget L, et al (2011) Extensions of recurrent neural network language model. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, pp 5528–5531

  • Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

  • Nawroth C, Schmedding M, Brocks H, Kaufmann M, Fuchs M, Hemmje M (2015) Towards cloud-based knowledge capturing based on natural language processing. Procedia Computer Science 68:206–216

    Article  Google Scholar 

  • Pennington J, Socher R, Glove MC (2014) Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  • Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multimodal sentiment analysis. Neurocomputing 261:217–230

    Article  Google Scholar 

  • Rei M, Crichton G K, Pyysalo S (2016) Attending to characters in neural sequence labeling models. In: International Conference on Computational Linguistics, pp 309–318

  • Rumelhart D, Hinton G, Williams R (1986) Learning representations by back-propagating errors. Nature 323:533–536

    Article  Google Scholar 

  • Santos R, Flores PM, Calado P et al (2017) Toponym matching through deep neural networks. Int J Geogr Inf Sci (3):1–25

  • Sarkar K, Shaw SK (2017) A memory-based learning approach for named entity recognition in Hindi. J Intell Syst 26(2):301–321

    Google Scholar 

  • Shen Y, Yun H, Lipton ZC et al (2017) Deep Active learning for named entity recognition. arXiv preprint arXiv:1707.05928

  • Sutskever I, Vinyals O, Le Q V (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112

  • Tsochantaridis I, Joachims T, Hofmann T et al (2005) Large margin methods for structured and interdependent output variables. J Mach Learn Res 6(2):1453–1484

    Google Scholar 

  • Unanue IJ, Borzeshi EZ, Piccardi M (2017) Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition. J Biomed Inform 76:102–109

    Article  Google Scholar 

  • Viterbi A (1967) Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transinformattheory 13(2):260–269

    Google Scholar 

  • Wang C, Chen J, Xiao F (2016a) Application of empirical model decomposition and independent component analysis to magnetic anomalies separation: a case study for Gobi Desert coverage in eastern tian Shan, China. In: Geostatistical and geospatial approaches for the characterization of natural resources in the environment. Springer, Cham, pp 593–598

    Chapter  Google Scholar 

  • Wang C, Chen J, Xiao F, Fode T, Li L (2016b) Radioelement distributions and analysis of micro topographical influences in a shallow covered area, Inner Mongolia, China: implications for mineral exploration. J Appl Geophys 133:62–69

    Article  Google Scholar 

  • Wang C, Ma X, Chen J (2018a) Ontology-driven data integration and visualization for exploring regional geologic time and paleontological information. Comput Geosci 115:12–19

    Article  Google Scholar 

  • Wang C, Ma X, Chen J, Chen J (2018b) Information extraction and knowledge graph construction from geoscience literature. Comput Geosci 112:112–120

    Article  Google Scholar 

  • Werbos PJ (1988) Generalization of backpropagation with application to a recurrent gas market model. Neural Netw 1(4):339–356

    Article  Google Scholar 

  • Wu L, Xue L, Li C, Lv X, Chen Z, Jiang B, Guo M, Xie Z (2017) A knowledge-driven geospatially enabled framework for geological big data. ISPRS Int J Geo Inf 6(6):166

    Article  Google Scholar 

  • Xiao F, Chen Z, Chen J, Zhou Y (2016) A batch sliding window method for local singularity mapping and its application for geochemical anomaly identification. Comput Geosci 90(PA):189–201

    Article  Google Scholar 

  • Xie S, Girshick R, Dollár P et al (2017) Aggregated residual transformations for deep neural networks. In: 2017 IEEE conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 5987–5995

  • Yang Z, Salakhutdinov R, Cohen WW (2017) Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint arXiv:1703.06345

  • Zeng D, Sun C, Lin L et al (2017) Lstm-crf for drug-named entity recognition. Entropy 19(6):283

    Article  Google Scholar 

  • Zhong J, Aydina A, McGuinness DL (2009) Ontology of fractures. J Struct Geol 31(3):251–259

    Article  Google Scholar 

  • Zhu Q, Li X, Conesa A et al (2017) GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics:btx815

  • Zhu Q, Li X, Conesa A, Pereira C (2018) GRAM-CNN: a deep learning approach with local context for named entity recognition in biomedical text. Bioinformatics 34(9):1547–1554

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank the Kai Ma and anonymous reviewers for carefully reading this paper and their very useful comments. This study was financially supported by the National Key Research and Development Program (2018YFB0505500, 2018YFB0505504, 2017YFB0503600, 2017YFC0602204), the National Natural Science Foundation of China (41871311, 41671400, 41871305).

Author information

Authors and Affiliations

  1. School of Geography and Information Engineering, China University of Geosciences, Wuhan, 430074, China

    Qinjun Qiu, Zhong Xie, Liang Wu, Liufeng Tao & Wenjia Li

  2. National Engineering Research Center of Geographic Information System, Wuhan, 430074, China

    Qinjun Qiu, Zhong Xie, Liang Wu, Liufeng Tao & Wenjia Li

Authors
  1. Qinjun Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhong Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liufeng Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenjia Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liang Wu.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Communicated by: H. Babaie

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiu, Q., Xie, Z., Wu, L. et al. BiLSTM-CRF for geological named entity recognition from the geoscience literature. Earth Sci Inform 12, 565–579 (2019). https://doi.org/10.1007/s12145-019-00390-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12145-019-00390-3

Keywords

Search

Navigation