Where Corpus Linguistics and Artificial Intelligence (AI) Meet



This chapter will provide a platform to showcase the more recent developments that have grown out of the early laid groundwork. The latest theories in the field of linguistics will be presented, based on empirical data taken from naturally occurring language. In particular, the lexical priming theory will be introduced as a way to explain structures of language that corpus linguists have uncovered. Furthermore, the chapter will discuss the development of increasingly sophisticated algorithms that also deal with the use of language. Here, the focus will be on key achievements in the 1980s by IBM which created a solid foundation for applications that are now widely used in mobile and desktop devices—namely “assistants” like Amazon’s Echo, Apple’s SIRI or Google’s (and Android’s) Google Go.


Hoey Quillian Norvig Lexical priming LSTM N-gram model Digital translators 


  1. Cambria, Erik, and White Bebo. 2014. Jumping NLP Curves: A Review of Natural Language Processing Research. EEE Computational Intelligence Magazine 9 (2): 48–57.CrossRefGoogle Scholar
  2. Canhasi, Ercan. 2016. GSolver: Artificial Solver of Word Association Game. In ICT Innovations 2015, ed. Suzana Loshkovska and Saso Koceski, 49–57. Cham: Springer. CrossRefGoogle Scholar
  3. Carroll, Glenn, and Eugene Charniak. 1991. A Probabilistic Analysis of Marker-Passing Techniques for Plan-Recognition. In Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, August, 69–76. Morgan Kaufmann Publishers Inc.CrossRefGoogle Scholar
  4. Charniak, Eugene. 1972. Toward a Model of Children’s Story Comprehension. AI-Tech, Rep-266. Cambridge, MA: MIT AI Labs.Google Scholar
  5. Charniak, Eugene. 1986. A Neat Theory of Marker Passing. AAAI, 584–588.Google Scholar
  6. Charniak, Eugene, and Robert Goldman. 1988. A Logic for Semantic Interpretation. In Proceedings of the 26th Annual Meeting on Association for Computational Linguistics, 87–94. Association for Computational Linguistics.Google Scholar
  7. Clark, Stephen. 2015. Vector Space Models of Lexical Meaning. In Handbook of Contemporary Semantic Theory, ed. Shalom Lappin and Chris Fox, 493–522. New York: Wiley. CrossRefGoogle Scholar
  8. Collins, Allan M., and Elizabeth F. Loftus. 1975. A Spreading-Activation Theory of Semantic Processing. Psychological Review 82 (6): 407–428.CrossRefGoogle Scholar
  9. Collobert, Ronan, and Jason Weston. 2008. A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. In Proceedings of the 25th International Conference on Machine Learning, 160–167. Helsinki, Finland: ACM.Google Scholar
  10. Damavandi, Babak, Shankar Kumar, Noam Shazeer, and Antoine Bruguier. 2016. NN-Grams: Unifying Neural Network and N-Gram Language Models for Speech Recognition. arXiv preprint arXiv:1606.07470.
  11. Das, Dipanjan, Desai Chen, André F.T. Martins, Nathan Schneider, and Noah A. Smith. 2014. Frame-Semantic Parsing. Computational Linguistics 40 (1): 9–56.CrossRefGoogle Scholar
  12. Erk, Katrin, and Sebastian Padó. 2008. A Structured Vector Space Model for Word Meaning in Context. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 897–906. Association for Computational Linguistics.Google Scholar
  13. Graves, Alex, Greg Wayne, and Ivo Danihelka. 2014. Neural Turing Machines. arXiv preprint arXiv:1410.5401.
  14. Harabagiu, Sanda M., and Dan I. Moldovan. 1997. Parallel Inference on a Linguistic Knowledge Base. In Parallel Processing Symposium, 1997. Proceedings, 11th International, 204–208. IEEE.Google Scholar
  15. Harrington, Brian. 2010. A Semantic Network Approach to Measuring Relatedness. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, 356–364.Google Scholar
  16. Henderson, Matthew. 2015. Machine Learning for Dialog State Tracking: A Review. Machine Learning in Spoken Language Processing Workshop. Last Accessed 11/2017.
  17. Hermann, Karl Moritz, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. Advances in Neural Information Processing Systems, 1693–1701.Google Scholar
  18. Hirschberg, Julia, and Christopher D. Manning. 2015. Advances in Natural Language Processing. Science 349 (6245): 261–266.CrossRefGoogle Scholar
  19. Hobbs, Jerry R., Mark Stickel, Paul Martin, and Douglas Edwards. 1988. Interpretation as Abduction. In Proceedings of the 26th Annual Meeting on Association for Computational Linguistics, 95–103. Association for Computational Linguistics.Google Scholar
  20. Hochreiter, Sepp, and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9 (8): 1735–1780.CrossRefGoogle Scholar
  21. Hoey, Michael. 1991. Patterns of Lexis in Text. Oxford: Oxford University Press.Google Scholar
  22. Hoey, Michael. 1995. The Lexical Nature of Intertextuality: A Preliminary Study. In Organization in Discourse: Proceedings from the Turku Conference, ed. B. Warvik, S. Tanskanen, and R. Hiltunen, 73–94. Anglicana Turkuensia 14.Google Scholar
  23. Hoey, Michael. 2005. Lexical Priming: A New Theory of Words and Language. London: Routledge.CrossRefGoogle Scholar
  24. Hoey, Michael. 2008. Lexical Priming and Literary Creativity. In Text, Discourse and Corpora, ed. M. Hoey, M. Mahlberg, M. Stubbs, and W. Teubert, 7–30. London: Continuum.Google Scholar
  25. Hoey, Michael. 2017. Cohesion and Coherence in a Content-Specific Corpus. In Lexical Priming: Applications and Advances, ed. M. Pace-Sigge and K. J. Patterson, 3–40. Amsterdam: John Benjamins. Google Scholar
  26. Jantunen, Jarmo Harri. 2017. Lexical and Morphological Priming. In Lexical Priming: Applications and Advances, ed. M. Pace-Sigge and K. J. Patterson, 253–272. Amsterdam: John Benjamins.Google Scholar
  27. Jantunen, Jarmo Harri, and Sisko Brunni. 2013. Morphology, Lexical Priming and Second Language Acquisition: A Corpus-Study on Learner Finnish. In Twenty Years of Learner Corpus Research: Looking Back, Moving Ahead, ed. Sylviane Granger, Gaëtanelle Gilquin, and Fanny Meunier, pp. 235–245. Louvain-la-Neuve: Presses universitaires de Louvain.Google Scholar
  28. Jean, Sébastien, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On Using Very Large Target Vocabulary for Neural Machine Translation. arXiv preprint arXiv:1412.2007.
  29. Johnson, Melvin, M. Schuster, Q.V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, and M. Hughes. 2016. Googles Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. arXiv preprint arXiv:1611.04558.
  30. Jozefowicz, Rafal, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410.
  31. Kaiser, Lukasz, Aidan N. Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. 2017. One Model to Learn Them All. arXiv preprint arXiv:1706.05137.
  32. Lehmann, Fritz. 1992. Semantic Networks. Computers & Mathematics with Applications 23 (2–5): 1–50.CrossRefGoogle Scholar
  33. Leviathan, Yanviv and Matias, Yossi. 2018. Google Duplex: An AI System for Accomplishing Real World Tasks Over the Phone. Google AI Blog. Last Accessed 09/2018.
  34. Lewis, Mike, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. 2018, Forthcoming. Deal or No Deal? End-to-End Learning for Negotiation Dialogues. arXiv:1706.05125.
  35. Li, Jiwei, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep Reinforcement Learning for Dialogue Generation. arXiv preprint arXiv:1606.01541.
  36. Louw, Bill. 1993. Irony in the Text or Insincerity in the Writer? The Diagnostic Potential of Semantic Prosodies. In Text and Technology, ed. M. Baker, G. Francis, and E. Tognini-Bonelli, 157–176. Amsterdam: Benjamins.CrossRefGoogle Scholar
  37. Luong, Minh-Thang, and Christopher D. Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models. arXiv preprint arXiv:1604.00788.
  38. Mac an tSaoir, Ronan. 2014. Using Spreading Activation to Evaluate and Improve Ontologies. COLING, 2237–2248.Google Scholar
  39. Manin, Yuri I., and Matilde Marcolli. 2016. Semantic Spaces. Mathematics in Computer Science 10 (4): 459–477.CrossRefGoogle Scholar
  40. Manning, Chris (with Richard Socher). 2017. Natural Language Processing with Deep Learning CS224N/Ling284. Lecture 11. Stanford University.Google Scholar
  41. Mikolov, Tomáš, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent Neural Network Based Language Model. Interspeech 2: 3–10.Google Scholar
  42. Mikolov, Tomáš, Stefan Kombrink, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2011. Extensions of Recurrent Neural Network Language Model. Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, 5528–5531.Google Scholar
  43. Mikolov, Tomas, and Geoffrey Zweig. 2012. Context Dependent Recurrent Neural Network Language Model. Microsoft Research Technical Report MSR-TR-2012-92, 234–239. Google Scholar
  44. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv preprint arXiv:1301.3781.
  45. Miller, George A. 1956. The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information. Psychological Review, 63 (2): 81–97. CrossRefGoogle Scholar
  46. Neely, James H. 1976. Semantic Priming and Retrieval from Lexical Memory: Evidence for Facilitatory and Inhibitory Processes. Memory and Cognition 4 (5): 648–654.CrossRefGoogle Scholar
  47. Noordman-Vonk, Wietske. 1979. Retrieval from Semantic Memory. Berlin, Heidelberg: Springer.CrossRefGoogle Scholar
  48. Norvig, P. 1983. Frame Activated Inferences in a Story Understanding Program. International Joint Conference on Artificial Intelligence (IJCAI), 624–626.Google Scholar
  49. Norvig, P. 1987. A Unified Theory of Inference for Text Understanding. PhD thesis, University of California, Berkeley.Google Scholar
  50. Norvig, P. 1989a. Marker Passing as a Weak Method for Text Inferencing. Cognitive Science 13 (4): 569–620.Google Scholar
  51. Norvig, P. 1989b. Building a Large Lexicon with Lexical Network Theory. In Proceedings of the IJCAI Workshop on Lexical Acquisition, 1–12.Google Scholar
  52. Norvig, P. 1992. Story Analysis. In Encyclopedia of AI, ed. Stuart Shapiro. New Jersey: Wiley.Google Scholar
  53. Norvig, P. 2011. On Chomsky and the Two Cultures of Statistical Learning. On-Line Essay in Response to Chomskys Remarks. Available from Last Accessed 11/2017.
  54. Och, Franz Josef. 2003. Minimum Error Rate Training in Statistical Machine Translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, vol. 1, 160–167. Association for Computational Linguistics.Google Scholar
  55. Och, Franz Josef, and Hermann Ney. 2002. Discriminative Training and Maximum Entropy Models for Statistical Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, PA.Google Scholar
  56. Och, Franz Josef, and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics 29: 19–51.CrossRefGoogle Scholar
  57. Och, Franz Josef, Michael E. Jahr, and Ignacio E. Thayer. 2014a. Minimum Error Rate Training with a Large Number of Features for Machine Learning. U.S. Patent 8,645,119.Google Scholar
  58. Och, F.J., J. Dean, T. Brants, A.M. Franz, J. Ponte, P. Xu, S.M. Teh, J. Chin, I.E. Thayer, A. Carver, and D. Rosart. 2014b. Encoding and Adaptive, Scalable Accessing of Distributed Models. U.S. Patent 8,738,357.Google Scholar
  59. Pace-Sigge, Michael. 2013. Lexical Priming in Spoken English Usage. Houndmills: Palgrave Macmillan.CrossRefGoogle Scholar
  60. Pace-Sigge, Michael, and Katie J. Patterson. 2017. Lexical Priming: Applications and Advances. Amsterdam: John Benjamins.CrossRefGoogle Scholar
  61. Patterson, Katie J. 2016. The Analysis of Metaphor: To What Extent Can the Theory of Lexical Priming Help Our Understanding of Metaphor Usage and Comprehension? Journal of Psycholinguistic Research 45 (2): 237–258.CrossRefGoogle Scholar
  62. Patterson, Katie J. 2018. Understanding Metaphor through Corpora: A Case Study of Metaphors in Nineteenth Century Writing. New York: Routledge.Google Scholar
  63. Quillian, M. Ross. 1966. Semantic Memory. Unpublished Doctoral Dissertation, Carnegie Institute of Technology (Reprinted in Part in M. Minsky (ed.), Semantic Information Processing. Cambridge: MIT Press, 1968).Google Scholar
  64. Quillian, M. Ross. 1969. The Teachable Language Comprehender: A Simulation Program and Theory of Language. Computational Linguistics 12 (8) (August): 459–476.CrossRefGoogle Scholar
  65. Sardinha, Tony Berber. 2017. Lexical Priming and Register Variation. In Lexical Priming: Applications and Advances, ed. M. Pace-Sigge and K. J. Patterson, 189–230. Amsterdam: John Benjamins.Google Scholar
  66. Shastri, Lokendra. 1992. Structured Connectionist Networks of Semantic Networks. Computers & Mathematics with Applications 23 (2–5): 293–328.CrossRefGoogle Scholar
  67. Simmons, Robert. 1963. Synthetic Language Behaviour. Data Processing Manager 5 (12): 11–18.Google Scholar
  68. Sinclair, John M. 1987. The Nature of the Evidence. In Looking Up, ed. J. Sinclair, 150–159. London: Collins.Google Scholar
  69. Sinclair, John M. 1991. Corpus, Concordance, Collocation. Oxford: Oxford University Press.Google Scholar
  70. Singhal, Amit, Mehran Sahami, John Lamping, Marcin Kaszkiel, and Monika H. Henzinger. Google Inc. 2011. Search Queries Improved Based on Query Semantic Information. U.S. Patent 8,055,669.Google Scholar
  71. Sowa, John F. 1987. Semantic Networks. In Encyclopedia of Artificial Intelligence, ed. Stuart C. Shapiro. London: Wiley.Google Scholar
  72. Steyvers, Mark, and Joshua B. Tenenbaum. 2005. The Large-Scale Structure of Semantic Networks: Statistical Analyses and a Model of Semantic Growth. Cognitive Science 29 (1): 41–78.CrossRefGoogle Scholar
  73. Stubbs, Michael. 1995. Collocations and Cultural Connotations of Common Words. Linguistics and Education 7 (4): 379–390.CrossRefGoogle Scholar
  74. Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, 3104–3112.Google Scholar
  75. Szymanski, Julian, and Duch Włodzisław. 2012. Annotating Words Using WordNet Semantic Glosses. In International Conference on Neural Information Processing (ICONIP) 2012, ed. Julian Szymański and Włodzisław Duch, 180–187. Part IV, LNCS 7666.CrossRefGoogle Scholar
  76. Teufl, Peter, and Stefan Kraxberger. 2011. Extracting Semantic Knowledge from Twitter. In Electronic Participation, 48–59.CrossRefGoogle Scholar
  77. Titchener, Edward B. 1922. A Note on Wundts Doctrine of Creative Synthesis. The American Journal of Psychology 33 (3): 351–360.CrossRefGoogle Scholar
  78. Touretzky, David. 1986. The Mathematics of Inheritance Systems. London: Pitman Publishing.Google Scholar
  79. Vasserman, Lucy, Vlad Schogol, and Keith Hall. 2015. Sequence-Based Class Tagging for Robust Transcription in ASR. In Sixteenth Annual Conference of the International Speech Communication Association.Google Scholar
  80. Whitsitt, Sam. 2005. A Critique of the Concept of Semantic Prosody. International Journal of Corpus Linguistics 10 (3): 283–305.CrossRefGoogle Scholar
  81. Wilensky, Robert. 1978. Understanding Goal Based Stories. Yale University Computer Science Research Report, New Haven, CT.Google Scholar
  82. Wilensky, Robert. 1982. Story Points, Strategies for Natural Language Processing. New York: Erlbaum.Google Scholar
  83. Wilensky, Robert. 1983. Memory and Inference. In International Joint Conference on Artificial Intelligence (IJCAI), 402–404.Google Scholar
  84. Wu, Dekai 1989. A Probabilistic Approach to Marker Propagation. In International Joint Conference on Artificial Intelligence (IJCAI), 574–582.Google Scholar
  85. Wundt, Wilhelm Max. 1862. Beiträge zur Theorie der Sinneswahrnehmung. Leipzig und Heidelberg: Wintersche Verlagsbuchhandlung.Google Scholar
  86. Xiao, Richard. n.d. Corpus Linguistics: The Basics. Making Statistical Claims (PPT). Last Accessed 10/2017.
  87. Xioa, Richard, and Tony McEnery. 2006. Collocation, Semantic Prosody, and Near Synonymy: A Cross-Linguistic Perspective. Applied Linguistics 27 (1): 103–129.CrossRefGoogle Scholar
  88. Yu, Yeong-Ho, and Robert F. Simmons. 1988. Constrained Marker Passing. Artificial Intelligence Laboratory, University of Texas at Austin.Google Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Department of English Language and CultureUniversity of Eastern FinlandJoensuuFinland

Personalised recommendations