Connectionist Natural Language Processing: A Status Report

  • Michael G. Dyer
Part of the The Springer International Series In Engineering and Computer Science book series (SECS, volume 292)


Connectionist networks (CNs) exhibit many useful properties. Their spreading activation processes are inherently parallel in nature and support associative retrieval of memories. The summation and thresholding of activation allows for smooth integration of multiple sources of knowledge. CNs with distributed representations (Rumelhart and McClelland, 1986) exhibit robustness in the face of noise/damage and can learn to perform complex mapping tasks just from examples. Connectionist networks are also able to dynamically reinterpret situations as new inputs are received. These features are very useful for natural language processing (NLP) and offer the hope that connectionist approaches to NLP will replace the more traditional, symbolic approaches to NLP.


Hide Layer Episodic Memory Natural Language Processing Symbolic System Recursive Structure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Ajjanagadde, V. and Shastri, L. (1989). Efficient Inference with Multi-Place Predicates and Variables in a Connectionist System. Proceedings of the 11th Annual Conference of the Cognitive Science Society. LEA Press, Hillsdale NJ. pp.396–403.Google Scholar
  2. [2]
    Alvarado, SJ. (1990). Understanding Editorial Text. Kluwer, Norwell, MA.Google Scholar
  3. [3]
    Alvarado, S., Dyer, M.G. and M. Flowers. (1990a). Argument Representation for Editorial Text. Knowledge-Based Systems, 3(2):87–107.CrossRefGoogle Scholar
  4. [4]
    Alvarado, S., Dyer, M.G. and M. Rowers. (1990b). Argument Comprehension and Retrieval for Editorial Text. Knowledge-Based Systems, 3(3).Google Scholar
  5. [5]
    Alvarado, S.J., Dyer, M. G. and M. Flowers. (1990c). Natural Language Processing: Computer Comprehension of Editorial Text. In H. Adeli (Ed.), Knowledge Engineering, Vol. 1, Fundamentals, pp. 286–344, McGraw-Hill, NY.Google Scholar
  6. [6]
    Barnden, J.A. (1991). Encoding complex symbolic data structures with some unusual connectionisai] techniques. In J.A. Barnden and J.B. Pollack (Eds.), High-Level Connectionist Models. Ablex, Norwood, NJ. pp. 180–240.Google Scholar
  7. [7]
    Barnden, J.A. (1992a). Connectionism, Generalization and Propositional Attitudes: A Catalogue of Challenging Issues. In J. Dinsmore (Ed.), The Symbolic and Connectionist Paradigms: Closing the Gap. LEA Press, Hillsdale NJ. pp. 149–178.Google Scholar
  8. [8]
    Barnden, J.A. (1992b). Connectionism, Structure-Sensitivity, and Systematicity: Refining the Task Requirements. Memoranda in Computer and Cognitive Science, No. MCCS-92-227, Computing Research Lab., New Mexico State University, Las Cruces NM.Google Scholar
  9. [9]
    Barnden, J.A. (1994). Complex Symbol-Processing in a Transiently Localist Connectionist Architecture. This volume.Google Scholar
  10. [10]
    Barnden, J. and Srinivas, K. (1991). Encoding Techniques for Complex Information Structures in Connectionist Systems. Connection Science, 3(3):269–315.CrossRefGoogle Scholar
  11. [11]
    Churchland, P.S. and T.J. Sejnowski. (1989). Neural Representation and Neural Computation. In L. Nadel, L.A. Cooper, P. Culicover and R.M. Harnish (Eds.), Neural Connections, Mental Computation. Bradford Book, MIT Press, Cambridge MA.Google Scholar
  12. [12]
    Churchland, P.S. (1986). Neurophilosophy: Toward a Unified Science for Mind-Brain. MIT Press, Cambridge MA.Google Scholar
  13. [13]
    Dolan, C.P. (1989). Tensor manipulation networks: Connectionist and symbolic approaches to comprehension, learning, and planning. Ph.d. Dissertation, Computer Science Dept. UCLA, Los Angeles, CA. (To be published by LEA Press).Google Scholar
  14. [14]
    Dolan, C.P. and Smolensky, P. (1989). Tensor product production system: A modular architecture and representation. Connection Science, 1:53–68.CrossRefGoogle Scholar
  15. [15]
    Dolan, C.P. and M.G. Dyer. (1989). Parallel Retrieval and Application of Conceptual Knowledge. In D. Touretzky, G. Hinton, T. Sejnowski (Eds.), Proceedings of the 1988 Connectionist Models Summer School. Morgan Kaufmann, pp. 273–280.Google Scholar
  16. [16]
    Dyer, M.G. (1983). In-Depth Understanding. MIT Press, Cambridge, MA.Google Scholar
  17. [17]
    Dyer, M.G. (1990). Distributed Symbol Formation and Processing in Connectionist Networks. Journal of Experimental and Theoretical Artificial Intelligence, 2:215–239.CrossRefGoogle Scholar
  18. [18]
    Dyer, M.G., Cullingford, R. & Alvarado, S. (1987). Scripts. In S. Shapiro (Ed.), Encyclopedia of Artificial Intelligence. John Wiley & Sons, pp. 980–994.Google Scholar
  19. [19]
    Dyer, M.G., Flowers, M. and Wang, Y.A. (1992). Distributed Symbol Discovery through Symbol Recirculation: Toward Natural Language Processing in Distributed Connectionist Networks. In R.G. Reilly and N.E. Sharkey (Eds.), Connectionist Approaches to Natural Language Processing, LEA Press, Hillsdale NJ, pp. 21–48.Google Scholar
  20. [20]
    Dyer, M.G. and Nenov, V.I. (1993). Language Learning via Perceptual/Motor Experiences. Proceedings of the Fifteenth Annual Conference of the Cognitive Science Society, LEA Press, Hillsdale NJ.Google Scholar
  21. [21]
    Elman, J.L. (1990). Finding structure in time. Cognitive Science, 14:179–211.CrossRefGoogle Scholar
  22. [22]
    Feldman, J.A. (1989). Neural Representation of Conceptual Knowledge. In L. Nadel, L. A. Cooper, P. Culiver and R.M. Harnish (Eds.), Neural Connections, Mental Computation. Bradford book/MIT PRess, Cambridge MA.Google Scholar
  23. [23]
    Feldman, J.A., Lakoff, G., Stolcke, A. and Hollbach Weber, S. (1990). Miniature Language Acquisition: A touchstone for cognitive science. Tech. Rep. TR-90-009, ICSI, Berkeley, CA.Google Scholar
  24. [24]
    Harnad, S. (1990). The Symbol Grounding Problem. Physica D, 42:335–346.CrossRefGoogle Scholar
  25. [25]
    Hinton, G.E., McClelland, J.L. and Rumelhart, D.E. (1986). Distributed Representation. In D.E. Rumelhart and J.L. Mcclelland (Eds.), Parallel Distributed processing, Vol. 1, MIT Press, Cambridge MA.Google Scholar
  26. [26]
    Holldobler, S. (1990). A structured connectionist unification algorithm. In Proceedings of the Ninth National Conference on Artificial Intelligence, Boston, MA.Google Scholar
  27. [27]
    Johnson-Laird, P.N. (1983). Mental Models: Towards a Cognitive Science of Language, Inference and Consciousness. Harvard University Press, Cambridge MA.Google Scholar
  28. [28]
    Kandel, E.R. (1985). Processing of Form and Movement in the Visual System. In E.R. Kandel and J.H. Schwartz (Eds.), Principles of Neuro-science (Second Edition), (pp. 366–383), Elsevier, NY.Google Scholar
  29. [29]
    Kohonen, T. (1988). Self-Organization and Associative Memory. Springer-Verlag (2nd ed.).Google Scholar
  30. [30]
    Kolodner, J.L. (1984). Retrieval and Organizational Strategies in Conceptual Memory: A Computer Model. LEA Press, Hillsdale NJ.Google Scholar
  31. [31]
    Lange, T.E. (1994). A structured connectionist approach to inferencing and retrieval. This volume.Google Scholar
  32. [32]
    Lange, T.E. and M.G. Dyer. (1989a). Frame Selection in a Connectionist Model of High-Level Inferencing. Proceedings of the Eleventh Annual Conference of the Cognitive Science Society (CogSci-89). LEA Press, Hillsdale NJ.Google Scholar
  33. [33]
    Lange, T. and M.G. Dyer. (1989b). Dynamic, Non-Local Role Bindings and Inferencing in a Localist Network for Natural Language Understanding. In D.S. Touretzky (Ed.), Advances in Neural Information Processing Systems 1. San Mateo, CA: Morgan Kauffman, pp. 545–552,.Google Scholar
  34. [34]
    Lange, T.E. and Dyer, M.G. (1989c). High-level inferencing in a connectionist network. Connection Science, 1:181–217.CrossRefGoogle Scholar
  35. [35]
    Lange, T.E., Vidal, JJ. and Dyer, M.G. (1991). Artificial Neural Oscillators for Inferencing. In A.V. Holden and V.I. Kryukov (Eds.), Neurocomputers and Attention, Vol. I, Manchester University Press.Google Scholar
  36. [36]
    Lee, G.(1991). Distributed Semantic Representations for Goal/Plan Analysis of Narratives in a Connectionist Architecture. Ph.D. CS Dept. UCLA.Google Scholar
  37. [37]
    Lee, G. and Dyer, M.G. (in press). Goal/Plan Analysis via Distributed Semantic Representations in a Connectionist System. Applied Intelligence.Google Scholar
  38. [38]
    Lee, G., Flowers M. and M.G. Dyer. (1990). Learning Distributed Representations for Conceptual Knowledge and their Application to Script-Based Story Processing. Connection Science, 2(4):313–345. [Also reprinted in N. Sharkey (Ed.), Connectionist Natural Language Processing: Readings from Connection Science. (Chapter 11, pp. 215–247), Kluwer Academic Publishers, Norwell, MA. 1992.]CrossRefGoogle Scholar
  39. [39]
    Miikkulainen, R. and Dyer, M.G. (1991). Natural language processing with modular PDP networks and distributed lexicon. Cognitive Science, 15(3):343–399.CrossRefGoogle Scholar
  40. [40]
    Miikkulainen, R. (1993). Subsymbolic Natural Language Processing: An Integrated Model of Scripts, Lexicon and Memory, Bradford/MIT Press, Cambridge MA.Google Scholar
  41. [41]
    Miikkulainen, R. (1994). Subsymbolic Parsing of Embedded Structures. This volume.Google Scholar
  42. [42]
    Minsky, M. (1985). The Society of Mind. Simon and Schuster, NY.Google Scholar
  43. [43]
    Nenov, V.I. (1991). Perceptually Grounded Language Acquisition: A Neural/Procedural Hybrid Model. Ph.D. Dissertation and Technical Report UCLA-AI-91-07, Computer Science Departments UCLA.Google Scholar
  44. [44]
    Nenov, V.I. and Dyer, M.G. (in press-a) Perceptually Grounded Language Learning: Part 1 — A Neural Network Architecture for Robust Sequence Association. Connection Science, 5(2).Google Scholar
  45. [45]
    Nenov, V.I. and Dyer, M.G. (in press-b) Perceptually Grounded Language Learning: Part 2 — DETE: A Neural/Procedural Model. Connection Science, 5(3).Google Scholar
  46. [46]
    Nenov, V.I. and Dyer, M.G. (in press-c) Language Learning via Perceptual/Motor Association: A Massively Parallel Model. In Hiroaki Kitano (Ed.), Massively Parallel Artificial Intelligence. AAAI/MIT Press.Google Scholar
  47. [47]
    Pazzani, M.J. (1990). Creating a Memory of Causal Relationships: An Integration of Empirical and Explanation-Based Learning Method. Lawrence Erlbaum Associates (LEA Press) Hillsdale, NJ, 1990.Google Scholar
  48. [48]
    Pazzani, M.J. and M.G. Dyer. (1989). Memory Organization and Explanation-Based Learning. International Journal of Expert Systems: Research & Applications, 2(3):331–358.Google Scholar
  49. [49]
    Pazzani, M. and M.G. Dyer. (1987). A Comparison of Concept Identification in Human Learning and Network Learning with the Generalized Delta Rule. Proceedings of 10th Inaternational Joint Conference on Artificial Intelligence (IJCAI-87). Morgan Kaufmann, Los Altos CA, pp. 147–150.Google Scholar
  50. [50]
    Pinker, S. (1989). Learnability and Cognition: The Acquisition of Argument Structure. MIT Press, CAmbridge MA.Google Scholar
  51. [51]
    Pinker, S. and Mehler, J. Eds., (1988). Connections and Symbols. Bradford/MIT Press. Cambridge, MA.Google Scholar
  52. [52]
    Pollack, J.B. (1988). Recursive Auto-Associative Memory: Devising Compositional Distributed Representations. Proceedings of the Tenth Annual Conference of the Cognitive Science Society Lawrence Erlbaum. Hillsdale, NJ.Google Scholar
  53. [53]
    Pollack, J.B. (1989). Implications of Recursive Distributed Representations. In D. S. Touretzky (Ed.), Advances in Neural Information Processing1, Morgan Kaufmann Publ. San Mateo, CA, pp. 527–536.Google Scholar
  54. [54]
    Pollack, J.B. (1990). Recursive Distributed Representations. Artificial Intelligence, 46:77–105.CrossRefGoogle Scholar
  55. [55]
    Regier, T. (1992). The Acquisition of Lexical Semantics for Spatial Terms: A Connectionist Model of Perceptual Categorization. Ph.D. Dissertation. University of California at Berkeley.Google Scholar
  56. [56]
    Rumelhart, D.E. and McClelland, J.L., Eds. (1986). Parallel Distributed Processing. Vol. 1. Bradford Books/MIT Press.Google Scholar
  57. [57]
    St. John, M.F. and McClelland, J.L. (1990). Learning and Applying Contextual Constraints in Sentence Comprehension. Artificial Intellgence, 46:217–257.CrossRefGoogle Scholar
  58. [58]
    Schank, R.C. and Abelson, R.P. (1977). Scripts, Plans, Goals, and Understanding. Lawrence Erlbaum, Hillsdale, NJ.MATHGoogle Scholar
  59. [59]
    Shastri, L. and Ajjanagadde, V. (1990). An Optimally Efficient Limited Inference System. Proceedings of Eighth National Conference on Artificial Intelligence. AAAI Press / MIT Press, Menlo Park, CA. pp. 563–570.Google Scholar
  60. [60]
    Shastri, L. and Ajjanagadde, V. (1993) From simple associations to systematic reasoning. A connectionist representation of rules, variables and dynamic bindings using temporal synchrony. Behavoral and Brain Sciences, 16(3):417–494.Google Scholar
  61. [61]
    Smolensky, P. (1990). Tensor Product Variable Binding and the Representation of Symbolic Structures in Connectionist Systems. Artificial Intelligence, 46:159–216.CrossRefMATHMathSciNetGoogle Scholar
  62. [62]
    Stolcke, A. (1989). Unification as Constraint Satisfaction in Structured Connectionist Networks. Neural Computation,1(4):559–567.CrossRefGoogle Scholar
  63. [63]
    Strong, G.W. and Whitehead, B.A. (1989). A solution to the tag-assignment problem for neural networks. Behavioral and Brain Sciences, 12:381–433.CrossRefGoogle Scholar
  64. [64]
    Sumida, R.A. and M.G. Dyer. (1989). Storing and Generalizing Multiple Instances while Maintaining Knowledge-Level Parallelism. Proceedings of Eleventh International Joint Conference on Artificial Intelligence (IJCAI-89). pp. 1426–1431, (Distributed by Morgan Kaufmann Publ.).Google Scholar
  65. [65]
    Sumida, R.A. (1991). Dynamic Inferencing in Parallel Distributed Semantic Networks. Proceedings of Thirteenth Annual Conference of the Cognitive Science Society. LEA Press, Hillsdale NJ. pp.913–917.Google Scholar
  66. [66]
    Sumida, R.A. and Dyer, M.G. (1992). Propagation Filters in PDS Networks for Sequencing and Ambiguity Resolution. In J.E. Moody, S.J. Hanson and R.P. Lippmann (Eds.), Advances in Neural Information Processing Systems 4, Morgan Kaufmann Publ., San Mateo, CA, pp. 233–240.Google Scholar
  67. [67]
    Sun, R. (1989). A discrete neural network model for conceptual representation and reasoning. Proceedings of the Eleventh Annual Conference of the Cognitive Science Society. LEA Press, Hillsdale NJ.Google Scholar
  68. [68]
    Sun, R. (1992). On Variable Binding in Connectionist Networks. Connection Science, 4(2):93–124.CrossRefGoogle Scholar
  69. [69]
    Sun, R. (1993). Integrating Rules and Connectionism for Robust Reasoning. John Wiley and Sons, NY.Google Scholar
  70. [70]
    Touretzky, D.S. and G.E. Hinton. (1988). A Distributed Connectionist Production System. Cognitive Science, 12(3):423–466.CrossRefGoogle Scholar
  71. [71]
    Tulving, E. (1972). Episodic and Semantic Memory. In E. Tulving and W. D. Donaldson (Eds.), Organization of Memory. Academic Press, NY.Google Scholar
  72. [72]
    Turner, S.R. (1992). MINSTREL: A Computer Model of Creativity and Storytelling. Ph.D. Dissertation, Computer Science Dept. UCLA (to be published by LEA Press).Google Scholar
  73. [73]
    von der Malsburg, C. (1981). The correlation theory of brain function. Internal Report 81-2. Dept. of Neurobiology, Max-Plank-institute for Biophysical Chemistry.Google Scholar
  74. [74]
    von der Malsburg, C. and Singer, W. (1988). Principles of cortical network organization. In P. Rakic and W. Singer (Eds.), Neurobiology of Neocortex. (pp. 69–99). John Wiley & Sons Ltd., London.Google Scholar
  75. [75]
    Waibel, A. (1989). Consonant recognition by modular construction of large phonemic time-delay neural networks. In D.S. Touretzky (Ed.), Advances in Neural Information Processing Systems I. (pp. 215–223). Morgan Kaufman, San Mateo, CA.Google Scholar
  76. [76]
    Werner, G. M. and M. G. Dyer. (1991). Evolution of Communication in Artificial Organisms. In J.D. Farmer, C. Langton, S. Rasmussen and C. Taylor (Eds.), Artificial Life II, Addison-Wesley.Google Scholar
  77. [77]
    Wilensky, R. (1983). Planning and Understanding: A Computational Approach to Human Reasoning. Addison-Wesley, Reading, MA.Google Scholar

Copyright information

© Kluwer Academic Publishers 1995

Authors and Affiliations

  • Michael G. Dyer
    • 1
  1. 1.Computer Science DepartmentUniversity of California, at Los AngelesLos Angeles

Personalised recommendations