Advertisement

Structural Complexity and Neural Networks

  • Alberto Bertoni
  • Beatrice Palano
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2486)

Abstract

We survey some relationships between computational complexity and neural network theory. Here, only networks of binary threshold neurons are considered.

We begin by presenting some contributions of neural networks in structural complexity theory. In parallel complexity, the class TC0 k of problems solvable by feed-forward networks with k levels and a polynomial number of neurons is considered. Separation results are recalled and the relation between TC0 =∪TC0 k and NC1 is analyzed. In particular, under the conjecture TC ≠ NC1, we characterize the class of regular languages accepted by feed-forward networks with a constant number of levels and a polynomial number of neurons.

We also discuss the use of complexity theory to study computational aspects of learning and combinatorial optimization in the context of neural networks. We consider the PAC model of learning, emphasizing some negative results based on complexity theoretic assumptions. Finally, we discussed some results in the realm of neural networks related to a probabilistic characterization of NP.

Keywords

Structural complexity Neural networks Finite state automata Learning Combinatorial optimization 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    W. B. Alexi, B. Chor, O. Goldreich AND C. P. Schnorr, RSA and Rabin function: Certain parts are as hard as the whole. SIAM J. Comput. 17 (1988) 194–209.zbMATHCrossRefMathSciNetGoogle Scholar
  2. [2]
    E. Amaldi AND V. Kann The complexity and approximability of finding maximum feasible subsystems of linear relations. Theoretical Computer Science 147 (1995) 181–210.zbMATHCrossRefMathSciNetGoogle Scholar
  3. [3]
    D. Angluin, Queries and Concept Learning. Machine Learning 2 (1988) 319–342.Google Scholar
  4. [4]
    M. M. Arbib, Brains, Machines and Mathematics (2nd ed.). Springer Verlag, 1987.Google Scholar
  5. [5]
    S. Arora, C. Lund, R. Motwani, M. Sudan AND M. Szegedy, Proof Verification and Hardness of Approximation. In: Proc. 33rd Am. IEEE Symp. on Found. Comp. Sci. (1992) 14–23.Google Scholar
  6. [6]
    D. Barrington, Bounded-width polynomial-size branching programs recognize exactly those languages in NC1. J. Comput. Syst. Sci. 38 (1989) 150–164.zbMATHCrossRefMathSciNetGoogle Scholar
  7. [7]
    D. A. Mix Barrington, K. Compton, H. Straubing AND D. Thérien, Regular languages in NC1. J. Comp. Syst. Sci. 44 (1992) 478–499.zbMATHCrossRefGoogle Scholar
  8. [8]
    F. Barahona, On the computational complexity of Ising spin glass models. J.of Physics A: Mathematical, nuclear and general 15 (1982) 3241–3253.CrossRefMathSciNetGoogle Scholar
  9. [9]
    P. Bartlett AND S. Ben-David, Hardness Results for Neural Network Approximation Problems. Proc. 4th European Conference on Comput. Learning Theory (1999) 50–62.Google Scholar
  10. [10]
    E. B. Baum, Neural Nets Algorithms that learn in polynomial time from examples and queries. IEEE Trans. on Neural Network 2 (1991) 5–19.CrossRefGoogle Scholar
  11. [11]
    E. B. Baum And D. Haussler, What size net gives valid generalization? Neura Computation 1 (1989) 151–160.CrossRefGoogle Scholar
  12. [12]
    S. Ben-David, N. Eiron AND P. M. Long, On the Difficulty of Approximately Maximizing Agreements. Proc. 13th Ann. Conference on Comput. Learning Theory (2000) 266–274.Google Scholar
  13. [13]
    A. Bertoni, P. Campadelli AND G. Molteni, On the approximability of energy function in Ising spin glasses. J. of Physics A: Mathematical, nuclear and general 27 (1994) 6719–6729.zbMATHCrossRefMathSciNetGoogle Scholar
  14. [14]
    A. Bertoni, P. Campadelli, A. Morpurgo AND S. Panizza, Polynomial uniform convergence and polynomial-sample learnability. In: Proc. 5th Ann.ACM Workshop on Computational Learning Theory (1992) 265–271.Google Scholar
  15. [15]
    I. Bieche, R. Maynard, R. Rammal AND J. P. Uhry, J. of Physics A: Mathematical, nuclear and general 13 (1980) 2553–2576.CrossRefMathSciNetGoogle Scholar
  16. [16]
    G. Benedek AND A. Itai, Learnability with respect to fixed distributions. Theoretical Computer Science 86 (1991) 377–389.zbMATHCrossRefMathSciNetGoogle Scholar
  17. [17]
    A. Blum AND R. L. Rivest, Training a 3-node neural network is NP-complete. Neural Networks 5 (1992) 117–127.CrossRefGoogle Scholar
  18. [18]
    A. Blumer, A. Ehrenfeucht, D. Haussler AND M.K. Warmuth, Learnability and the Vapnik-Chervonenkis Dimension. J. ACM 36 (1989) 929–965.zbMATHCrossRefMathSciNetGoogle Scholar
  19. [19]
    E. R. Caianiello, Outline of a theory of thought processes and thinking machines. J. Theoretical Biology 1 (1961) 1–27.CrossRefMathSciNetGoogle Scholar
  20. [20]
    S.A. Cook, The Complexity of Theorem Proving Procedures. In: Proc. 3rd Ann. ACM Symposium on Theory of Computing (1971) 151–158.Google Scholar
  21. [21]
    S. A. Cook, A taxonomy of problems with fast parallel algorithms. Information and Control 64 (1985) 2–22.zbMATHCrossRefMathSciNetGoogle Scholar
  22. [22]
    A. Ehrenfeucht, D. Haussler, M. Kearns AND L. Valiant, A general lower bound on the number of examples needed for learning. Information and Computation 82 (1989) 247–261.zbMATHCrossRefMathSciNetGoogle Scholar
  23. [23]
    M. R. Garey AND D.S. Johnson, Computers and intractability. A guide to the theory of NP-completeness. W.H. Freeman, 1979.Google Scholar
  24. [24]
    K. Goedel, Uber Formal Unentscheidbare Satze der Principia Matematica und verwandter Systeme I. Monatshefte fur Matematik und Physik 38 (1931) 173–198.zbMATHCrossRefGoogle Scholar
  25. [25]
    M. Goldmann AND M. Karpinski, Simulating Threshold Circuits by Majority Circuits. SIAM J. Comput. 98 (1998) 230–246.CrossRefMathSciNetGoogle Scholar
  26. [26]
    A. Hajnal, W. Maass, P. Pudlák, M. Szegedy AND G. Turań, Threshold circuits of bounded depth. J. Comput. Sys. Sci. 46 (1993) 129–154.zbMATHCrossRefGoogle Scholar
  27. [27]
    J. Hartmanis AND R. E. Stearns, On the computational complexity of algorithms. Trans. Am. Math. Soc. 117 (1965) 285–306.zbMATHCrossRefMathSciNetGoogle Scholar
  28. [28]
    J. Hartmanis, Goedel, Von Neumann and P =?NP problem. Bull. of EATCS, 38, 101–107, 1989.zbMATHGoogle Scholar
  29. [29]
    D. Haussler, Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications. Information and Computation 100 (1992) 78–150.zbMATHCrossRefMathSciNetGoogle Scholar
  30. [30]
    J. Hastad, Some optimal inapproximability results. Royal Institute of Technology (1999).Google Scholar
  31. [31]
    D.O. Hebb, The Organization of Behavior. Wiley, 1949.Google Scholar
  32. [32]
    T. Hegedus, Can complexity theory benefit from learning theory? In: European Conf. on Machine Learning (1993) 354–359.Google Scholar
  33. [33]
    W. Hesse, Division is in Uniform TC0. In: ICALP: Annual International Colloquium on Automata, Languages and Programming (2001) 104–114.Google Scholar
  34. [34]
    T. Hofmeister, Depth-efficient threshold circuits for arithmetic functions. In: V. Roychowdhury, K.-Y. Siu AND A. Orlitsky (eds.), Theoretical advances in Neural Computation and Learning. Kluwer Academic, Boston, London, Dordrecht (1994) 37–84.Google Scholar
  35. [35]
    J. Hopfield, Neural networks and physical systems with emergent collective computational abilities. In: Proc. of the National Academy of Science, USA (1982) 2554–2558.Google Scholar
  36. [36]
    D. Johnson, A catalog of complexity classes. In: J. Van Leeuwen (eds.), Handbook of Theoretical Computer Science. North-Holland (1990) 142–143.Google Scholar
  37. [37]
    J. S. Judd, Neural Network Design and the Complexity of Learning. The MIT Press, 1990.Google Scholar
  38. [38]
    N. Karmarkar, A new polynomial-time algorithm for linear programming. Combinatorica 4 (1984) 373–395.zbMATHCrossRefMathSciNetGoogle Scholar
  39. [39]
    R. M. Karp, Reducibility Among Combinatorial Problems. Complexity of Computer Computations (1972) 85–103.Google Scholar
  40. [40]
    M. Kearns AND L. G. Valiant, Cryptographic limitations on learning Boolean formulae and finite automata. In: Proc. of ACM Symp. on Theory of Computing (1989) 15–17.Google Scholar
  41. [41]
    L. A. Levin, Universal Search Problems. Problemy Peredachi Informatsii 9 (1973) 265–266.Google Scholar
  42. [42]
    J. H. Lin AND J. S. Vitter, Complexity result on learning with neural nets. Machine Learning 6 (1991) 211–230.Google Scholar
  43. [43]
    W. Maass, Neural nets with superlinear VC dimension. Neural Computation 6 (1994) 877–884.zbMATHCrossRefGoogle Scholar
  44. [44]
    C. Mereghetti AND B. Palano, The Parallel Complexity of Deterministic and Probabilistic Automata. J. Aut., Lang. and Comb. To be published.0Google Scholar
  45. [45]
    M.L. Minsky, Steps toward artificial intelligence. In: Proc. IRE 49 (1961) 8–30.CrossRefMathSciNetGoogle Scholar
  46. [46]
    W. S. McCulloch AND W. Pitts, A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5 (1943) 115–133.zbMATHCrossRefMathSciNetGoogle Scholar
  47. [47]
    S. Muroga, Threshold Logic and its Applications. Wiley, 1971.Google Scholar
  48. [48]
    L. Pitt AND L. G. Valiant, Computational limitations on learning from examples. J. ACM 35 (1988) 965–984.zbMATHCrossRefMathSciNetGoogle Scholar
  49. [49]
    F. Rosenblatt, Principles of Neurodynamics. Spartan Books, 1962.Google Scholar
  50. [50]
    V. Roychowdhury, K.-Y. Siu AND A. Orlitsky, Theoretical advances in Neural Computation and Learning. Kluwer Academic, Boston, London, Dordrecht, 1994.zbMATHGoogle Scholar
  51. [51]
    V. Roychowdhury, K.-Y. Siu AND A. Orlitsky, Neural models and spectral methods. In: V. Roychowdhury, K.-Y. Siu AND A. Orlitsky (eds.), Theoretical advances in Neural Computation and Learning. Kluwer Academic, Boston, London, Dordrecht (1994) 3–36.Google Scholar
  52. [52]
    D.E. Rumelhart, G. E. Hinton AND R. J. Williams, Learning representations by back-propagating errors. Nature 323 (1986) 533–536.CrossRefGoogle Scholar
  53. [53]
    W. R. Scott, Group Theory. Prentice-Hall, 1964. Reprinted by Dover, 1987.Google Scholar
  54. [54]
    H. Straubing, Finite Automata, Formal Logic, and Circuit Complexity. Birkhäuser, 1994.Google Scholar
  55. [55]
    A.M. Turing, On computable numbers with an application to the Entscheidungs problem. Proc. London Math. Soc. 2-42, (1936) 230–265.Google Scholar
  56. [56]
    L. Valiant, A theory of the learnable. Communications of the ACM 27 (1984) 1134–1142.zbMATHCrossRefGoogle Scholar
  57. [57]
    V. N. Vapnik AND A. Y. Chervonenkis, On the uniform convergence of relative frequencies of events to their probabilities. Theory of Prob. and its Applications 16 (1971) 264–280.CrossRefzbMATHGoogle Scholar
  58. [58]
    R. S. Wenocur AND R. M. Dudley, Some Special Vapnik-Chervonenkis classes. Discrete Mathematics 33 (1981) 313–318.zbMATHCrossRefMathSciNetGoogle Scholar
  59. [59]
    I. Wegener, The Complexity of Boolean Functions. Teubner, Stuttgart, 1987.Google Scholar
  60. [60]
    I. Wegener, Optimal lower bounds on the depth of polynomial-size threshold circuits for some arithmetic functions. Information Processing Letters 46 (1993) 85–87.zbMATHCrossRefMathSciNetGoogle Scholar
  61. [61]
    P. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Science. PhD thesis, Harvard University, 1974.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Alberto Bertoni
    • 1
  • Beatrice Palano
    • 2
  1. 1.Dipartimento di Scienze dell’InformazioneUniversità degli Studi di MilanoMilanoItaly
  2. 2.Dipartimento di InformaticaUniversità degli Studi di TorinoTorinoItaly

Personalised recommendations