Basic Concepts of Probabilistic Neural Networks

  • Leszek RutkowskiEmail author
  • Maciej Jaworski
  • Piotr Duda
Part of the Studies in Big Data book series (SBD, volume 56)


Probabilistic neural networks (PNN), introduced by Specht [1, 2] have their predecessors in the theory of statistical pattern classification. In the fifties and sixties, problems of statistical pattern classification in the stationary case were accomplished by means of parametric methods, using the available apparatus of statistical mathematics (e.g. [3, 4, 5, 6, 7]). The knowledge of the probability density to an accuracy of unknown parameters was assumed and the parameters were estimated based on the learning sequence. Having observed tendencies present in the literature within the next decades, we should say that these methods have been almost completely replaced by the non-parametric approach (see e.g. [8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]). In the non-parametric approach it is assumed that a functional form of probability densities is unknown.


  1. 1.
    Specht, D.: Probabilistic neural networks and the polynomial Adaline as complementary techniques for classification. IEEE Trans. Neural Netw. 1, 111–121 (1990)CrossRefGoogle Scholar
  2. 2.
    Specht, D.F.: A general regression neural network. IEEE Trans. Neural Netw. 2(6), 568–576 (1991)CrossRefGoogle Scholar
  3. 3.
    Bishop, C.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)zbMATHGoogle Scholar
  4. 4.
    Duda, R., Hart, P., Stork, D.: Pattern Classification. Wiley, London (2001)Google Scholar
  5. 5.
    Fu, K.: Sequential Methods in Pattern Recognition and Machine Learning. Academic, New York (1968)zbMATHGoogle Scholar
  6. 6.
    Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Academic, New York (1990)zbMATHGoogle Scholar
  7. 7.
    Webb, A.: Statistical Pattern Recognition. Wiley, Chichester (2002)Google Scholar
  8. 8.
    Devroye, L., Györfi, L.: Nonparametric Density Estimation: The \(L_1\) View. Wiley, New York (1985)Google Scholar
  9. 9.
    Devroye, L., Györfi, L., Lugosi, G.: Probabilistic Theory of Pattern Recognition. Springer, New York (1996)Google Scholar
  10. 10.
    Devroye, L., Lugosi, G.: Combinatorial Methods in Density Estimation. Springer, New York (2001)Google Scholar
  11. 11.
    Efromovich, S.: Nonparametric Curve Estimation. Methods, Theory and Applications. Springer, New York (1999)Google Scholar
  12. 12.
    Eubank, R.L.: Spline Smoothing and Nonparametric Regression. Marcel Dekker, INC., New York (1988)Google Scholar
  13. 13.
    Eubank, R.: Nonparametric Regression and Spline Smoothing. Marcel Dekker, New York (1999)zbMATHGoogle Scholar
  14. 14.
    Györfi, L., Hżrdle, W., Sarda, P., Vieu, P.: Nonparametric Curve Estimation from Time Series. Springer, New York (1989)CrossRefGoogle Scholar
  15. 15.
    Györfi, L., Kohler, M., Krzyżak, A., Walk, H.: A Distribution-Free Theory of Nonparametric Regression. Springer, New York (2002)CrossRefGoogle Scholar
  16. 16.
    Härdle, W.: Applied Nonparametric Regression. Cambridge University Press, Cambridge (1990)Google Scholar
  17. 17.
    Härdle, W., Kerkyacharian, G., Picard, D., Tsybakov, A.: Wavelets, Approximation, and Statistical Applications. Springer, New York (1998)CrossRefGoogle Scholar
  18. 18.
    Ibragimov, I., Khasminskii, R.: Statistical Estimation: Asymptotic Theory. Springer, New York (1981)CrossRefGoogle Scholar
  19. 19.
    Pagan, A., Ullah, A.: Nonparametric Econometrics. Cambridge University Press, London (1999)CrossRefGoogle Scholar
  20. 20.
    Rafajłowicz, E.: Consistency of orthogonal series density estimators based on grouped observations. IEEE Trans. Inf. Theory 43(1), 283–285 (1997)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Rao, B.L.S.P.: Nonparametric Functional Estimatio. Academic, New York (1983)Google Scholar
  22. 22.
    Thompson, J., Tapia, R.: Nonparametric Function Estimation and Simulation. SIAM, Philadelphia (1990)Google Scholar
  23. 23.
    Wertz, W.: Statistical Density Estimation: a Survey. Vandenhoeck & Ruprecht, Göttingen (1978)Google Scholar
  24. 24.
    Wertz, W., Schneider, B.: Statistical density estimation: a bibliography. Int. Stat. Rev. 47, 155–175 (1979)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Rosenblatt, M.: Remarks on some estimates of a density function. Ann. Math. Stat. 27, 155–175 (1956)MathSciNetGoogle Scholar
  26. 26.
    Parzen, E.: On estimation of probability density function and mode. Ann. Math. Stat. 33, 1065–1076 (1962)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Cacoullos, T.: Estimation of a multivariate density. Ann. Inst. Stat. Math. 18, 179–189 (1965)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Čencov, N.: Evaluation of an unknown distribution density from observations. Sov. Math. 3, 1559–1562 (1962)Google Scholar
  29. 29.
    Schwartz, S.: Estimation of probability density by an orthogonal series. Ann. Math. Stat. 1261–1265 (1967)Google Scholar
  30. 30.
    Kronmal, R., Tarter, M.: The estimation of probability densities and cumulatives by Fourier series methods. J. Am. Stat. Assoc. (1968)Google Scholar
  31. 31.
    Walter, G.: Properties of Hermite series estimation of probability density. Ann. Stat. 5, 1258–1264 (1977)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13, 21–27 (1967)CrossRefGoogle Scholar
  33. 33.
    Loftsgaarden, D., Quesenberry, C.: A nonparametric estimate of a multivariate density function. Ann. Math. Stat. 36, 1049–1051 (1965)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Devroye, L.: Universal consistency in nonparametric regression and nonparametric discrimination. Technical report. School of Computer Science, Mc Gill University (1978)Google Scholar
  35. 35.
    Greblicki, W.: Asymptotically optimal pattern recognition procedures with density estimate. IEEE Trans. Inf. Theory 24, 250–251 (1978)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Greblicki, W., Rutkowski, L.: Density-free Bayes risk consistency of nonparametric pattern recognition procedures. Proc. IEEE 69(4), 482–483 (1981)CrossRefGoogle Scholar
  37. 37.
    Rutkowski, L.: Sequential estimates of probability densities by orthogonal series and their application in pattern classification. IEEE Trans. Syst. Man Cybern. SMC-10(12), 918–920 (1980)Google Scholar
  38. 38.
    Rutkowski, L.: Sequential estimates of a regression function by orthogonal series with applications in discrimination. Lectures Notes in Statistics, vol. 8, pp. 236–244. Springer, New York (1981)Google Scholar
  39. 39.
    Rutkowski, L.: Sequential pattern recognition procedures derived from multiple Fourier series. Pattern Recognit. Lett. 8, 213–216 (1988)CrossRefGoogle Scholar
  40. 40.
    Ryzin, J.: Bayes risk consistency of classification procedures using density estimation. Sankhya Ser. A (1966)Google Scholar
  41. 41.
    Wolverton, C., Wagner, T.: Asymptotically optimal discriminant functions for pattern classification. IEEE Trans. Inf. Theory 15, 258–265 (1969)MathSciNetCrossRefGoogle Scholar
  42. 42.
    Kramer, C., Mckay, B., Belina, J.: Probabilistic neural network array architecture for ECG classification. In: Proceedings of the Annual International Conference on IEEE Engineering in Medicine and Biology Society, vol. 17, pp. 807–808 (1995)Google Scholar
  43. 43.
    Musavi, M., Chan, K., Hummels, D., Kalantri, K.: On the generalization ability of neural-network classifier. IEEE Trans. Pattern Anal. Mach. Intell. 16, 659–663 (1994)CrossRefGoogle Scholar
  44. 44.
    Raghu, P., Yegnanarayana, B.: Supervised texture classification using a probabilistic neural network and constraint satisfaction model. IEEE Trans. Neural Netw. 9, 516–522 (1998)CrossRefGoogle Scholar
  45. 45.
    Romero, R., Touretzky, D., Thibadeau, G.: Optical Chinese character recognition using probabilistic neural networks. Pattern Recognit. 3, 1279–1292 (1997)CrossRefGoogle Scholar
  46. 46.
    Streit, R.L., Luginbuhl, T.: Maximum likelihood training of probabilistic neural networks. IEEE Trans. Neural Netw. 5, 764–783 (1994)CrossRefGoogle Scholar
  47. 47.
    Burrascano, P.: Learning vector quantization for the probabilistic neural network. IEEE Trans. Neural Netw. 2, 458–461 (1991)CrossRefGoogle Scholar
  48. 48.
    Zaknich, A.: A vector quantization reduction method for the probabilistic neural network. In: Proceedings of the IEEE International Conference on Neural Networks: Piscataway, NJ, pp. 1117–1120 (1997)Google Scholar
  49. 49.
    Specht, D.: Enhancements to the probabilistic neural networks. In: Proceedings of the IEEE International Joint Conference on Neural Networks: Baltimore, MD, pp. 761–768 (1992)Google Scholar
  50. 50.
    Mao, K., Tan, K.C., Ser, W.: Probabilistic neural-network structure determination for pattern classification. IEEE Trans. Neural Netw. 11(4), 501–507 (2000)CrossRefGoogle Scholar
  51. 51.
    Jones, M., Marron, J., Sheather, S.: A brief survey of bandwidth selection for density estimation. J. Am. Stat. Assoc. 91, 401–407 (1996)MathSciNetCrossRefGoogle Scholar
  52. 52.
    Földes, A., Révész, P.: A general method for density estimation. Stud. Sci. Math. Hung. 9, 81–92 (1974)MathSciNetzbMATHGoogle Scholar
  53. 53.
    Nikolsky, S.: A Course of Mathematical Analysis. Mir Publishers, Moscow (1977)Google Scholar
  54. 54.
    Szegö, G.: Orthogonal Polynomials, vol. 23. American Mathematical Society, Colloquium Publications (1959)Google Scholar
  55. 55.
    Sansone, G.: Orthogonal Functions. Interscience Publishers Inc., New York (1959)zbMATHGoogle Scholar
  56. 56.
    Alexits, G.: Convergence Problems of Orthogonal Series. Akademiai Kiado, Hungary, Budapest (1961)Google Scholar
  57. 57.
    Zygmund, A.: Trigonometric Series. Cambridge University Press, Cambridge (1959)zbMATHGoogle Scholar
  58. 58.
    Sjölin, P.: Convergence almost everywhere of certain singular integrals and multiple Fourier series. Ark. Math. 9, 65–90 (1971)MathSciNetCrossRefGoogle Scholar
  59. 59.
    Yamato, H.: Sequential estimation of a continuous probability density function and the mode. Bull. Math. Stat. 14, 1–12 (1971)MathSciNetCrossRefGoogle Scholar
  60. 60.
    Nadaraya, E.A.: On estimating regression. Theory Probab. Appl. 9(1), 141–142 (1964)CrossRefGoogle Scholar
  61. 61.
    Watson, G.S.: Smooth regression analysis. Sankhyā: Indian J. Stat. Ser. A 359–372 (1964)Google Scholar
  62. 62.
    Devroye, L., Wagner, T.: On the convergence of kernel estimators of regression functions with applications in discrimination. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 51, 15–21 (1980)MathSciNetCrossRefGoogle Scholar
  63. 63.
    Devroye, L.: On the almost everywhere convergence of nonparametric regression function estimates. Ann. Stat. 9, 1301–1309 (1981)MathSciNetCrossRefGoogle Scholar
  64. 64.
    Devroye, L., Krzyżak, A.: An equivalence theorem for \(l_1\) convergence of the kernel regression estimate. J. Stat. Plan. Inference 23, 71–82 (1989)Google Scholar
  65. 65.
    Rutkowski, L.: On system identification by nonparametric function fitting. IEEE Trans. Autom. Control AC-27, 225–227 (1982)Google Scholar
  66. 66.
    Rutkowski, L., Rafajłowicz, E.: On global rate of convergence of some nonparametric identification procedures. IEEE Trans. Autom. Control AC-34(10), 1089–1091 (1989)Google Scholar
  67. 67.
    Rutkowski, L.: Identification of MISO nonlinear regressions in the presence of a wide class of disturbance. IEEE Trans. Inf. Theory IT-37, 214–216 (1991)Google Scholar
  68. 68.
    Rutkowski, L.: Multiple Fourier series procedures for extraction of nonlinear regressions from noisy data. IEEE Trans. Signal Process. (1993)Google Scholar
  69. 69.
    Gałkowski, T., Rutkowski, L.: Nonparametric recovery of multivariate functions with applications to system identification. Proc. IEEE 73, 942–943 (1985)CrossRefGoogle Scholar
  70. 70.
    Gałkowski, T., Rutkowski, L.: Nonparametric fitting of multivariable functions. IEEE Trans. Autom. Control AC-31, 785–787 (1986)Google Scholar
  71. 71.
    Devroye, L., Wagner, T.: Nonparametric discrimination and density estimation. Technical report 183, Electronic Research Center, University of Texas (1976)Google Scholar
  72. 72.
    Wahba, G.: Interpolating spline methods for density estimation, variable knots. Technical report 337, Department of Statistics, University of Wisconsin, Madison (1973)Google Scholar
  73. 73.
    Wahba, G.: Optimal convergence properties of variable knot, kernel, and orthogonal series methods for density estimatio. Ann. Stat. (1975)Google Scholar
  74. 74.
    Wahba, G.: Smoothing noisy data with spline function. Numer. Math. (1975)Google Scholar
  75. 75.
    Wahba, G.: Interpolating spline methods for density estimation, equi-spaced knot. Ann. Stat. (1975)Google Scholar
  76. 76.
    Wahba, G.: A survey of some smoothing problems and the method of generalized cross-validation for solving the, TR-457, Department of Statistics, University of Wisconsin, p. brak (1976)Google Scholar
  77. 77.
    Wahba, G.: Spline Models for Observational Data. SIAM, Philadelphia (1990)CrossRefGoogle Scholar
  78. 78.
    Devroye, L.: Necessary and sufficient conditions for the almost everywhere convergence of nearest neighbor regression function estimates. Zeitschrift für Wahrscheinlichkeitstheorie und verwandte Gebiete 61, 467–481 (1982)MathSciNetCrossRefGoogle Scholar
  79. 79.
    Devroye, L., Györfi, L., Krzyżak, A., Lugosi, G.: On the strong universal consistency of nearest neighbor regression function estimates. Ann. Stat. 22, 1371–1385 (1994)MathSciNetCrossRefGoogle Scholar
  80. 80.
    Aizerman, M., Braverman, E., Rozonoer, L.: Theoretical foundations of the potential function method in pattern recognition learning. Autom. Remote Control 25, 821–837 (1964)zbMATHGoogle Scholar
  81. 81.
    Chen, S., Cowan, C., Grant, P.: Orthogonal least squares learning algorithm for radial basis network. IEEE Trans. Neural Netw. 2, 302–309 (1991)CrossRefGoogle Scholar
  82. 82.
    Kecman, V.: Learning and Soft Computing. MIT, Cambridge (2001)zbMATHGoogle Scholar
  83. 83.
    Chui, C.: Wavelets: a Tutorial in Theory and Applications. Academic, Boston (1992)zbMATHGoogle Scholar
  84. 84.
    Meyer, Y.: Wavelets: Algorithms and Applications. SIAM, Philadelphia (1993)zbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Leszek Rutkowski
    • 1
    • 2
    Email author
  • Maciej Jaworski
    • 1
  • Piotr Duda
    • 1
  1. 1.Institute of Computational IntelligenceCzestochowa University of TechnologyCzęstochowaPoland
  2. 2.Information Technology InstituteUniversity of Social SciencesLodzPoland

Personalised recommendations