• Thomas A. Runkler


Regression estimates functional dependencies between features to understand dependencies and to systematically control systems. Linear regression models can be efficiently computed from covariances but are restricted to linear dependencies. Substitution allows us to identify specific nonlinear dependencies by linear regression. Robust regression finds models that are robust against outliers. A popular family of nonlinear regression methods are universal approximators.We present two well-known examples for universal approximators from the field of artificial neural networks: the multilayer perceptron and radial basis function networks. Universal approximators can realize arbitrarily small training errors, but cross-validation is required to find models with low validation errors that generalize well on other data sets. Feature selection allows us to include only relevant features in regression models leading to more accurate models.


Feature Selection Hide Layer Radial Basis Function Radial Basis Function Network Multilayer Perceptron 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    H. Akaike. A new look at the statistical model identification. IEEE Transactions on Automatic Control, AC–19:716–723, 1974.Google Scholar
  2. 2.
    P. Craven and G. Wahba. Smoothing noisy data with spline functions: Estimating the correct degree of smoothing by the method of generalized cross–validation. Numerical Mathematics, 31:377–403, 1979.MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    R. Hecht-Nielsen. Neurocomputing. Addison-Wesley, 1990.Google Scholar
  4. 4.
    K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989.CrossRefGoogle Scholar
  5. 5.
    P. J. Huber. Robust Statistics. Wiley, New York, 2nd edition, 2009.MATHCrossRefGoogle Scholar
  6. 6.
    A. N. Kolmogorov. On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition. Doklady Akademii Nauk SSSR, 144:679–681, 1957.MathSciNetGoogle Scholar
  7. 7.
    R. Meiria and J. Zahavi. Using simulated annealing to optimize the feature selection problem in marketing applications. European Journal of Operational Research, 171(3):842–858, 2006.CrossRefGoogle Scholar
  8. 8.
    J. Moody and C. Darken. Fast learning in networks of locally-tuned processing units. Neural Computation, 1:281–294, 1989.CrossRefGoogle Scholar
  9. 9.
    F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Reviews, 65:386–408, 1958.MathSciNetCrossRefGoogle Scholar
  10. 10.
    P. J. Rousseeuw and A. M. Leroy. Robust Regression and Outlier Detection. Wiley, New York, 1987.MATHCrossRefGoogle Scholar
  11. 11.
    D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error backpropagation. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing. Explorations in the Microstructure of Cognition, volume 1, pages 318–362. MIT Press, Cambridge, 1986.Google Scholar
  12. 12.
    W. Siedlecki and J. Sklansky. A note on genetic algorithms for large–scale feature selection. Pattern Recognition Letters, 10(5):335–347, 1989.MATHCrossRefGoogle Scholar
  13. 13.
    S. Vieira, J. M. Sousa, and T. A. Runkler. Multi-criteria ant feature selection using fuzzy classifiers. In C. A. Coello Coello, S. Dehuri, and S. Ghosh, editors, Swarm Intelligence for Multi-objective Problems in Data Mining, pages 19–36. Springer, 2009.Google Scholar
  14. 14.
    P. J. Werbos. The Roots of Backpropagation: From Ordered Derivatives to Neural Networks and Political Forecasting (Adaptive and Learning Systems for Signal Processing, Communications and Control Series). Wiley–Interscience, 1994.Google Scholar

Copyright information

© Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden 2012

Authors and Affiliations

  1. 1.MünchenGermany

Personalised recommendations