Substructural Surrogates for Learning Decomposable Classification Problems

  • Albert Orriols-Puig
  • Kumara Sastry
  • David E. Goldberg
  • Ester Bernadó-Mansilla
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4998)


This paper presents a learning methodology based on a substructural classification model to solve decomposable classification problems. The proposed method consists of three important components: (1) a structural model, which represents salient interactions between attributes for a given data, (2) a surrogate model, which provides a functional approximation of the output as a function of attributes, and (3) a classification model, which predicts the class for new inputs. The structural model is used to infer the functional form of the surrogate. Its coefficients are estimated using linear regression methods. The classification model uses a maximally-accurate, least-complex surrogate to predict the output for given inputs. The structural model that yields an optimal classification model is searched using an iterative greedy search heuristic. Results show that the proposed method successfully detects the interacting variables in hierarchical problems, groups them in linkages groups, and builds maximally accurate classification models. The initial results on non-trivial hierarchical test problems indicate that the proposed method holds promise and also shed light on several improvements to enhance the capabilities of the proposed method.


Linkage Group Test Error Surrogate Function Greedy Search Parity Block 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Baluja, S.: Incorporating a priori Knowledge in Probabilistic-Model Based Optimization. In: Pelikan, M., Sastry, K., Cantú-Paz, E. (eds.) Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications ch. 9, pp. 205–219. Springer, Berlin (2006)CrossRefGoogle Scholar
  2. 2.
    Bernadó-Mansilla, E., Garrell, J.M.: Accuracy-Based Learning Classifier Systems: Models, Analysis and Applications to Classification Tasks. Evolutionary Computation 11(3), 209–238 (2003)CrossRefGoogle Scholar
  3. 3.
    Butz, M.V.: Rule-Based Evolutionary Online Learning Systems: A Principled Approach to LCS Analysis and Design. In: Studies in Fuzziness and Soft Computing, vol. 109. Springer, Heidelberg (2006)Google Scholar
  4. 4.
    Butz, M.V., Pelikan, M., Llorà, X., Goldberg, D.E.: Automated Global Structure Extraction for Effective Local Building Block Processing in XCS. Evolutionary Computation 14(3), 345–380 (2006)CrossRefGoogle Scholar
  5. 5.
    Drapper, N.R., Smith, H.: Applied Regression Analysis. John Wiley & Sons, New York (1966)Google Scholar
  6. 6.
    Gibson, J.J.: The Ecological Approach to Visual Perception. Lawrence Erlbaum Associates, Mahwah (1979)Google Scholar
  7. 7.
    Goldberg, D.E.: Genetic Algorithms in Search, Optimization & Machine Learning, 1st edn. Addison Wesley, Reading (1989)zbMATHGoogle Scholar
  8. 8.
    Goldberg, D.E.: The Design of Innovation: Lessons from and for Competent Genetic Algorithms, 1st edn. Kluwer Academic Publishers, Dordrecht (2002)CrossRefzbMATHGoogle Scholar
  9. 9.
    Harik, G.: Linkage Learning via Probabilistic Modeling in the ECGA. Technical report. University of Illinois at Urbana-Champaign, Urbana, IL (January 1999) (IlliGAL Report No. 99010)Google Scholar
  10. 10.
    Harik, G.R., Lobo, F.G., Sastry, K.: Linkage Learning via Probabilistic Modeling in the ECGA. In: Pelikan, M., Sastry, K., Cantú-Paz, E. (eds.) Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications ch. 3, pp. 39–61. Springer, Berlin (2006) (Also IlliGAL Report No. 99010) CrossRefGoogle Scholar
  11. 11.
    Holland, J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press (1975)Google Scholar
  12. 12.
    De Jong, K.A., Spears, W.M.: Learning Concept Classification Rules Using Genetic Algorithms. In: Proceedings of the International Joint Conference on Artificial Intelligence, Sidney, Australia, pp. 651–656 (1991)Google Scholar
  13. 13.
    Keerthi, S.S., Lin, C.J.: Asymptotic Behaviors of Support Vector Machines with Gaussian Kernel. Neural Computation 15(7), 1667–1689 (2003)CrossRefzbMATHGoogle Scholar
  14. 14.
    Korst, J., Aarts, E.: Simulated Annealing and Boltzmann Machines. Wiley-Interscience, New York (1997)zbMATHGoogle Scholar
  15. 15.
    Kovacs, T.: Deletion Schemes for Classifier Systems. In: GECCO 1999: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 329–336. Morgan Kaufmann, San Francisco (1999)Google Scholar
  16. 16.
    Llorà, X., Sastry, K., Goldberg, D.E., de la Ossa, L.: The χ-ary extended compact classifier system: Linkage learning in Pittsburgh LCS. In: Proceedings of the 2006 Genetic and Evolutionary Computation Conference Workshop Program. ACM Press, Berlin (2006) (Also IlliGAL Report No. 2006015) Google Scholar
  17. 17.
    Llorà, X., Sastry, K., Yu, T.-L., Goldberg, D.E.: Do not match, inherit: Fitness surrogates for genetics-based machine learning. In: Proceedings of the 2007 Genetic and Evolutionary Computation Conference, vol. 2, pp. 1798–1805 (2007)Google Scholar
  18. 18.
    Pelikan, M.: Hierarchical Bayesian Optimization Algorithm: Toward a new Generation of Evolutionary Algorithms. Springer, Berlin (2005)CrossRefzbMATHGoogle Scholar
  19. 19.
    Pelikan, M., Sastry, K.: Fitness inheritance in the Bayesian optimization algorithm. In: Proceedings of the 2004 Genetic and Evolutionary Computation Conference, vol. 2, pp. 48–59 (2004) (Also IlliGAL Report No. 2004009)Google Scholar
  20. 20.
    Pelikan, M., Sastry, K., Cantú-Paz, E. (eds.): Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications. Studies in Computational Intelligence, vol. 33. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  21. 21.
    Platt, J.: Fast Training of Support Vector Machines using Sequential Minimal Optimization. In: Advances in Kernel Methods - Support Vector Learning, pp. 557–563. MIT Press, Cambridge (1998)Google Scholar
  22. 22.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo (1995)Google Scholar
  23. 23.
    Rao, C.R., Toutenburg, H.: Linear Models: Least Squares and Alternatives. Springer, Berlin (1999)zbMATHGoogle Scholar
  24. 24.
    Recktenwald, G.: Numerical Methods with MATLAB: Implementations and Applications. Prentice Hall, Englewood Cliffs (2000)Google Scholar
  25. 25.
    Sastry, K., Goldberg, D.E.: Probabilistic Model Building and Competent Genetic Programming. In: Riolo, R.L., Worzel, B. (eds.) Genetic Programming Theory and Practise, ch. 13, pp. 205–220. Kluwer, Dordrecht (2003)CrossRefGoogle Scholar
  26. 26.
    Sastry, K., Lima, C.F., Goldberg, D.E.: Evaluation Relaxation Using Substructural Information and Linear Estimation. In: GECCO 2006: Proceedings of the 8th annual Conference on Genetic and Evolutionary Computation, pp. 419–426. ACM Press, New York (2006)Google Scholar
  27. 27.
    Sastry, K., Pelikan, M., Goldberg, D.E.: Efficiency enhancement of genetic algorithms via building-block-wise fitness estimation. In: Proceedings of the IEEE International Conference on Evolutionary Computation, pp. 720–727 (2004) (Also IlliGAL Report No. 2004010)Google Scholar
  28. 28.
    Simon, H.A.: Sciences of the Artificial. MIT Press, Cambridge (1969)Google Scholar
  29. 29.
    Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Comp. 10(7), 1895–1924 (1998)CrossRefGoogle Scholar
  30. 30.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, New York (1995)CrossRefzbMATHGoogle Scholar
  31. 31.
    Wilson, S.W.: Quasi-Darwinian Learning in a Classifier System. In: 4th IWML, pp. 59–65. Morgan Kaufmann, San Francisco (1987)Google Scholar
  32. 32.
    Wilson, S.W.: Classifier Fitness Based on Accuracy. Evolutionary Computation 3(2), 149–175 (1995)CrossRefGoogle Scholar
  33. 33.
    Wilson, S.W.: Generalization in the XCS Classifier System. In: 3rd Annual Conf. on Genetic Programming, pp. 665–674. Morgan Kaufmann, San Francisco (1998)Google Scholar
  34. 34.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)zbMATHGoogle Scholar
  35. 35.
    Yu, T.-L.: A matrix approach for finding extrema: Problems with modularity, hierarchy, and overlap. PhD thesis, University of Illinois at Urbana-Champaign, Urbana, IL (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Albert Orriols-Puig
    • 1
    • 2
  • Kumara Sastry
    • 2
  • David E. Goldberg
    • 2
  • Ester Bernadó-Mansilla
    • 1
  1. 1.Grup de Recerca en Sistemes Intel·ligents, Enginyeria i Arquitectura La SalleUniversitat Ramon LlullBarcelonaSpain
  2. 2.Illinois Genetic Algorithms Laboratory, Department of Industrial and Enterprise Systems EngineeringUniversity of IllinoisUrbana-ChampaignUSA

Personalised recommendations