Maximum Margin Decision Surfaces for Increased Generalisation in Evolutionary Decision Tree Learning

  • Alexandros Agapitos
  • Michael O’Neill
  • Anthony Brabazon
  • Theodoros Theodoridis
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6621)


Decision tree learning is one of the most widely used and practical methods for inductive inference. We present a novel method that increases the generalisation of genetically-induced classification trees, which employ linear discriminants as the partitioning function at each internal node. Genetic Programming is employed to search the space of oblique decision trees. At the end of the evolutionary run, a (1+1) Evolution Strategy is used to geometrically optimise the boundaries in the decision space, which are represented by the linear discriminant functions. The evolutionary optimisation concerns maximising the decision-surface margin that is defined to be the smallest distance between the decision-surface and any of the samples. Initial empirical results of the application of our method to a series of datasets from the UCI repository suggest that model generalisation benefits from the margin maximisation, and that the new method is a very competent approach to pattern classification as compared to other learning algorithms.


Support Vector Machine Genetic Programming Generalisation Error Maximum Margin Model Generalisation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Koza, J.R.: Genetic Programming: on the programming of computers by means of natural selection. MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  2. 2.
    Vladimir, V.: The nature of statistical learning theory, 2nd edn. Springer, Heidelberg (1999)Google Scholar
  3. 3.
    Koza, J.R.: Concept formation and decision tree induction using the genetic programming paradigm. In: Schwefel, H.-P., Männer, R. (eds.) PPSN 1990. LNCS, vol. 496, pp. 124–128. Springer, Heidelberg (1991)CrossRefGoogle Scholar
  4. 4.
    Folino, G., Pizzuti, C., Spezzano, G.: Genetic Programming and Simulated Annealing: A Hybrid Method to Evolve Decision Trees. In: Poli, R., Banzhaf, W., Langdon, W.B., Miller, J., Nordin, P., Fogarty, T.C. (eds.) EuroGP 2000. LNCS, vol. 1802, pp. 294–303. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  5. 5.
    Eggermont, J.: Evolving Fuzzy Decision Trees with Genetic Programming and Clustering. In: Foster, J.A., Lutton, E., Miller, J., Ryan, C., Tettamanzi, A.G.B. (eds.) EuroGP 2002. LNCS, vol. 2278, pp. 71–82. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  6. 6.
    Rouwhorst, S.E., Engelbrecht, A.P.: Searching the forest: Using decision trees as building blocks for evolutionary search in classification databases. In: Proceedings of the, Congress on Evolutionary Computation CEC 2000, vol. 1, pp. 633–638 (2000)Google Scholar
  7. 7.
    Bot, M., Langdon, W.B.: Application of genetic programming to induction of linear classification trees. In: Proceedings of the Eleventh Belgium/Netherlands Conference on Artificial Intelligence, BNAIC 1999 (1999)Google Scholar
  8. 8.
    Marmelstein, R.E., Lamont, G.B.: Pattern classification using a hybrid genetic program decision tree approach. In: Genetic Programming 1998: Proceedings of the Third Annual Conference (1998)Google Scholar
  9. 9.
    Tsakonas, A.: A comparison of classification accuracy of four genetic programming-evolved intelligent structures. Information Sciences 176(6), 691–724 (2006)CrossRefGoogle Scholar
  10. 10.
    Mugambi, E.M., Hunter, A., Oatley, G., Kennedy, L.: Polynomial-fuzzy decision tree structures for classifying medical data. Knowledge-Based Systems 17(2-4), 81–87 (2004)CrossRefGoogle Scholar
  11. 11.
    Mitchel, T.: Machine Learning. McGraw-Hill, New York (1997)Google Scholar
  12. 12.
    Estrada-Gil, J.K., Fernandez-Lopez, J.C., Hernandez-Lemus, E., Silva-Zolezzi, I., Hidalgo-Miranda, A., Jimenez-Sanchez, G., Vallejo-Clemente, E.E.: GPDTI: A genetic programming decision tree induction method to find epistatic effects in common complex diseases. Bioinformatics 13(13), i167–i174 (2007)CrossRefGoogle Scholar
  13. 13.
    Kuo, C.-S., Hong, T.-P., Chen, C.-L.: Applying genetic programming technique in classification trees. Soft Computing 11(12), 1165–1172 (2007)CrossRefzbMATHGoogle Scholar
  14. 14.
    Haruyama, S., Zhao, Q.: Designing smaller decision trees using multiple objective optimization based gps. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 6, p. 5 (2002)Google Scholar
  15. 15.
    Folino, G., Pizzuti, C., Spezzano, G.: Improving induction decision trees with parallel genetic programming. In: Proceedings 10th Euromicro Workshop on Parallel, Distributed and Network-based Processing, Canary Islands, January 9-11, pp. 181–187. IEEE, Los Alamitos (2002)CrossRefGoogle Scholar
  16. 16.
    Agapitos, A., O’Neill, M., Brabazon, A.: Evolutionary Learning of Technical Trading Rules without Data-Mining Bias. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 294–303. Springer, Heidelberg (2010)Google Scholar
  17. 17.
    Vapnik, V.N., Chervonenkis, A.Y.: On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications 16(2), 264–280 (1971)CrossRefzbMATHGoogle Scholar
  18. 18.
    Shawe-Taylor, J., Bartlett, P.L., Williamson, R.C., Anthony, M.: Structural risk minimization over data-dependent hierarchies. IEEE Transactions on Information Theory 44(5) (1998)Google Scholar
  19. 19.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  20. 20.
    Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J.: UCI repository of machine learning databases (1998)Google Scholar
  21. 21.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: an update. SIGKDD Explor. Newsl. 11, 10–18 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Alexandros Agapitos
    • 1
  • Michael O’Neill
    • 1
  • Anthony Brabazon
    • 1
  • Theodoros Theodoridis
    • 2
  1. 1.Financial Mathematics and Computation Research Cluster Natural Computing Research and Applications GroupUniversity College DublinIreland
  2. 2.School of Computer Science and Electronic EngineeringUniversity of EssexUK

Personalised recommendations