Advertisement

Estimating Classifier Performance with Genetic Programming

  • Leonardo Trujillo
  • Yuliana Martínez
  • Patricia Melin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6621)

Abstract

A fundamental task that must be addressed before classifying a set of data, is that of choosing the proper classification method. In other words, a researcher must infer which classifier will achieve the best performance on the classification problem in order to make a reasoned choice. This task is not trivial, and it is mostly resolved based on personal experience and individual preferences. This paper presents a methodological approach to produce estimators of classifier performance, based on descriptive measures of the problem data. The proposal is to use Genetic Programming (GP) to evolve mathematical operators that take as input descriptors of the problem data, and output the expected error that a particular classifier might achieve if it is used to classify the data. Experimental tests show that GP can produce accurate estimators of classifier performance, by evaluating our approach on a large set of 500 two-class problems of multimodal data, using a neural network for classification. The results suggest that the GP approach could provide a tool that helps researchers make a reasoned decision regarding the applicability of a classifier to a particular problem.

Keywords

Genetic Programming Problem Data Descriptive Measure Genetic Programming System Terminal Element 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cantú-Paz, E., Kamath, C.: An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems. IEEE Trans. on Syst., Man, and Cyber., Part B 35(5), 915–927 (2005)CrossRefGoogle Scholar
  2. 2.
    Ho, T.K., Basu, M.: Complexity measures of supervised classification problems. IEEE Trans. Pattern Anal. Mach. Intell. 24, 289–300 (2002)CrossRefGoogle Scholar
  3. 3.
    Hordijk, W.: A measure of landscapes. Evol. Comput. 4, 335–360 (1996)CrossRefGoogle Scholar
  4. 4.
    Luke, S., Panait, L.: Lexicographic parsimony pressure. In: Proceedings of GECCO 2002, pp. 829–836. Morgan Kaufmann, San Francisco (2002)Google Scholar
  5. 5.
    Mansilla, E.B., Ho, T.K.: On classifier domains of competence. In: Proceedings of ICPR 2004, vol. 1, pp. 136–139. IEEE Computer Society, Washington, DC, USA (2004)Google Scholar
  6. 6.
    McDermott, J., Galvan-Lopez, E., O’Neill, M.: A fine-grained view of GP locality with binary decision diagrams as ant phenotypes. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 164–173. Springer, Heidelberg (2010)Google Scholar
  7. 7.
    Michie, D., Spiegelhalter, D.J., Taylor, C.C., Campbell, J. (eds.): Machine learning, neural and statistical classification, NJ, USA (1994)Google Scholar
  8. 8.
    Ou, G., Murphey, Y.L.: Multi-class pattern classification using neural networks. Pattern Recogn. 40, 4–18 (2007)CrossRefzbMATHGoogle Scholar
  9. 9.
    Poli, R., Graff, M.: There is a free lunch for hyper-heuristics, genetic programming and computer scientists. In: Vanneschi, L., Gustafson, S., Moraglio, A., De Falco, I., Ebner, M. (eds.) EuroGP 2009. LNCS, vol. 5481, pp. 195–207. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  10. 10.
    Poli, R., Graff, M., McPhee, N.F.: Free lunches for function and program induction. In: Proceedings of FOGA 2009, pp. 183–194. ACM, New York (2009)Google Scholar
  11. 11.
    Poli, R., Vanneschi, L.: Fitness-proportional negative slope coefficient as a hardness measure for genetic algorithms. In: Proceedings of GECCO 2007, pp. 1335–1342. ACM, New York (2007)Google Scholar
  12. 12.
    Silva, S., Almeida, J.: Gplab–a genetic programming toolbox for matlab. In: Proceedings of the Nordic MATLAB Conference, pp. 273–278 (2003)Google Scholar
  13. 13.
    Silva, S., Costa, E.: Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories. Genetic Programming and Evolvable Machines 10(2), 141–179 (2009)CrossRefGoogle Scholar
  14. 14.
    Sohn, S.Y.: Meta analysis of classification algorithms for pattern recognition. IEEE Trans. Pattern Anal. Mach. Intell. 21, 1137–1144 (1999)CrossRefGoogle Scholar
  15. 15.
    Vanneschi, L., Castelli, M., Silva, S.: Measuring bloat, overfitting and functional complexity in genetic programming. In: Proceedings of GECCO 2010, pp. 877–884. ACM, New York (2010)Google Scholar
  16. 16.
    Vanneschi, L., Tomassini, M., Collard, P., Vérel, S., Pirola, Y., Mauri, G.: A comprehensive view of fitness landscapes with neutrality and fitness clouds. In: Ebner, M., O’Neill, M., Ekárt, A., Vanneschi, L., Esparcia-Alcázar, A.I. (eds.) EuroGP 2007. LNCS, vol. 4445, pp. 241–250. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  17. 17.
    Whitley, D., Watson, J.: Complexity theory and the no free lunch theorem, ch. 11, pp. 317–339 (2005)Google Scholar
  18. 18.
    Wolpert, D., Macready, W.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Leonardo Trujillo
    • 1
  • Yuliana Martínez
    • 1
  • Patricia Melin
    • 1
  1. 1.Instituto Tecnológico de TijuanaTijuanaMéxico

Personalised recommendations