Skip to main content
SpringerLink
Log in

How should accuracy be measured?

  • Chapter
Machine Learning Methods for Ecological Applications

Abstract

The apparently simple problem of measuring classification accuracy is reviewed. Guidelines are suggested for the choice of appropriate measures of classification accuracy, including those that assess improvement over chance and the imposition of misclassification costs. Particular attention is paid to the selection of appropriate data for accuracy assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Augustin, N. H., Mugglestone, M. A. and Buckland, S. T. 1996. An autologistic model for the spatial distribution of wildlife. Journal of Applied Ecology, 33: 339–347.

    Article  Google Scholar 

  • Blayo, F., Chevenal, Y., Guérin-Dugué, A., Chentouf, R., Aviles-Cruz, G, Madrenas, J., Moreno, M. and Voz, J. L. 1995. Enhanced Learning for Evolutive Neural Architecture.

    Google Scholar 

  • Deliverable R3-B4-P Task B4: Benchmarks. ESPIRIT Basic Research Project Number 6891. Available from ftp.dice.ucl.ac.be/pub/neural-nets/ELANA/databases

    Google Scholar 

  • Bradley, A. P. 1997. The use of the area under the ROC curve in the estimation of machine learning algorithms. Pattern Recognition, 30: 1145–1159.

    Article  Google Scholar 

  • Brennan, L. A., Block, W. M. and Gutiérrez, R. J. 1986. The use of multivariate statistics for developing habitat suitability index models. In: Wildlife 2000: Modelling habitat relationships of terrestrial vertebrates, ed. J. A. Verner, M. L. Morrison and C. J. Ralph, pp. 177–182. University of Wisconsin Press, Madison.

    Google Scholar 

  • Buckland, S. T. and Elston, D. A. 1993. Empirical models for the spatial distribution of wildlife. Journal of Applied Ecology, 30: 478–495.

    Article  Google Scholar 

  • Capen, D. E., Fenwick, J. W., Inkley, D. B. and Boynton, A. C. 1986. Multivariate models of songbird habitat in New England forests. In: Wildlife 2000: Modelling habitat relationships of terrestrial vertebrates, ed. J. A. Verner, M. L. Morrison and C. J. Ralph, pp. 171–175. University of Wisconsin Press, Madison.

    Google Scholar 

  • Chatfield, C. 1995. Model uncertainty, data mining and statistical inference. Journal of the Royal Statistical Society, Series A, 158: 419–466.

    Article  Google Scholar 

  • Cohen, P. 1995. Empirical methods for artificial intelligence. MIT, Cambridge, MA.

    Google Scholar 

  • Deleo, J. M. 1993. Receiver operating characteristic laboratory (ROCLAB): software for developing decision strategies that account for uncertainty, pp 318–325 in Proceedings of the Second International Symposium on Uncertainty Modelling and Analysis. IEEE, Computer Society Press, College Park, MD.

    Google Scholar 

  • Deleo, J. M. and Campbell, G. 1990. The fuzzy receiver operating characteristic function and medical decisions with uncertainty. In: Proceedings of the First International Symposium on Uncertainty Modelling and Analysis. IEEE, Computer Society Press, College Park, MD.

    Google Scholar 

  • Donázar, J. A., Hiraldo, F. and Bustamante, J. 1993. Factors influencing nest site selection, breeding density and breeding success in bearded vulture (Gypaetus barbatus). Journal of Applied Ecology, 30: 504–514.

    Article  Google Scholar 

  • Efron, B. and Tibshirani, R. 1997. Improvements on cross-validation: the.632+ bootstrap method. Journal of the American Statistical Association, 92: 548–506.

    Google Scholar 

  • Fielding, A. H. and Bell, J. F. 1997. A review of methods for the assessment of prediction errors in conservation presence/absence models. Environmental Conservation, 24: 38–49.

    Article  Google Scholar 

  • Fielding, A. H. and Haworth, P. F. 1995. Testing the generality of bird-habitat models. Conservation Biology, 9: 1466–1481.

    Article  Google Scholar 

  • Forbes, A. D. 1995. Classification algorithm evaluation: five performance measures based on confusion matrices. Journal of Clinical Monitoring, 11: 189–206.

    Article  PubMed  CAS  Google Scholar 

  • Guégan, J-F, Lek, S. and Oberdorff, T. 1998. Energy availability and habitat heterogeneity predict global riverine fish diversity. Nature, 391: 382–384.

    Article  Google Scholar 

  • Gordon, A. D. 1981. Classification: methods for the exploratory analysis of multivariate data. Chapman and Hall, London.

    Google Scholar 

  • Hand, D. J. 1997. Construction and assessment of classification rules. Wiley, Chicester.

    Google Scholar 

  • Henery, R. J. 1994. Classification. Pp 6–16 in D. Michie, D. J. Spiegelhalter and C. C. Taylor (eds) Machine learning, neural and statistical classification. Ellis Horwood, New York.

    Google Scholar 

  • Huberty, C. J. 1994. Applied Discriminant Analysis. Wiley Interscience, New York.

    Google Scholar 

  • Landis, J. R. and Koch, G. C. 1977. The measurement of observer agreement for categorical data. Biometrics, 33: 159–174.

    Article  PubMed  CAS  Google Scholar 

  • Lin, T. C. and Pourhmadi, M. 1998. Nonparametric and non-linear models and data mining in time series: a case study on the Canadian lynx data. Applied Statistics, 47: 187–201.

    Google Scholar 

  • Lynn, H., Mohler, C. L., DeGloria, S. D. and McCulloch, C. E. 1995, Error assessment in decision-tree models applied to vegetation analysis. Landscape Ecology, 10: 323–335.

    Article  Google Scholar 

  • Ma, Z. and Redmond, R. L. 1995. Tau coefficients for accuracy assessment of classifications of remote sensing data. Photogrammetric Engineering and Remote Sensing, 61: 435–439.

    Google Scholar 

  • Madger, L. S. and Hughes, J. P. 1997. Logistic regression when the outcome is measured with uncertainty. Amer dan Journal of Epidemiology, 146: 195–203.

    Article  Google Scholar 

  • Marsden, S. and Fielding, A. H. In press. Habitat associations of parrots on the islands of Bum, Seram and Sumba. Journal of Biogeography.

    Google Scholar 

  • Mastrorillo, S., Lek, S., Dauba, F. and Belaud, A. 1997. The use of artificial neural networks to predict the presence of small-bodied fish in a river. Freshwater Biology, 38: 237–246.

    Article  Google Scholar 

  • May, L. 1998. Vocalizations in the magpie and the corncrake: methods of analysis, individual differences and geographical variation (2 volumes). Unpublished PhD thesis, the Manchester Metropolitan University.

    Google Scholar 

  • Michie, D., Spiegelhalter, D. J. and Taylor, C. C. 1994. Machine learning, neural and statistical classification. Ellis Horwood, New York.

    Google Scholar 

  • Morrison, M. L., Timoss, I. C. and With, K. A. 1987. Development and testing linear regression models predicting bird-habitat relationships. Journal of Wildlife Management 51:247–253.

    Article  Google Scholar 

  • Morrison, M. L., Marcot, B. G. and Mannan, R. W. 1992. Wildlife Habitat Relationships. Concepts and Applications. University Wisconsin Press, Madison.

    Google Scholar 

  • Osborne, P. E and Tigar, B. J. 1992. Interpreting bird atlas data using logistic models: an example from Lesotho, Southern Africa. Journal of Applied Ecology 29: 55–62.

    Article  Google Scholar 

  • Pereira, J. M. C. and Itami, R. C. 1991. GIS-based habitat modelling using logistic multiple regression: a study of the Mt. Graham red squirrel. Photogrammetric Engineering & Remote Sensing, 57: 1475–1486.

    Google Scholar 

  • Provost, F. and Fawcett, T. 1997. Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. Pp 43–48 in Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD-97). AAAI Press.

    Google Scholar 

  • Reby, D., Joachim, J., Lauga, J., Lek, S. and Aulagnier, S 1998. Individuality in the groans of fallow deer (Dama dama) bucks. Journal of the Zoological Society of London, 245: 79–84.

    Article  Google Scholar 

  • Rencher, A. C. 1995. Methods of Multivariate Analysis. Wiley, New York.

    Google Scholar 

  • Riordan, P. 1998. Unsupervised recognition of indivdual tigers and snow leopards from their footprints. Animal Conservation, 1: 253–262.

    Article  Google Scholar 

  • Ripley, B. D. 1994. Neural network and related methods for classification. Journal of the Royal Statistical Society, Series B, 56: 409–456.

    Google Scholar 

  • Ruttiman, U. E. 1994. Statistical approaches to development and validation of predictive instruments. Critical Care Clinics 10: 19–35.

    Google Scholar 

  • Salzberg, S. L. 1997. On comparing classifiers: Pitfalls to avoid and a recommended approach Data mining and knowledge discovery, 1:317–328

    Article  Google Scholar 

  • Schaafsma, W. and van Vark, G. N. 1979. Classification and discrimination problems with applications. Part Ha. Statistica Neerlandica, 33:91–126.

    Article  Google Scholar 

  • Schiffers, J. 1997. A classification approach incorporating misclassification costs. Intelligent Data Analysis, 1(1): an electronic journal http://www.elsevier.com/locate/ida

  • Turney, P. D. 1995. Cost-sensitive classification: empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research, 2: 369–409.

    Google Scholar 

  • Turney, P. D. 1998. Types of Cost. Accessed 21/10/98, document dated 28 April 1998 http://ai.iit.nrc.ca/bibliographies/cost-types.html

    Google Scholar 

  • Verbyla, D. L. and Litvaitis, J. A. 1989. Resampling methods for evaluating classification accuracy of wildlife habitat models. Environmental Management, 13: 783–787.

    Article  Google Scholar 

  • Zweig, M. H. and Campbell, G. 1993. Receiver-Operating Characteristic (ROC) Plots: A fundamental evaluation tool in clinical medicine. Clinical Chemistry 39: 561–577.

    PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Behavioural and Environmental Biology Research Group, Biological Sciences, the Manchester Metropolitan University, Manchester, M1 5GD, UK

    Alan Fielding

Authors
  1. Alan Fielding
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Biological Science, The Manchester Metropolitan University, UK

    Alan H. Fielding

Rights and permissions

Reprints and permissions

Copyright information

© 1999 Springer Science+Business Media New York

About this chapter

Cite this chapter

Fielding, A. (1999). How should accuracy be measured?. In: Fielding, A.H. (eds) Machine Learning Methods for Ecological Applications. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-5289-5_8

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-5289-5_8

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-7413-8

  • Online ISBN: 978-1-4615-5289-5

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation