Abstract
We introduce a novel framework for classification with a rejection option that consists of simultaneously learning two functions: a classifier along with a rejection function. We present a full theoretical analysis of this framework including new data-dependent learning bounds in terms of the Rademacher complexities of the classifier and rejection families as well as consistency and calibration results. These theoretical guarantees guide us in designing new algorithms that can exploit different kernel-based hypothesis sets for the classifier and rejection functions. We compare and contrast our general framework with the special case of confidence-based rejection for which we devise alternative loss functions and algorithms as well. We report the results of several experiments showing that our kernel-based algorithms can yield a notable improvement over the best existing confidence-based rejection algorithm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bartlett, P., Wegkamp, M.: Classification with a reject option using a hinge loss. JMLR 9, 1823–1840 (2008)
Bounsiar, A., Grall, E., Beauseroy, P.: Kernel based rejection method for supervised classification. WASET 3, 312–321 (2007)
Capitaine, H.L., Frelicot, C.: An optimum class-rejective decision rule and its evaluation. In: ICPR (2010)
Chaudhuri, K., Zhang, C.: Beyond disagreement-based agnostic active learning. In: NIPS (2014)
Chow, C.: An optimum character recognition system using decision function. IEEE T. C. (1957)
Chow, C.: On optimum recognition error and reject trade-off. IEEE T. C. (1970)
Cortes, C., DeSalvo, G., Mohri, M.: Learning with rejection. In: arXiv (2016)
I. Cvx Research. CVX: Matlab software for disciplined convex programming, version 2.0, August 2012
DeSalvo, G., Mohri, M., Syed, U.: Learning with deep cascades. In: Chaudhuri, K., Gentile, C., Zilles, S. (eds.) ALT 2015. LNCS, vol. 9355, pp. 254–269. Springer, Heidelberg (2015). doi:10.1007/978-3-319-24486-0_17
Dubuisson, B., Masson, M.: Statistical decision rule with incomplete knowledge about classes. Pattern Recognit. 26, 155–165 (1993)
El-Yaniv, R., Wiener, Y.: On the foundations of noise-free selective classification. JMLR 11, 1605–1641 (2010)
El-Yaniv, R., Wiener, Y.: Agnostic selective classification. In: NIPS (2011)
Elkan, C.: The foundations of cost-sensitive learning. In: IJCAI (2001)
Freund, Y., Mansour, Y., Schapire, R.: Generalization bounds for averaged classifiers. Ann. Stat. (2004)
Fumera, G., Roli, F.: Support vector machines with embedded reject option. In: ICPR (2002)
Fumera, G., Roli, F., Giacinto, G.: Multiple reject thresholds for improving classification reliability. In: ICAPR (2000)
Grandvalet, Y., Keshet, J., Rakotomamonjy, A., Canu, S.: Suppport vector machines with a reject option. In: NIPS (2008)
Herbei, R., Wegkamp, M.: Classification with reject option. Can. J. Stat. (2005)
Koltchinskii, V., Panchenko, D.: Empirical margin distributions and bounding the generalization error of combined classifiers. Ann. Stat. 30, 1–50 (2002)
Landgrebe, T., Tax, D., Paclik, P., Duin, R.: Interaction between classification and reject performance for distance-based reject-option classifiers. PRL 27, 908–917 (2005)
Ledoux, M., Talagrand, M.: Probability in Banach Spaces: Isoperimetry and Processes. Springer, New York (1991)
Littman, M., Li, L., Walsh, T.: Knows what it knows: a framework for self-aware learning. In: ICML (2008)
Long, P.M., Servedio, R.A.: Consistency versus realizable H-consistency for multiclass classification. In: ICML, vol. 3, pp. 801–809 (2013)
Melvin, I., Weston, J., Leslie, C.S., Noble, W.S.: Combining classifiers for improved classification of proteins from sequence or structure. BMCB 9, 1 (2008)
Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. The MIT Press, Cambridge (2012)
Pereira, C.S., Pires, A.: On optimal reject rules and ROC curves. PRL 26, 943–952 (2005)
Pietraszek, T.: Optimizing abstaining classifiers using ROC. In: ICML (2005)
Tax, D., Duin, R.: Growing a multi-class classifier with a reject option. Pattern Recognit. Lett. 29, 1565–1570 (2008)
Tortorella, F.: An optimal reject rule for binary classifiers. In: ICAPR (2001)
Trapeznikov, K., Saligrama, V.: Supervised sequential classification under budget constraints. In: AISTATS (2013)
Wang, J., Trapeznikov, K., Saligrama, V.: An LP for sequential learning under budgets. In: JMLR (2014)
Yuan, M., Wegkamp, M.: Classification methods with reject option based on convex risk minimizations. In: JMLR (2010)
Yuan, M., Wegkamp, M.: SVMs with a reject option. In: Bernoulli (2011)
Zhang, C., Chaudhuri, K.: The extended Littlestone’s dimension for learning with mistakes and abstentions. In: COLT (2016)
Acknowledgments
This work was partly funded by NSF IIS-1117591, CCF-1535987, and DGE-1342536.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Cortes, C., DeSalvo, G., Mohri, M. (2016). Learning with Rejection. In: Ortner, R., Simon, H., Zilles, S. (eds) Algorithmic Learning Theory. ALT 2016. Lecture Notes in Computer Science(), vol 9925. Springer, Cham. https://doi.org/10.1007/978-3-319-46379-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-46379-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46378-0
Online ISBN: 978-3-319-46379-7
eBook Packages: Computer ScienceComputer Science (R0)