Abstract
The Conformal Prediction framework guarantees error calibration in the online setting, but its practical usefulness in real-world problems is affected by its efficiency, i.e. the size of the prediction region. Narrow prediction regions that maintain validity would be the most useful conformal predictors. In this work, we use the efficiency of conformal predictors as a measure to perform model selection in classifiers. We pose this objective as an optimization problem on the model parameters, and test this approach with the k-Nearest Neighbour classifier. Our results on the USPS and other standard datasets show promise in this approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer-Verlag New York Inc., Secaucus (2005)
Balasubramanian, V., Ho, S.S., Vovk, V., eds.: Conformal Prediction for Reliable Machine Learning: Theory, Adaptations and Applications. Morgan Kaufmann, Amsterdam (2014)
Vovk, V.: On-line confidence machines are well-calibrated. In: Proceedings of the The 43rd Annual IEEE Symposium on Foundations of Computer Science, pp. 187–196 (2002)
Balasubramanian, V., Chakraborty, S., Panchanathan, S., Ye, J.: Kernel learning for efficiency maximization in the conformal predictions framework. In: 2010 Ninth International Conference on Machine Learning and Applications (ICMLA), pp. 235–242, December 2010
Vovk, V., Fedorova, V., Nouretdinov, I., Gammerman, A.: Criteria of efficiency for conformal prediction. Technical report, Royal Holloway University of London (April 2014)
Bishop, C.M.: Pattern Recognition and Machine Learning. 1st ed. corr. 2nd printing 2011, edition edn. Springer, New York (February 2010)
Pekala, M., Llorens, A., Wang, I.J.: Local distance metric learning for efficient conformal predictors. In: 2012 IEEE International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6, September 2012
Yang, F., Chen, Z., Shao, G., Wang, H.: Distance Metric Learning-Based Conformal Predictor. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Karatzas, K., Sioutas, S. (eds.) Artificial Intelligence Applications and Innovations, Part II. IFIP AICT, vol. 382, pp. 254–259. Springer, Heidelberg (2012)
Hardoon, D.R., Hussain, Z., Shawe-Taylor, J.: A nonconformity approach to model selection for SVMs. arXiv:0909.2332 [stat] (September 2009) arXiv:0909.2332
Bache, K., Lichman, M.: UCI machine learning repository (2013). http://archive.ics.uci.edu/ml
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Jaiswal, R., Balasubramanian, V.N. (2015). Model Selection Using Efficiency of Conformal Predictors. In: Gammerman, A., Vovk, V., Papadopoulos, H. (eds) Statistical Learning and Data Sciences. SLDS 2015. Lecture Notes in Computer Science(), vol 9047. Springer, Cham. https://doi.org/10.1007/978-3-319-17091-6_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-17091-6_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-17090-9
Online ISBN: 978-3-319-17091-6
eBook Packages: Computer ScienceComputer Science (R0)