A Classification Approach with a Reject Option for Multi-label Problems

  • Ignazio Pillai
  • Giorgio Fumera
  • Fabio Roli
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6978)


We investigate the implementation of multi-label classification algorithms with a reject option, as a mean to reduce the time required to human annotators and to attain a higher classification accuracy on automatically classified samples than the one which can be obtained without a reject option. Based on a recently proposed model of manual annotation time, we identify two approaches to implement a reject option, related to the two main manual annotation methods: browsing and tagging. In this paper we focus on the approach suitable to tagging, which consists in withholding either all or none of the category assignments of a given sample. We develop classification reliability measures to decide whether rejecting or not a sample, aimed at maximising classification accuracy on non-rejected ones. We finally evaluate the trade-off between classification accuracy and rejection rate that can be attained by our method, on three benchmark data sets related to text categorisation and image annotation tasks.


Multi-label classification Reject option 


  1. 1.
    Aronson, A., Rogers, W., Lang, F., Névéol, A.: 2008 report to the board of scientific counselors (2008),
  2. 2.
    Boutell, M.R., Luo, J., Shen, X., Brown, C.M.: Learning multi-label scene classification. Pattern Recognition 37(9), 1757–1771 (2004)CrossRefGoogle Scholar
  3. 3.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001), software available at
  4. 4.
    Chow, C.K.: On optimum recognition error and reject tradeoff. IEEE Transactions in Information Theory 16(1), 41–16 (1970)Google Scholar
  5. 5.
    Fan, R.E., Lin, C.J.: A study on threshold selection for multi-label. Tech. rep., National Taiwan University (2007)Google Scholar
  6. 6.
    Fumera, G., Pillai, I., Roli, F.: Classification with reject option in text categorisation systems. In: Int. Conf. Image Analysis and Proc. (2003)Google Scholar
  7. 7.
    Fumera, G., Pillai, I., Roli, F.: A Two-Stage Classifier with Reject Option for Text Categorisation. In: Structural, Syntactic, and Statistical Patt. Rec. (2004)Google Scholar
  8. 8.
    Lewis, D.D., Schapire, R.E., Callan, J.P., Papka, R.: Training algorithms for linear text classifiers. In: SIGIR, pp. 298–306 (1996)Google Scholar
  9. 9.
    Nowak, S., Huiskes, M.: New strategies for image annotation: Overview of the photo annotation task at imageclef 2010. Working Notes of CLEF 2010 (2010)Google Scholar
  10. 10.
    Pudil, P., Novovicova, J., Blaha, S., Kittler, J.V.: Multistage pattern recognition with reject option. In: ICPR, pp. II:92–II:95 (1992)Google Scholar
  11. 11.
    Ruiz, M., Aronson, A.: User-centered evaluation of the medical text indexing (mti) system (2007),
  12. 12.
    Sebastiani, F.: Machine learning in automated text categorization. ACM Computing Surveys 34(1), 1–47 (2002)CrossRefGoogle Scholar
  13. 13.
    Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Data Mining and Knowledge Discovery Handbook, pp. 667–685 (2010)Google Scholar
  14. 14.
    Yan, R., Natsev, A., Campbell, M.: An efficient manual image annotation approach based on tagging and browsing. In: Workshop on Multimedia Inf. Retr. on The Many Faces of Multimedia Semantics, pp. 13–20 (2007)Google Scholar
  15. 15.
    Yang, Y.: A study of thresholding strategies for text categorization. In: Int. Conf. on Research and Development in Information Retrieval, New York, USA (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Ignazio Pillai
    • 1
  • Giorgio Fumera
    • 1
  • Fabio Roli
    • 1
  1. 1.Deparment of Electrical and Electronic EngineeringUniv. of CagliariCagliariItaly

Personalised recommendations