Employ Decision Values for Soft-Classifier Evaluation with Crispy References

  • Lei ZhuEmail author
  • Tao Ban
  • Takeshi Takahashi
  • Daisuke Inoue
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11304)


Evaluation of classification performance has been comprehensively studied for both crispy and fuzzy classification tasks. In this paper, we address the hybrid case: evaluating fuzzy prediction results against crispy references. The proposal is motivated by the following facts: (1) most datasets in practice are produced with crispy labels due to the excessive cost of fuzzy labelling; and (2) many state-of-the-art classifiers can yield fuzzy decision values even if they are trained from data with crispy labels. We derive our fuzzy-crispy evaluation criterion based on a widely adopted fuzzy-set-based evaluation method. By exploiting the distribution of decision values, the proposed criterion bears more comprehensive information than conventional crispy classification evaluation criteria. The advantages of the proposed criterion are demonstrated in artificial and real-world classification case studies.


Classification evaluation Decision value Crispy reference 


  1. 1.
    Binaghi, E., Brivio, P.A., Ghezzi, P., Rampini, A.: A fuzzy set-based accuracy assessment of soft classification. Pattern Recogn. Lett. 20(9), 935–948 (1999)CrossRefGoogle Scholar
  2. 2.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 27:1–27:27 (2011). Scholar
  3. 3.
    Dou, W., Ren, Y., Wu, Q., Ruan, S., Chen, Y., Bloyet, D., Constans, J.M.: Fuzzy kappa for the agreement measure of fuzzy classifications. Neurocomputing 70(4–6), 726–734 (2007)CrossRefGoogle Scholar
  4. 4.
    Draper-Gil, G., Lashkari, A.H., Mamun, M.S.I., Ghorbani, A.A.: Characterization of encrypted and VPN traffic using time-related features. In: Proceedings of the 2nd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 407–414. INSTICC, SciTePress (2016).
  5. 5.
    Hagen, A.: Fuzzy set approach to assessing similarity of categorical maps. Int. J. Geogr. Inf. Sci. 17(3), 235–249 (2003)CrossRefGoogle Scholar
  6. 6.
    Jager, G., Benz, U.: Measures of classification accuracy based on fuzzy similarity. IEEE Trans. Geosci. Remote. Sens. 38(3), 1462–1467 (2000)CrossRefGoogle Scholar
  7. 7.
    Keller, J.M., Gray, M.R., Givens, J.A.: A fuzzy k-nearest neighbor algorithm. IEEE Trans. Syst. Man Cybern. 4, 580–585 (1985)CrossRefGoogle Scholar
  8. 8.
    Klir, G.J., Folger, T.A.: Fuzzy Sets, Uncertainty, and Information. Prentice-Hall, Inc., Upper Saddle River (1987)zbMATHGoogle Scholar
  9. 9.
    Lashkari, A.H., Gil, G.D., Mamun, M.S.I., Ghorbani, A.A.: Characterization of tor traffic using time based features. In: Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 253–262. INSTICC, SciTePress (2017).
  10. 10.
    Liu, Y., Zhang, H.H., Wu, Y.: Hard or soft classification large-margin unified machines. J. Am. Stat. Assoc. 106(493), 166–177 (2011)MathSciNetCrossRefGoogle Scholar
  11. 11.
    Sokolova, M., Lapalme, G.: A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 45(4), 427–437 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Lei Zhu
    • 1
    Email author
  • Tao Ban
    • 1
  • Takeshi Takahashi
    • 1
  • Daisuke Inoue
    • 1
  1. 1.National Institute of Information and Communications TechnologyTokyoJapan

Personalised recommendations