Employ Decision Values for Soft-Classifier Evaluation with Crispy References
Evaluation of classification performance has been comprehensively studied for both crispy and fuzzy classification tasks. In this paper, we address the hybrid case: evaluating fuzzy prediction results against crispy references. The proposal is motivated by the following facts: (1) most datasets in practice are produced with crispy labels due to the excessive cost of fuzzy labelling; and (2) many state-of-the-art classifiers can yield fuzzy decision values even if they are trained from data with crispy labels. We derive our fuzzy-crispy evaluation criterion based on a widely adopted fuzzy-set-based evaluation method. By exploiting the distribution of decision values, the proposed criterion bears more comprehensive information than conventional crispy classification evaluation criteria. The advantages of the proposed criterion are demonstrated in artificial and real-world classification case studies.
KeywordsClassification evaluation Decision value Crispy reference
- 4.Draper-Gil, G., Lashkari, A.H., Mamun, M.S.I., Ghorbani, A.A.: Characterization of encrypted and VPN traffic using time-related features. In: Proceedings of the 2nd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 407–414. INSTICC, SciTePress (2016). https://doi.org/10.5220/0005740704070414
- 9.Lashkari, A.H., Gil, G.D., Mamun, M.S.I., Ghorbani, A.A.: Characterization of tor traffic using time based features. In: Proceedings of the 3rd International Conference on Information Systems Security and Privacy - Volume 1: ICISSP, pp. 253–262. INSTICC, SciTePress (2017). https://doi.org/10.5220/0006105602530262