Abstract
For any corporation the interaction with its customers is an important business process. This is especially the case for resolving various business-related issues that customers encounter. Classifying the type of such customer service e-mails to provide improved customer service is thus important. The classification of e-mails makes it possible to direct them to the most suitable handler within customer service. We have investigated the following two aspects of customer e-mail classification within a large Swedish corporation. First, whether a multi-label classifier can be introduced that performs similarly to an already existing multi-class classifier. Second, whether conformal prediction can be used to quantify the certainty of the predictions without loss in classification performance. Experiments were used to investigate these aspects using several evaluation metrics. The results show that for most evaluation metrics, there is no significant difference between multi-class and multi-label classifiers, except for Hamming loss where the multi-label approach performed with a lower loss. Further, the use of conformal prediction did not introduce any significant difference in classification performance for neither the multi-class nor the multi-label approach. As such, the results indicate that conformal prediction is a useful addition that quantifies the certainty of predictions without negative effects on the classification performance, which in turn allows detection of statistically significant predictions.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Alsmadi, I., Alhami, I.: Clustering and classification of email contents. J. King Saud Univ. - Comput. Inf. Sci. 27(1), 46–57 (2015). https://doi.org/10.1016/j.jksuci.2014.03.014
Balasubramanian, V., Ho, S.S., Vovk, V.: Conformal prediction for reliable machine learning: Theory, Adaptations and applications. Newnes (2014)
Borg, A., Boldt, M.: Clustering residential burglaries using modus operandi and spatiotemporal information. Int. J. Inf. Technol. Decis. Mak. 15(01), 23–42 (2016). https://doi.org/10.1142/S0219622015500339
Borg, A., Lavesson, N.: E-mail classification using social network information. In: 2012 Seventh International Conference on Availability, Reliability and Security, pp. 168–173, August 2012. https://doi.org/10.1109/ARES.2012.84
Cavnar, W.B., Trenkle, J.M.: N-gram-based text categorization. In: Proceedings of the Third Symposium on Document Analysis and Information Retrieval, pp. 161–175 (1994)
Cohen, J.: Stat. Power Anal. Behav. Sci., 2nd edn. Lawrence Earlbaum Associates, Hillsdale (1988)
Coussement, K., den Poel, D.V.: Improving customer complaint management by automatic email classification using linguistic style features as predictors. Decis. Support Syst. 44(4), 870–882 (2008). https://doi.org/10.1016/j.dss.2007.10.010
Dredze, M., Brooks, T., Carroll, J., Magarick, J., Blitzer, J., Pereira, F.: Intelligent email: reply and attachment prediction. In: Proceedings of the 13th International Conference on Intelligent User Interfaces, IUI 2008, pp. 321–324. ACM, New York (2008). https://doi.org/10.1145/1378773.1378820
Fawcett, T.: ROC graphs: notes and practical considerations for researchers. Mach. Learn. 31(1), 1–38 (2004)
Flach, P.: Machine Learning: The Art and Science of Algorithms that Make Sense of Data. Cambridge University Press, Cambridge (2012)
Galar, M., Fernández, A., Barrenechea, E., Bustince, H., Herrera, F.: An overview of ensemble methods for binary classifiers in multi-class problems: experimental study on one-vs-one and one-vs-all schemes. Pattern Recogn. 44(8), 1761–1776 (2011)
Gupta, N., Gilbert, M., Fabbrizio, G.D.: Emotion detection in email customer care. Comput. Intell. 29(3), 489–505 (2013). https://doi.org/10.1111/j.1467-8640.2012.00454.x
Ha, Q.M., Tran, Q.A., Luyen, T.T.: Personalized email recommender system based on user actions. In: Bui, L.T., Ong, Y.S., Hoai, N.X., Ishibuchi, H., Suganthan, P.N. (eds.) SEAL 2012. LNCS, vol. 7673, pp. 280–289. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34859-4_28
Halpin, N.: The customer service report: why great customer service matters even more in the age of e-commerce and the channels that perform best (2016). http://www.businessinsider.com/customer-service-experiences-are-more-important-than-ever-in-the-age-of-e-commerce-2016-3?r=US&IR=T&IR=T
Wang, H., Liu, X., Lv, B., Yang, F., Hong, Y.: Reliable multi-label learning via conformal predictor and random forest for syndrome differentiation of chronic fatigue in traditional Chinese medicine. PLOS One 9(6), 1–14 (2014). https://doi.org/10.1371/journal.pone.0099565
Witten, I.H., Frank, E., Hall, M.: Data Mining - Practical Machine Learning Tools and Techniques, 3rd edn. Elsevier, Amsterdam (2011)
Koren, Y., Liberty, E., Maarek, Y., Sandler, R.: Automatically tagging email by leveraging other users’ folders. In: In Proceedings of KDD 2011, pp. 913–921. ACM (2011)
Nenkova, A., Bagga, A.: Email classification for contact centers. In: Proceedings of the 2003 ACM Symposium on Applied Computing, SAC 2003, pp. 789–792. ACM, New York (2003). https://doi.org/10.1145/952532.952689
Papadopoulos, H.: Inductive conformal prediction: theory and application to neural networks. In: Tools in Artificial Intelligence. InTech (2008)
Papadopoulos, H.: A cross-conformal predictor for multi-label classification. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) AIAI 2014. IAICT, vol. 437, pp. 241–250. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-44722-2_26
Sebastiani, F.: Machine learning in automated text categorization. ACM Comput. Surv. (CSUR) (2002). http://portal.acm.org/citation.cfm?id=505283
Shafer, G., Vovk, V.: A tutorial on conformal prediction. J. Mach. Learn. Res. 9(Mar), 371–421 (2008)
Tsoumakas, G., Katakis, I., Vlahavas, I.: Mining multi-label data. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 667–685. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-09823-4_34
Tsoumakas, G., Zhang, M.L., Zhou, Z.H.: Introduction to the special issue on learning from multi-label data. Mach. Learn. 88(1), 1–4 (2012). https://doi.org/10.1007/s10994-012-5292-9
Wang, M.F., Jheng, S.L., Tsai, M.F., Tang, C.H.: Enterprise email classification based on social network features. In: 2011 International Conference on Advances in Social Networks Analysis and Mining, pp. 532–536, July 2011. https://doi.org/10.1109/ASONAM.2011.89
Yang, Y.: An evaluation of statistical approaches to text categorization. Inf. Retrieval 1(1), 69–90 (1999). https://doi.org/10.1023/A:1009982220290
Yelupula, K., Ramaswamy, S.: Social network analysis for email classification. In: Proceedings of the 46th Annual Southeast Regional Conference on XX, ACM-SE 46, pp. 469–474. ACM, New York (2008). https://doi.org/10.1145/1593105.1593229
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Borg, A., Boldt, M., Svensson, J. (2019). Using Conformal Prediction for Multi-label Document Classification in e-Mail Support Systems. In: Wotawa, F., Friedrich, G., Pill, I., Koitz-Hristov, R., Ali, M. (eds) Advances and Trends in Artificial Intelligence. From Theory to Practice. IEA/AIE 2019. Lecture Notes in Computer Science(), vol 11606. Springer, Cham. https://doi.org/10.1007/978-3-030-22999-3_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-22999-3_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-22998-6
Online ISBN: 978-3-030-22999-3
eBook Packages: Computer ScienceComputer Science (R0)