Healthcare Text Classification System and its Performance Evaluation: A Source of Better Intelligence by Characterizing Healthcare Text
- 25 Downloads
A machine learning (ML)-based text classification system has several classifiers. The performance evaluation (PE) of the ML system is typically driven by the training data size and the partition protocols used. Such systems lead to low accuracy because the text classification systems lack the ability to model the input text data in terms of noise characteristics. This research study proposes a concept of misrepresentation ratio (MRR) on input healthcare text data and models the PE criteria for validating the hypothesis. Further, such a novel system provides a platform to amalgamate several attributes of the ML system such as: data size, classifier type, partitioning protocol and percentage MRR. Our comprehensive data analysis consisted of five types of text data sets (TwitterA, WebKB4, Disease, Reuters (R8), and SMS); five kinds of classifiers (support vector machine with linear kernel (SVM-L), MLP-based neural network, AdaBoost, stochastic gradient descent and decision tree); and five types of training protocols (K2, K4, K5, K10 and JK). Using the decreasing order of MRR, our ML system demonstrates the mean classification accuracies as: 70.13 ± 0.15%, 87.34 ± 0.06%, 93.73 ± 0.03%, 94.45 ± 0.03% and 97.83 ± 0.01%, respectively, using all the classifiers and protocols. The corresponding AUC is 0.98 for SMS data using Multi-Layer Perceptron (MLP) based neural network. All the classifiers, the best accuracy of 91.84 ± 0.04% is shown to be of MLP-based neural network and this is 6% better over previously published. Further we observed that as MRR decreases, the system robustness increases and validated by standard deviations. The overall text system accuracy using all data types, classifiers, protocols is 89%, thereby showing the entire ML system to be novel, robust and unique. The system is also tested for stability and reliability.
KeywordsHealthcare text classification Machine learning Classifiers Misrepresentation ratio Reliability Stability
Compliance with Ethical Standards
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
- 2.Lee, K., Agrawal, A., and Choudhary, A., Real-time disease surveillance using twitter data: demonstration on flu and cancer. In Proceedings of the 19th ACM SIGKDD international conference on knowledge discovery and data mining, 1474-1477, 2013.Google Scholar
- 4.Li, G. Z., Yang, J., Liu, G. P., and Xue, L., Feature selection for multi-class problems using support vector machines. In PRICAI, 292-300, 2004.Google Scholar
- 9.Japkowicz, N., and Shah, M., Evaluating learning algorithms: a classification perspective. Cambridge University Press. 2011.Google Scholar
- 11.Huang, J., and Ling, C. X., Constructing new and better evaluation measures for machine learning. In IJCAI, 859-864, 2007.Google Scholar
- 12.Wong, A. K., Lee, J. W., and Yeung, D. S., Improving text classifier performance based on AUC. In Pattern Recognition, 2006. ICPR 2006. 18 th , 1-4, 2006. Google Scholar
- 14.Sriram, B., Fuhry, D., Demir, E., Ferhatosmanoglu, H., and Demirbas, M., Short text classification in twitter to improve information filtering. In Proceedings of the33rd international ACM SIGIR conference on Research and development in information retrieval, 841-842, 2010.Google Scholar
- 15.Caragea, C., Wu, J., Gollapalli, S. D., and Giles, C. L., Document Type Classification in Online Digital Libraries. AAAI, 3997-4002, 2016.Google Scholar
- 18.Cormack, G. V., Gómez Hidalgo, J. M., and Sánz, E. P., Spam filtering for short messages. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, 313-320, 2007.Google Scholar
- 20.Lu, C., Zhang, X., Park, J. R., Hu, X., & He, T., Web clustering based on the information of sibling pages. In Granular Computing, 2008. GrC 2008. IEEE International Conference, 480–485, 2008.Google Scholar
- 22.Roesslein, J. (2009). tweepy documentation. Online http://tweepy.readthedocs.io/en/v3, 5.
- 25.Acharya, U. R., Mookiah, M. R. K., Sree, S. V., Afonso, D., Sanches, J., Shafique, S., and Suri, J. S., Atherosclerotic plaque tissue characterization in 2D ultrasound longitudinal carotid scans for automated classification: a paradigm for stroke risk assessment. Med Biol. Eng. Comput. 51(5):513–523, 2013.CrossRefPubMedGoogle Scholar
- 29.Chakravarty, S. (2011). Stochastic Gradient Descent Methods for large scale pattern classification.Google Scholar
- 30.Martineau, J., and Finin, T., Delta TFIDF: an improved feature space for sentiment analysis. Icwsm 9:106, 2009.Google Scholar
- 31.Robert, M. H., & Linda, G. S., Computer and robot vision. Vol. I, Addison-Wesley, 28–48, 1992.Google Scholar
- 32.Suri, J. S., Haralick, R. M., and Sheehan, F.H., Left ventricle longitudinal axis fitting and its apex estimation using a robust algorithm and its performance: a parametric apex model. In Image Processing, 1997. Proceedings., International Conference on (Vol. 3, pp. 118-121). IEEE, 1997.Google Scholar