An Empirical Study to Classify Website Using Thresholds from Data Characteristics

  • Ruchika MalhotraEmail author
  • Anjali Sharma
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 904)


The advent of web had resulted in a plethora of information and data. However, its volume heterogeneity and unstructured organization makes information retrieval difficult. To the existing practice where website categorization is largely based on style rather than text, addition of an extra dimension in form of genre is expected to significantly improve the search outcome. Keeping this in view, we attempt to build a novel classification model to categorize websites into genres using thresholds of the web metrics. Statistical measures of central tendency are assumed to render a value that distinguish websites from a sample space containing News, Travel and Tourism, Entertainment and Social media. Through the statistical analysis of the data we find that the data distribution of all metrics which constitute the website properties are highly skewed. Hence, conventional analysis based on normal distribution statistics fails to apply. Adopting to a systematic empirical approach, we find that the classification performance measure identified through the Area Under the Curve is maximized around a threshold value which is twice the value of the “median-absolute-deviation” of the web metrics.


Web genre HTML metrics Threshold Median-Absolute-Deviation Naive Bayes 


  1. 1.
    Chetry, R.: Web genre classification using feature selection and semi-supervised learning (2011)Google Scholar
  2. 2.
    Gatto, M.: Web as Corpus: Theory and Practice. Bloomsbury Academic, London (2014)Google Scholar
  3. 3.
    Stein, B., Zu Eissen, S.M., Lipka, N.: Web genre analysis: use cases, retrieval models, and implementation issues. In: Genres on the Web, pp. 167–189. Springer, Dordrecht (2010)Google Scholar
  4. 4.
    Ponzanelli, L., Mocci, A., Lanza, M.: Summarizing complex development artifacts by mining heterogeneous data. In: Proceedings of the 12th Working Conference on Mining Software Repositories, pp. 401–405. IEEE Press, New York (2015)Google Scholar
  5. 5.
    Wu, L., Du, L., Liu, B., Xu, G., Ge, Y., Fu, Y., Li, J., Zhou, Y., Xiong, H.: Heterogeneous metric learning with content-based regularization for software artifact retrieval. In: IEEE International Conference on Data Mining (ICDM), pp. 610–619. IEEE, New York (2014)Google Scholar
  6. 6.
    Shepherd, M., Watters, C.: The functionality attribute of cybergenres. In: Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences, HICSS-32, p. 9. IEEE, New York (1999)Google Scholar
  7. 7.
    Shepherd, M., Watters, C.: Identifying web genre: hitting a moving target. In: Proceedings of the WWW Conference. Workshop on Measuring Web Search Effectiveness: The User Perspective, vol. 18, New York (2004)Google Scholar
  8. 8.
    Rosso, M.A.: Using genre to improve web search. Doctoral dissertation, University of North Carolina, Chapel Hill (2005)Google Scholar
  9. 9.
    Williams, K.C.M.: Reproduced and emergent genres of communication on the World Wide Web. Inf. Soc. 16, 201–215 (2000)CrossRefGoogle Scholar
  10. 10.
    Santini, M.: Characterizing genres of web pages: genre hybridism and individualization. In: HICSS 40th Annual Hawaii International Conference on System Sciences, pp. 71–71. IEEE, New York (2007)Google Scholar
  11. 11.
    Roussinov, D., Crowston, K., Nilan, M., Kwasnik, B., Cai, J., Liu, X.: Genre based navigation on the web. In: Proceedings of the 34th Annual Hawaii International Conference on System Sciences, p. 10. IEEE, New York (2001)Google Scholar
  12. 12.
    Crowston, K., Kwasnik, B.H.: A framework for creating a facetted classification for genres: addressing issues of multidimensionality. In: Proceedings of the 37th Annual Hawaii International Conference on System Sciences, p. 9. IEEE, New York (2004)Google Scholar
  13. 13.
    Copestake, A.: Errors in wikis. In: Proceedings of the Workshop on NEW TEXT Wikis and Blogs and Other Dynamic Text Sources (2006)Google Scholar
  14. 14.
    Mehler, A.: Text linkage in the wiki medium: a comparative study. In: Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy, April 3–7, 2006 (EACL 2006): Workshop on New Text—Wikis and blogs and other dynamic text sources, pp. 1–8 (2006)Google Scholar
  15. 15.
    Lindemann, C., Littig, L.: Classification of web sites at super-genre level. In: Genres on the Web, pp. 211–235. Springer Netherlands (2010)Google Scholar
  16. 16.
    Erni, K., Lewerentz, C.: Applying design-metrics to object-oriented frameworks. In: Proceedings of the 3rd International on Software Metrics Symposium, pp. 64–74. IEEE, New York (1996)Google Scholar
  17. 17.
    French, V.: Establishing software metric thresholds. In: Proceedings of the 9th International Workshop on Software Measurement (1999)Google Scholar
  18. 18.
    de Siqueira, G.O., de Assis, G.T., Almeida Ferreira, A., Mangaravite, V., Cardeal P´adua, F.L.: Strategies for automatic determination of similarity threshold for genre-aware focused crawling processes. In: IADIS International Journal on WWW/Internet, vol. 15 (2017)Google Scholar
  19. 19.
    Shatnawi, R., Li, W., Swain, J., Newman, T.: Finding software metrics threshold values using ROC curves. J. Softw. Maint. Evol.: Res. Pract. 22, 1–16 (2010)CrossRefGoogle Scholar
  20. 20.
    Shatnawi, R.: A quantitative investigation of the acceptable risk levels of OO metrics in open-source systems. IEEE Trans. Softw. Eng. 36, 216–225 (2010)CrossRefGoogle Scholar
  21. 21.
    Bender, R.: Quantitative risk assessment in epidemiological studies investigating threshold effects. Biom. J.: J. Math. Methods Biosci. 41, 305–319 (1999)CrossRefGoogle Scholar
  22. 22.
    Malhotra, R., Bansal, A.J.: Fault prediction considering threshold effects of object-oriented metrics. Expert. Syst. 32, 203–219 (2015)CrossRefGoogle Scholar
  23. 23.
    Shatnawi, R.: Deriving metrics thresholds using log transformation. J. Softw.: Evol. Process. 27, 95–113 (2015)Google Scholar
  24. 24.
    Alves, T.L., Ypma, C., Visser, J.: Deriving metric thresholds from benchmark data. In: IEEE International Conference on Software Maintenance (ICSM), pp. 1–10. (2010)Google Scholar
  25. 25.
    Ferreira, K.A., Bigonha, M.A., Bigonha, R.S., Mendes, L.F., Almeida, H.C.: Identifying thresholds for object-oriented software metrics. J. Syst. Softw. 85, 244–257 (2012)CrossRefGoogle Scholar
  26. 26.
    Hussain, S., Keung, J., Khan, A.A., Bennin, K.E.: Detection of fault-prone classes using logistic regression based object-oriented metrics thresholds. In: IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 93–100 (2016)Google Scholar
  27. 27.
    Malhotra, R., Sharma, A.: A web metric collection and reporting system. In: Proceedings of the Third International Symposium on Women in Computing and Informatics, pp. 661–667. ACM, New York (2015)Google Scholar
  28. 28.
    Malhotra, R., Sharma, A.: Quantitative evaluation of web metrics for automatic genre classification of web pages. Int. J. Syst. Assur. Eng. Manag. 8, 1567–1579 (2017)CrossRefGoogle Scholar
  29. 29.
    Frman, G., Cohen, I.: Learning from little: comparison of classifiers given little training. In: European Conference on Principles of Data Mining and Knowledge Discovery, pp. 161–172. Springer, Berlin (2004)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Delhi Technological UniversityNew DelhiIndia
  2. 2.CSIR-NPLNew DelhiIndia

Personalised recommendations