Automatic Performance Evaluation of Web Search Systems using Rough Set based Rank Aggregation

  • Rashid Ali
  • M. M. Sufyan Beg


Web searching is such an activity that its importance can just not be ignored in the current scenario. Since there are a large number of publicly accessible search engines, shich differ in their indexing algorithms and hence the search results, the evaluation of search engines performance is needed to determine which one is the best. The human intelligence may be used to measure the search engine effectiveness. But, a subjective evaluation done on the basis of user-feedback is costly in terms of the time required. Therefore, it is also not scalable. So, there is a need of an automatic evaluation method. In this paper, we present the architecture of an automatic Web search evaluation system that combines the different evaluation techniques using a Rough Set based Rank aggregation technique. The rough set based rank aggregation models the user’s feedback based rank aggregation. In the rough set based aggregation technique, the ranking rules are learnt on the basis of the user feedback in the training data sets. The learned rules are then used to estimate the overall ranking for the other data sets, for which user feedback is not available. We show our experimental results pertaining to seven public search engines.


Search Engine User Feedback Vector Space Model Relevance Judgment Rank Aggregation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Salton, G., Wong, A., and Yang C. S.: A vector space model for automatic indexing. Communications of the ACM 18, 613–620 (1975)MATHCrossRefGoogle Scholar
  2. 2.
    Li, S. H. and Danzig, P. B. Boolean similarity measures for resource discovery. IEEE Transactions on Knowledge and Data Engineering 9, 863–876 (1997)CrossRefGoogle Scholar
  3. 3.
    Ali, R. and Beg, M. M. S.: Rough set based rank aggregation for the Web. In Proceedings of 3rd Indian International Conference on Artificial Intelligence (IICAI-07), Pune, India pp. 683–698 (2007)Google Scholar
  4. 4.
    Soboroff, I., Nicholas, C., and Cahan, P.: Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, U.S.A. pp. 66–73 (2001)Google Scholar
  5. 5.
    Chowdhury, A. and Soboroff, I.: Automatic evaluation of World Wide Web search services. In Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval, Tampere, Finland, ACM Press pp. 421–422 (2002)CrossRefGoogle Scholar
  6. 6.
    Shang, Y. and Li, L.: Precision evaluation of search engines. World Wide Web 5, 159–173 (2002)CrossRefGoogle Scholar
  7. 7.
    Wu, S. and Crestani, F.: Methods for ranking information retrieval systems without relevance judgments. In Proceedings of the ACM Symposium on Applied Computing, Melbourne, Florida, U.S.A.) pp. 811–816 (2003)Google Scholar
  8. 8.
    Can, F., Nuray, R., and Sevdik, A. B.: Automatic performance evaluation of Web search engines. Information Processing and Management 40, 495–514 (2004)CrossRefGoogle Scholar
  9. 9.
    Beitzel, S. M., Jensen, E. C., Chowdhury, A., and Grossman, D.: Using titles and category names from editor-driven taxonomies for automatic evaluation. In Proceedings of the 12th International Conference on Information and Knowledge Management, New Orleans, LA, U.S.A. pp. 17–23 (2003)Google Scholar
  10. 10.
    Sharma, H. and Jansen, B. J.: Automated evaluation of search engine performance via implicit user feedback. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval, Salvador, Brazil pp. 649–650 (2005)Google Scholar
  11. 11.
    Aslam, J. A., Pavlu, V., and Yilmaz, E.: A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, U.S.A. pp. 541–548 (2006)Google Scholar
  12. 12.
    Nuray R. and Can, F.: Automatic ranking of information retrieval systems using data fusion. Information Processing and Management 42, 595–614 (2006)MATHCrossRefGoogle Scholar
  13. 13.
    Open Directory Project. Scholar
  14. 14.
    Weisstein, E. W.: Spearman Rank Correlation Coefficient. From MathWorld — A Wolfram Web Resource, ©1999–2004 Wolfram Research, Inc. Scholar
  15. 15.
    Pawlak, Z.: Rough sets. International Journal of Computer and Information Sciences 11, 341–356 (1982)MATHCrossRefMathSciNetGoogle Scholar
  16. 16.
    Komorowski, J., Pawlak, Z., Polkowski, L., Skowron, A.: Rough sets: A tutorial In Rought Fuzzy Hybridization: A New Trend in Decision-Making, S.K. Pal, A. Skowron (Eds.), Springer-Verlag, Singapore pp. 3–98 (1999)Google Scholar
  17. 17.
    Yao, Y.Y., Sai, Y.: Mining ordering rules using rough set theory. [J] Bulletin of International Rough Set Society pp. 599–106 (2001)Google Scholar
  18. 18.
    Rosetta, a rough set toolkit for analyzing data.Google Scholar
  19. 19.
    Beg, M. M. S.: A subjective measure of Web search quality. International Journal of Information Sciences. 169, 365–381 (2005)Google Scholar
  20. 20.
    Page, L., Brin, S., Motwani, R., and Winograd, T.: The PageRank citation ranking: Bringing order to the Web. Technical report, Computer Science Department, Stanford University, U.S.A. (1999)Google Scholar
  21. 21.
    Beg, M. M. S. and Ahmad, N.: Web search enhancement by mining user actions. International Journal of Information Sciences. 177, 5203–5218 (2007)Google Scholar

Copyright information

© Indian Institute of Information Technology, India 2009

Authors and Affiliations

  • Rashid Ali
    • 1
  • M. M. Sufyan Beg
    • 2
  1. 1.Department of Computer EngineeringA.M. U.AligarhIndia
  2. 2.Department of Computer EngineeringJ.M.I.New DelhiIndia

Personalised recommendations