Abstract
Web searching is such an activity that its importance can just not be ignored in the current scenario. Since there are a large number of publicly accessible search engines, shich differ in their indexing algorithms and hence the search results, the evaluation of search engines performance is needed to determine which one is the best. The human intelligence may be used to measure the search engine effectiveness. But, a subjective evaluation done on the basis of user-feedback is costly in terms of the time required. Therefore, it is also not scalable. So, there is a need of an automatic evaluation method. In this paper, we present the architecture of an automatic Web search evaluation system that combines the different evaluation techniques using a Rough Set based Rank aggregation technique. The rough set based rank aggregation models the user’s feedback based rank aggregation. In the rough set based aggregation technique, the ranking rules are learnt on the basis of the user feedback in the training data sets. The learned rules are then used to estimate the overall ranking for the other data sets, for which user feedback is not available. We show our experimental results pertaining to seven public search engines.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
References
Salton, G., Wong, A., and Yang C. S.: A vector space model for automatic indexing. Communications of the ACM 18, 613–620 (1975)
Li, S. H. and Danzig, P. B. Boolean similarity measures for resource discovery. IEEE Transactions on Knowledge and Data Engineering 9, 863–876 (1997)
Ali, R. and Beg, M. M. S.: Rough set based rank aggregation for the Web. In Proceedings of 3rd Indian International Conference on Artificial Intelligence (IICAI-07), Pune, India pp. 683–698 (2007)
Soboroff, I., Nicholas, C., and Cahan, P.: Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, LA, U.S.A. pp. 66–73 (2001)
Chowdhury, A. and Soboroff, I.: Automatic evaluation of World Wide Web search services. In Proceedings of the 25th annual international ACM SIGIR conference on research and development in information retrieval, Tampere, Finland, ACM Press pp. 421–422 (2002)
Shang, Y. and Li, L.: Precision evaluation of search engines. World Wide Web 5, 159–173 (2002)
Wu, S. and Crestani, F.: Methods for ranking information retrieval systems without relevance judgments. In Proceedings of the ACM Symposium on Applied Computing, Melbourne, Florida, U.S.A.) pp. 811–816 (2003)
Can, F., Nuray, R., and Sevdik, A. B.: Automatic performance evaluation of Web search engines. Information Processing and Management 40, 495–514 (2004)
Beitzel, S. M., Jensen, E. C., Chowdhury, A., and Grossman, D.: Using titles and category names from editor-driven taxonomies for automatic evaluation. In Proceedings of the 12th International Conference on Information and Knowledge Management, New Orleans, LA, U.S.A. pp. 17–23 (2003)
Sharma, H. and Jansen, B. J.: Automated evaluation of search engine performance via implicit user feedback. In Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval, Salvador, Brazil pp. 649–650 (2005)
Aslam, J. A., Pavlu, V., and Yilmaz, E.: A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA, U.S.A. pp. 541–548 (2006)
Nuray R. and Can, F.: Automatic ranking of information retrieval systems using data fusion. Information Processing and Management 42, 595–614 (2006)
Open Directory Project. http://dmoz.org/
Weisstein, E. W.: Spearman Rank Correlation Coefficient. From MathWorld — A Wolfram Web Resource, ©1999–2004 Wolfram Research, Inc. http://mathworld.wolfram.com/SpearmanRankCorrelationCoefficient.html
Pawlak, Z.: Rough sets. International Journal of Computer and Information Sciences 11, 341–356 (1982)
Komorowski, J., Pawlak, Z., Polkowski, L., Skowron, A.: Rough sets: A tutorial In Rought Fuzzy Hybridization: A New Trend in Decision-Making, S.K. Pal, A. Skowron (Eds.), Springer-Verlag, Singapore pp. 3–98 (1999)
Yao, Y.Y., Sai, Y.: Mining ordering rules using rough set theory. [J] Bulletin of International Rough Set Society pp. 599–106 (2001)
Rosetta, a rough set toolkit for analyzing data.
Beg, M. M. S.: A subjective measure of Web search quality. International Journal of Information Sciences. 169, 365–381 (2005)
Page, L., Brin, S., Motwani, R., and Winograd, T.: The PageRank citation ranking: Bringing order to the Web. Technical report, Computer Science Department, Stanford University, U.S.A. (1999)
Beg, M. M. S. and Ahmad, N.: Web search enhancement by mining user actions. International Journal of Information Sciences. 177, 5203–5218 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Indian Institute of Information Technology, India
About this paper
Cite this paper
Ali, R., Beg, M.M.S. (2009). Automatic Performance Evaluation of Web Search Systems using Rough Set based Rank Aggregation. In: Tiwary, U.S., Siddiqui, T.J., Radhakrishna, M., Tiwari, M.D. (eds) Proceedings of the First International Conference on Intelligent Human Computer Interaction. Springer, New Delhi. https://doi.org/10.1007/978-81-8489-203-1_34
Download citation
DOI: https://doi.org/10.1007/978-81-8489-203-1_34
Publisher Name: Springer, New Delhi
Print ISBN: 978-81-8489-404-2
Online ISBN: 978-81-8489-203-1
eBook Packages: Computer ScienceComputer Science (R0)