Aggregating Subjective and Objective Measures of Web Search Quality using Modified Shimura Technique

Abstract

Web Searching is perhaps the second most popular activity on the Internet. But, as there are a number of search engines available, there must be some procedure to evaluate them. So, in this chapter, we propose an evaluation system for web search results. We are taking into consideration the “satisfaction” a user gets when presented with search results. The subjective evaluation based on the user feedback is augmented with the objective evaluation. The feedback of the user is inferred from watching the actions of the user on the search results presented before him in response to his query, rather than by a form filling method. This gives an implicit ranking of documents by the user. Then, the classical vector space model is used for computing the similarity of the documents selected by the user to that of the query. Also, the Boolean similarity measure is used to compute the similarity of the documents selected by the user to that of the query and thus another ranking of the documents based on this similarity measure is obtained. All the three rankings obtained in the process are then aggregated using the Modified Shimura Technique of Rank aggregation. The aggregated ranking is then compared with the original ranking given by the search engine. The correlation coefficient thus obtained is averaged for a set of queries. The averaged correlation coefficient is thus a measure of web search quality of the search engine. We show our experimental results pertaining to seven public search engines and fifteen queries.