Advertisement

User Preference Prediction in Mobile Search

  • Mengyang Liu
  • Cheng Luo
  • Yiqun LiuEmail author
  • Min Zhang
  • Shaoping Ma
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10390)

Abstract

As search requests from mobile devices are growing very quickly, mobile search evaluation becomes one of the central concerns in mobile search studies. Beyond traditional Cranfield paradigm, side-by-side user preference between two ranked lists does not rely on user behavior assumptions and has been shown to produce more accurate results comparing to traditional evaluation methods based on “query-document” relevance. On the other hand, result list preference judgements have very high annotation cost. Previous studies attempted to assist human judges by automatically predicting preference. However, whether these models are effective in mobile search environment is still under investigation. In this paper, we proposed a machine learning model to predict user preference automatically in mobile search environment. We find that the relevance features can predict user preference very well, so we compare the agreement of evaluation metrics with side-by-side user preferences on our dataset. We get inspiration from the agreement comparison method and proposed new relevance features to build models. Experimental results show that our proposed model can predict user preference very effectively.

Keywords

Mobile search Search evaluation User preference prediction 

References

  1. 1.
    Cleverdon, C.W., Keen, M.: Aslib Cranfield research project-factors determining the performance of indexing systems (1966)Google Scholar
  2. 2.
    Moffat, A., Thomas, P., Scholer, F.: Users versus models: what observation tells us about effectiveness metrics. In: Proceedings of the 22nd ACM international Conference on Information & Knowledge Management. ACM (2013)Google Scholar
  3. 3.
    Sanderson, M., et al.: Do user preferences and evaluation measures line up? In: Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2010)Google Scholar
  4. 4.
    Hassan Awadallah, A., Zitouni, I.: Machine-assisted search preference evaluation. In: Proceedings of the 23rd ACM International Conference on Information and Knowledge Management. ACM (2014)Google Scholar
  5. 5.
    Guo, Q., et al.: Mining touch interaction data on mobile devices to predict web search result relevance. In: Proceedings of the 36th international ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2013)Google Scholar
  6. 6.
    Jones, M., et al.: Improving web interaction on small displays. Comput. Networks 31(11), 1129–1137 (1999)CrossRefGoogle Scholar
  7. 7.
    Jrvelin, K., Keklinen, J.: Cumulated gain-based evaluation of IR techniques. ACM Trans. Inform. Syst. (TOIS) 20(4), 422–446 (2002)CrossRefGoogle Scholar
  8. 8.
    Chapelle, O., et al.: Expected reciprocal rank for graded relevance. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management. ACM (2009)Google Scholar
  9. 9.
    Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inform. Syst. (TOIS) 27(1), 2 (2008)Google Scholar
  10. 10.
    Smucker, M.D., Clarke, C.L.A.: Time-based calibration of effectiveness measures. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2012)Google Scholar
  11. 11.
    Al-Maskari, A., Sanderson, M., Clough, P.: The relationship between IR effectiveness measures and user satisfaction. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2007)Google Scholar
  12. 12.
    Joachims, T.: Evaluating retrieval performance using clickthrough data, pp. 79–96 (2003)Google Scholar
  13. 13.
    Thomas, P., Hawking, D.: Evaluation by comparing result sets in context. In: Proceedings of the 15th ACM International Conference on Information and Knowledge Management. ACM (2006)Google Scholar
  14. 14.
    Voorhees, E.M., Harman, D.K.: Experiment and evaluation in information retrieval (2005)Google Scholar
  15. 15.
    Zhou, K., et al.: Evaluating aggregated search pages. In: Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2012)Google Scholar
  16. 16.
    Hersh, W., et al.: Do batch and user evaluations give the same results? In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2000)Google Scholar
  17. 17.
    Luo, C., et al.: Evaluating mobile search with height-biased gain. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM (2017)Google Scholar
  18. 18.
    Song, Y., et al.: Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance. In: Proceedings of the 22nd International Conference on World Wide Web. ACM (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Mengyang Liu
    • 1
  • Cheng Luo
    • 1
  • Yiqun Liu
    • 1
    Email author
  • Min Zhang
    • 1
  • Shaoping Ma
    • 1
  1. 1.State Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and TechnologyTsinghua UniversityBeijingChina

Personalised recommendations