User-Centric vs. System-Centric Evaluation of Recommender Systems

  • Paolo Cremonesi
  • Franca Garzotto
  • Roberto Turrin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8119)


Recommender Systems (RSs) aim at helping users search large amounts of contents and identify more effectively the items (products or services) that are likely to be more useful or attractive. The quality of a RS can be defined from two perspectives: system-centric, in which quality measures (e.g., precision, recall) are evaluated using vast datasets of preferences and opinions on items previously collected from users that are not interacting with the RS under study; user-centric, in which user measures are collected from users interacting with the RS under study. Prior research in e-commerce has provided some empirical evidence that system-centric and user-centric quality methods may lead to inconsistent results, e.g., RSs that were “best” according to system-centric measures were not the top ones according to user-centric measures. The paper investigates if a similar mismatch also exists in the domain of e-tourism. We discuss two studies that have adopted a system-centric approach using data from 210000 users, and a user-centric approach involving 240 users interacting with an online hotel booking service. In both studies, we considered four RSs that employ an implicit user preference elicitation technique and different baseline and state-of-the-art recommendation algorithms. In these four experimental conditions, we compared system-centric quality measures against user-centric evaluation results. System-centric quality measures were consistent with user-centric measures, in contrast with past studies in e-commerce. This pinpoints that the relationship between the two kinds of metrics may depend on the business sector, is more complex that we may expect, and is a challenging issues that deserves further research.


Recommender systems E-tourism Evaluation Decision Making 


  1. 1.
    Bambini, R., Cremonesi, P., Turrin, R.: A Recommender System for an IPTV Service Provider: a Real Large-Scale Production Environment. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 299–331. Springer, US (2011)Google Scholar
  2. 2.
    Bollen, D., Knijnenburg, B.P., Willemsen, M.C., Graus, M.: Understanding choice overload in recommender systems. In: Proc. of the Fourth ACM Conference on Recommender Systems (RecSys 2010), pp. 63–70. ACM, New York (2010)CrossRefGoogle Scholar
  3. 3.
    Burke, R.: Hybrid web recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 377–408. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Celma, O., Herrera, P.: A new approach to evaluating novel recommendations. In: Proceedings of the 2008 ACM Conference on Recommender Systems (RecSys 2008), pp. 179–186. ACM, New York (2008)CrossRefGoogle Scholar
  5. 5.
    Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A., Turrin, R.: Looking for “good” recommendations: a comparative evaluation of recommender systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 152–168. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Cremonesi, P., Garzotto, F., Turrin, R.: Investigating the Persuasion Potential of Recommender Systems from a Quality Perspective: an Empirical Study. ACM Transactions on Interactive Intelligent Systems 2(2), 41 pages (2012)Google Scholar
  7. 7.
    Delgado, J., Davidson, R.: Knowledge bases and user profiling in travel and hospitality recommender systems (2002)Google Scholar
  8. 8.
    Deshpande, M., Karypis, G.: Item-centric top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004)CrossRefGoogle Scholar
  9. 9.
    Drenner, S., Sen, S., Terveen, L.: Crafting the initial user experience to achieve community goals. In: Proc. of RecSys 2008, pp. 187–194. ACM, New York (2008)Google Scholar
  10. 10.
    Golbandi, N., Koren, Y., Lempel, R.: On bootstrapping recommender systems. In: Proc. of the 19th ACM Int. Conf. on Information and Knowledge Management, pp. 1805–1808. ACM, New York (2010)Google Scholar
  11. 11.
    Hennig-Thurau, T., Gwinner, K.P., Walsh, G., Gremler, D.D.: Electronic Word-of Mouth via Consumer-Opinion Platforms: What Motivates Consumers to Articulate Themselves on the Internet? Journal of Interactive Marketing 18(1), 38–52 (2004)CrossRefGoogle Scholar
  12. 12.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 53 (2004)CrossRefGoogle Scholar
  13. 13.
    Jannach, D., Zanker, M., Jessenitschnig, M., Seidler, O.: Developing a conversational travel advisor with advisor suite. In: Sigala, M., Mich, L., Murphy, J. (eds.) Information and Communication Technologies in Tourism, pp. 43–52. Springer, Vienna (2007)Google Scholar
  14. 14.
    Jones, N., Pu, P.: User technology adoption issues in recommender systems. In: Proc. of Networking and Electronic Commerce Research Conf. (NAEC 2007), pp. 379–394 (2007)Google Scholar
  15. 15.
    Konstan, J.A., Riedl, J.: Recommender systems: from algorithms to user experience. User Model. User-Adapt. Interact. 22, 101–123 (2012)CrossRefGoogle Scholar
  16. 16.
    Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2008), pp. 426–434. ACM, New York (2008)CrossRefGoogle Scholar
  17. 17.
    Levi, A., Mokryn, O., Diot, C., Taft, N.: Finding a needle in a haystack of reviews: cold start context-centric hotel recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys 2012), pp. 115–122. ACM, New York (2012)CrossRefGoogle Scholar
  18. 18.
    Lops, P., De Gemmis, M., Semeraro, G.: Content-centric recommender systems: State of the art and trends. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 73–105. Springer, US (2011)Google Scholar
  19. 19.
    Mahmood, T., Ricci, F., Venturini, A., Höpken, W.: Adaptive recommender systems for travel planning. In: Proceedings of the International Conference on Information Technology and Travel & Tourism, ENTER, pp. 1–11. Springer (2008)Google Scholar
  20. 20.
    Mcnee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: Extended Abstracts on Human Factors in Computing Systems (CHI 2006), pp. 1097–1101. ACM, New York (2006)Google Scholar
  21. 21.
    Porter, S.R., Whitcomb, M.E.: The Impact of Lottery Incentives on Survey Response Rates. Research in Higher Education 44(4), 389–407 (2003)CrossRefGoogle Scholar
  22. 22.
    Pu, P., Chen, L., Hu, R.: A User-Centric Evaluation Framework for Recommender Systems. In: Proc. of RecSys 2011, pp. 157–164. ACM, New York (2011)Google Scholar
  23. 23.
    Rashid, A.M., Karypis, G., Riedl, J.: Learning preferences of new users in recommender systems: an information theoretic approach. SIGKDD Explorer Newsletter 10, 90–100 (2008)CrossRefGoogle Scholar
  24. 24.
    Ricci, F., Missier, F.: Supporting travel decision making through personalized recommendation. In: Karat, C.-M., Blom, J., Karat, J. (eds.) Designing Personalized User Experiences in eCommerce. Human–Computer Interaction Series, vol. 5, pp. 231–251. Springer, Netherlands (2004)Google Scholar
  25. 25.
    Ricci, F., Wietsma, R.: Product Reviews in Travel Decision Making. In: Proc. of the International Conference on Information and Communication Technologies in Tourism 2006, Lausanne, Switzerland, pp. 296–307. Springer Verlag (2006)Google Scholar
  26. 26.
    Senecal, S., Nantel, J.: The Influence of Online Product Recommendations on Consumers’ Online Choices. Journal of Retailing 80(2), 159–169 (2004)CrossRefGoogle Scholar
  27. 27.
    Swearingen, K., Sinha, R.: Beyond algorithms: An hci perspective on recommender systems. In: Proceedings of the ACM SIGIR Workshop on Recommender Systems (2001)Google Scholar
  28. 28.
    Takács, G., Pilászy, I., Németh, B., Tikk, D.: Scalable Collaborative Filtering Approaches for Large Recommender Systems. J. Mach. Learn. Res. 10, 623–656 (2009)Google Scholar
  29. 29.
    Tintarev, N.: Explanations of recommendations. In: Proceedings of the 2007 ACM Conference on Recommender Systems (RecSys 2007), pp. 203–206. ACM, New York (2007)CrossRefGoogle Scholar
  30. 30.
    Werthner, H., Ricci, F.: E-commerce and tourism. Commun. ACM 47(12), 101–105 (2004)CrossRefGoogle Scholar
  31. 31.
    Xie, M., Lakshmanan, L.V.S., Wood, P.T.: Comprec-trip: A composite recommendation system for travel planning. In: 2011 IEEE 27th International Conference on Data Engineering (ICDE), pp. 1352–1355 (April 2011)Google Scholar
  32. 32.
    Zhang, Y., Callan, J., Minka, T.: Novelty and redundancy detection in adaptive filtering. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), pp. 81–88. ACM, New York (2002)CrossRefGoogle Scholar
  33. 33.
    Zanker, M., Fuchs, M.W., Mario Tuta, H., Müller, N.: Evaluating recommender systems in tourism - a case study from austria. In: Proceedings of the International Conference on Information Technology and Travel & Tourism, ENTER, pp. 24–34. Springer (2008)Google Scholar
  34. 34.
    Ziegler, C.N., McNee, S.M., Konstan, J.A.: Lausen. G., Improving recommendation lists through topic diversification. In: Proceedings of the 14th International Conference on World Wide Web (WWW 2005), pp. 22–32. ACM, New York (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Paolo Cremonesi
    • 1
  • Franca Garzotto
    • 1
  • Roberto Turrin
    • 2
  1. 1.Politecnico di MilanoMilanoItaly
  2. 2.ContentWiseMilanoItaly

Personalised recommendations