Abstract
Recommender Systems (RSs) aim at helping users search large amounts of contents and identify more effectively the items (products or services) that are likely to be more useful or attractive. The quality of a RS can be defined from two perspectives: system-centric, in which quality measures (e.g., precision, recall) are evaluated using vast datasets of preferences and opinions on items previously collected from users that are not interacting with the RS under study; user-centric, in which user measures are collected from users interacting with the RS under study. Prior research in e-commerce has provided some empirical evidence that system-centric and user-centric quality methods may lead to inconsistent results, e.g., RSs that were “best” according to system-centric measures were not the top ones according to user-centric measures. The paper investigates if a similar mismatch also exists in the domain of e-tourism. We discuss two studies that have adopted a system-centric approach using data from 210000 users, and a user-centric approach involving 240 users interacting with an online hotel booking service. In both studies, we considered four RSs that employ an implicit user preference elicitation technique and different baseline and state-of-the-art recommendation algorithms. In these four experimental conditions, we compared system-centric quality measures against user-centric evaluation results. System-centric quality measures were consistent with user-centric measures, in contrast with past studies in e-commerce. This pinpoints that the relationship between the two kinds of metrics may depend on the business sector, is more complex that we may expect, and is a challenging issues that deserves further research.
Chapter PDF
Similar content being viewed by others
References
Bambini, R., Cremonesi, P., Turrin, R.: A Recommender System for an IPTV Service Provider: a Real Large-Scale Production Environment. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 299–331. Springer, US (2011)
Bollen, D., Knijnenburg, B.P., Willemsen, M.C., Graus, M.: Understanding choice overload in recommender systems. In: Proc. of the Fourth ACM Conference on Recommender Systems (RecSys 2010), pp. 63–70. ACM, New York (2010)
Burke, R.: Hybrid web recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 377–408. Springer, Heidelberg (2007)
Celma, O., Herrera, P.: A new approach to evaluating novel recommendations. In: Proceedings of the 2008 ACM Conference on Recommender Systems (RecSys 2008), pp. 179–186. ACM, New York (2008)
Cremonesi, P., Garzotto, F., Negro, S., Papadopoulos, A., Turrin, R.: Looking for “good” recommendations: a comparative evaluation of recommender systems. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 152–168. Springer, Heidelberg (2011)
Cremonesi, P., Garzotto, F., Turrin, R.: Investigating the Persuasion Potential of Recommender Systems from a Quality Perspective: an Empirical Study. ACM Transactions on Interactive Intelligent Systems 2(2), 41 pages (2012)
Delgado, J., Davidson, R.: Knowledge bases and user profiling in travel and hospitality recommender systems (2002)
Deshpande, M., Karypis, G.: Item-centric top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2004)
Drenner, S., Sen, S., Terveen, L.: Crafting the initial user experience to achieve community goals. In: Proc. of RecSys 2008, pp. 187–194. ACM, New York (2008)
Golbandi, N., Koren, Y., Lempel, R.: On bootstrapping recommender systems. In: Proc. of the 19th ACM Int. Conf. on Information and Knowledge Management, pp. 1805–1808. ACM, New York (2010)
Hennig-Thurau, T., Gwinner, K.P., Walsh, G., Gremler, D.D.: Electronic Word-of Mouth via Consumer-Opinion Platforms: What Motivates Consumers to Articulate Themselves on the Internet? Journal of Interactive Marketing 18(1), 38–52 (2004)
Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 53 (2004)
Jannach, D., Zanker, M., Jessenitschnig, M., Seidler, O.: Developing a conversational travel advisor with advisor suite. In: Sigala, M., Mich, L., Murphy, J. (eds.) Information and Communication Technologies in Tourism, pp. 43–52. Springer, Vienna (2007)
Jones, N., Pu, P.: User technology adoption issues in recommender systems. In: Proc. of Networking and Electronic Commerce Research Conf. (NAEC 2007), pp. 379–394 (2007)
Konstan, J.A., Riedl, J.: Recommender systems: from algorithms to user experience. User Model. User-Adapt. Interact. 22, 101–123 (2012)
Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2008), pp. 426–434. ACM, New York (2008)
Levi, A., Mokryn, O., Diot, C., Taft, N.: Finding a needle in a haystack of reviews: cold start context-centric hotel recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems (RecSys 2012), pp. 115–122. ACM, New York (2012)
Lops, P., De Gemmis, M., Semeraro, G.: Content-centric recommender systems: State of the art and trends. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 73–105. Springer, US (2011)
Mahmood, T., Ricci, F., Venturini, A., Höpken, W.: Adaptive recommender systems for travel planning. In: Proceedings of the International Conference on Information Technology and Travel & Tourism, ENTER, pp. 1–11. Springer (2008)
Mcnee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: Extended Abstracts on Human Factors in Computing Systems (CHI 2006), pp. 1097–1101. ACM, New York (2006)
Porter, S.R., Whitcomb, M.E.: The Impact of Lottery Incentives on Survey Response Rates. Research in Higher Education 44(4), 389–407 (2003)
Pu, P., Chen, L., Hu, R.: A User-Centric Evaluation Framework for Recommender Systems. In: Proc. of RecSys 2011, pp. 157–164. ACM, New York (2011)
Rashid, A.M., Karypis, G., Riedl, J.: Learning preferences of new users in recommender systems: an information theoretic approach. SIGKDD Explorer Newsletter 10, 90–100 (2008)
Ricci, F., Missier, F.: Supporting travel decision making through personalized recommendation. In: Karat, C.-M., Blom, J., Karat, J. (eds.) Designing Personalized User Experiences in eCommerce. Human–Computer Interaction Series, vol. 5, pp. 231–251. Springer, Netherlands (2004)
Ricci, F., Wietsma, R.: Product Reviews in Travel Decision Making. In: Proc. of the International Conference on Information and Communication Technologies in Tourism 2006, Lausanne, Switzerland, pp. 296–307. Springer Verlag (2006)
Senecal, S., Nantel, J.: The Influence of Online Product Recommendations on Consumers’ Online Choices. Journal of Retailing 80(2), 159–169 (2004)
Swearingen, K., Sinha, R.: Beyond algorithms: An hci perspective on recommender systems. In: Proceedings of the ACM SIGIR Workshop on Recommender Systems (2001)
Takács, G., Pilászy, I., Németh, B., Tikk, D.: Scalable Collaborative Filtering Approaches for Large Recommender Systems. J. Mach. Learn. Res. 10, 623–656 (2009)
Tintarev, N.: Explanations of recommendations. In: Proceedings of the 2007 ACM Conference on Recommender Systems (RecSys 2007), pp. 203–206. ACM, New York (2007)
Werthner, H., Ricci, F.: E-commerce and tourism. Commun. ACM 47(12), 101–105 (2004)
Xie, M., Lakshmanan, L.V.S., Wood, P.T.: Comprec-trip: A composite recommendation system for travel planning. In: 2011 IEEE 27th International Conference on Data Engineering (ICDE), pp. 1352–1355 (April 2011)
Zhang, Y., Callan, J., Minka, T.: Novelty and redundancy detection in adaptive filtering. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002), pp. 81–88. ACM, New York (2002)
Zanker, M., Fuchs, M.W., Mario Tuta, H., Müller, N.: Evaluating recommender systems in tourism - a case study from austria. In: Proceedings of the International Conference on Information Technology and Travel & Tourism, ENTER, pp. 24–34. Springer (2008)
Ziegler, C.N., McNee, S.M., Konstan, J.A.: Lausen. G., Improving recommendation lists through topic diversification. In: Proceedings of the 14th International Conference on World Wide Web (WWW 2005), pp. 22–32. ACM, New York (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cremonesi, P., Garzotto, F., Turrin, R. (2013). User-Centric vs. System-Centric Evaluation of Recommender Systems. In: Kotzé, P., Marsden, G., Lindgaard, G., Wesson, J., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2013. INTERACT 2013. Lecture Notes in Computer Science, vol 8119. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40477-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-642-40477-1_21
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40476-4
Online ISBN: 978-3-642-40477-1
eBook Packages: Computer ScienceComputer Science (R0)