User Modeling and User-Adapted Interaction

, Volume 22, Issue 4–5, pp 317–355 | Cite as

Evaluating recommender systems from the user’s perspective: survey of the state of the art

  • Pearl Pu
  • Li Chen
  • Rong Hu
Original Paper


A recommender system is a Web technology that proactively suggests items of interest to users based on their objective behavior or explicitly stated preferences. Evaluations of recommender systems (RS) have traditionally focused on the performance of algorithms. However, many researchers have recently started investigating system effectiveness and evaluation criteria from users’ perspectives. In this paper, we survey the state of the art of user experience research in RS by examining how researchers have evaluated design methods that augment RS’s ability to help users find the information or product that they truly prefer, interact with ease with the system, and form trust with RS through system transparency, control and privacy preserving mechanisms finally, we examine how these system design features influence users’ adoption of the technology. We summarize existing work concerning three crucial interaction activities between the user and the system: the initial preference elicitation process, the preference refinement process, and the presentation of the system’s recommendation results. Additionally, we will also cover recent evaluation frameworks that measure a recommender system’s overall perceptive qualities and how these qualities influence users’ behavioral intentions. The key results are summarized in a set of design guidelines that can provide useful suggestions to scholars and practitioners concerning the design and development of effective recommender systems. The survey also lays groundwork for researchers to pursue future topics that have not been covered by existing methods.


Research survey Recommender systems User experience research Explanation interface Design guidelines 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Adomavicius G., Tuzhilin A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. Adomavicius, G., Tuzhilin, A.: Context-aware recommender systems. In: The 2nd ACM Conference on Recommender Systems (RecSys ’08), pp. 335–336. ACM, New York (2008)Google Scholar
  3. Basartan, Y.: Amazon versus the shopbot: an experiment about how to improve the shopbots. Unpublished Ph.D. Summer Paper, Carnegie Mellon University, Pittsburgh, PA (2001)Google Scholar
  4. Beenen, G., Ling, K., Wang, X., Chang, K., Frankowski, D., Resnick, P., Kraut, R.E.: Using social psychology to motivate contributions to online communities. In: 2004 ACM Conference on Computer Supported Cooperative Work (CSCW ’04), pp. 212–221. ACM, New York (2004)Google Scholar
  5. Berger H., Denk M., Dittenbach M., Merkl D., Pesenhofer A.: Quo Vadis Homo Turisticus? Towards a picture-based tourist profiler. Inf. Commun. Technol. Tour. 2, 87–96 (2007)Google Scholar
  6. Bollen, D.G.F.M., Knijnenburg, B.P., Willemsen, M.C., Graus, M.P.: Understanding choice overload in recommender systems. In: The 4th ACM Conference on Recommender Systems (RecSys’10), pp. 63–70. ACM, New York (2010)Google Scholar
  7. Brodie C., Karat C.M., Karat J.: Creating an E-commerce environment where consumers are willing to share personal information. In: Karat, C., Blom, J.O., Karat, J. (eds) Designing Personalized User Experiences in eCommerce, pp. 185–206. Springer, Netherlands (2004)CrossRefGoogle Scholar
  8. Burke R.: Hybrid recommender systems: survey and experiments. User Model. User Adapt. Interact. 12(4), 331–370 (2002)zbMATHCrossRefGoogle Scholar
  9. Burke R., Hammond K., Young B.: The FindMe approach to assisted browsing. IEEE Expert Intell. Syst. Appl. 12(4), 32–40 (1997)Google Scholar
  10. Chen, L., Pu, P.: Preference-based organization interface: aiding user critiques in recommender systems. In: International Conference on User Modeling (UM’07), Corfu, Greece, 25–29 June, pp. 77–86 (2007)Google Scholar
  11. Chen L., Pu P.: Interaction design guidelines on critiquing-based recommender systems. User Model. User Adapt. Interact. 19(3), 167–206 (2009)CrossRefGoogle Scholar
  12. Chen L., Pu P.: Experiments on the preference-based organization interface in recommender systems. ACM Trans. Comput. Hum. Interact. 17(1), 1–33 (2010)Google Scholar
  13. Cosley, D., Lam, S.K., Albert, I., Konstan, J.A., Riedl, J.: Is seeing believing? How recommender system interfaces affect users’ opinions. In: SIGCHI Conference on Human Factors in Computing Systems (CHI ’03), pp. 585–592. ACM, New York (2003)Google Scholar
  14. Davis F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13, 319–339 (1989)CrossRefGoogle Scholar
  15. Drenner, S., Sen, S., Terveen, L.: Crafting the initial user experience to achieve community goals. In: 2008 ACM Conference on Recommender Systems (RecSys ’08), pp. 187–194. ACM, New York (2008)Google Scholar
  16. Dunn, G., Wiersema, J., Ham, J., Aroyo, L.: Evaluating interface variants on personality acquisition for recommender systems. In: Houben, G.-J., McCalla, G., Pianesi, F., Zancanaro, M. (eds.) User Modeling, Adaptation, and Personalization, vol. 5535, pp. 259–270. Springer-Verlag, Berlin (2009)Google Scholar
  17. Einhorn H., Hogarth R.: Confidence in judgment: persistence of the illusion of validity. Psychol. Rev. 85, 395–416 (1978)CrossRefGoogle Scholar
  18. Guttman, R.H.: Merchant differentiation through integrative negotiation in agent-mediated electronic commerce. Master’s Thesis, School of Architecture and Planning, Program in Media Arts and Sciences, Massachusetts Institute of Technology (1998)Google Scholar
  19. Haubl G., Trifts V.: Consumer decision making in online shopping environments: the effects of interactive decision aids. Mark. Sci. 19, 4–21 (2000)CrossRefGoogle Scholar
  20. Herlocker, J.L., Konstan, J.A., Riedl, J.: Explaining collaborative filtering recommendations. In: 2000 ACM Conference on Computer Supported Cooperative Work (CSCW ’00), pp. 241–250. ACM, New York (2000)Google Scholar
  21. Herlocker J.L., Konstan J.A., Terveen L.G., Riedl J.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004)CrossRefGoogle Scholar
  22. Hu, R., Pu, P.: A comparative user study on rating vs. personality quiz based preference elicitation methods. In: The 14th International Conference on Intelligent User Interfaces (IUI ’09), 8–11 February, pp. 367–372. ACM, Sanibel Island (2009a)Google Scholar
  23. Hu, R., Pu, P.: Acceptance issues of personality-based recommender systems. In: The 3rd ACM Conference on Recommender Systems (RecSys 2009), 22–25 October, pp. 221–224. ACM, New York (2009b)Google Scholar
  24. Hu, R., Pu, P.: A study on user perception of personality-based recommender systems. In: De Bra, P., Kobsa, A., Chin, D. (eds.) User Modeling, Adaptation, and Personalization, LNCS 6075, pp. 291–302. Springer, Heidelberg (2010)Google Scholar
  25. Hu, R., Pu, P.: Enhancing recommendation diversity with organization interfaces. In: the 16th International Conference on Intelligent user Interfaces (IUI ’11), pp. 347–350. ACM, New York (2011)Google Scholar
  26. Jones, N., Pu, P.: User technology adoption issues in recommender systems. In: Networking and Electronic Commerce Research Conference (NAEC ’07), pp. 379–394 (2007)Google Scholar
  27. Karau, S.J., Williams, K.D.: Understanding individual motivation in groups: the collective effort model. In: Turner, M.E. (ed.) Groups at Work: Theory and Research, pp. 113–141. LEA, Mahwah (2001)Google Scholar
  28. Kirakowski J.: SUMI: the software usability measurement inventory. Br. J. Educ. Technol. 24(3), 210–214 (1993)CrossRefGoogle Scholar
  29. Kleinmuntz D.N., Schkade D.A.: Information displays and decision processes. Psychol. Sci. 4, 221–227 (1993)CrossRefGoogle Scholar
  30. Knijnenburg, B.P., Willemsen, M.C., Hirtbach, S.: Receiving recommendations and providing feedback: the user-experience of a recommender system. In: The 11th International Conference on Electronic Commerce and Web Technologies, pp. 207–216 (2010)Google Scholar
  31. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User Adapt. Interact. 22 (2012) doi: 10.1007/s11257-011-9118-4
  32. Kobsa A., Schreck J.: Privacy through pseudonymity in user-adaptive systems. ACM Trans. Internet Technol. 3(2), 149–183 (2003)CrossRefGoogle Scholar
  33. Lam, S.K., Frankowski, D., Riedl, J.: Do you trust your recommendations? An exploration of security and privacy issues in recommender systems. In: 2006 International Conference on Emerging Trends in Information and Communication Security (ETRICS), Freiburg, Germany, pp. 14–29 (2006)Google Scholar
  34. Linden, G., Hanks, S., Lesh, N.: Interactive assessment of user preference models: the automated travel assistant. In: The Sixth International Conference on User Modeling, pp. 67–78. Springer, Chia Laguna (1997)Google Scholar
  35. Locke E.A., Latham G.P.: Building a practically useful theory of goal setting and task motivation: a 35 year odyssey. Am. Psychol. 57(9), 705–717 (2002)CrossRefGoogle Scholar
  36. Mahmood T., Ricci F., Venturini A.: Improving recommendation effectiveness by adapting the dialogue strategy in online travel planning. Int. J. Inf. Technol. Tour. 11(4), 285–302 (2010)Google Scholar
  37. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Thinking positively—explanatory feedback for conversational recommender systems. In: European Conference on Case-Based Reasoning (ECCBR-04) Explanation Workshop, Madrid, Spain, pp. 115–124 (2004)Google Scholar
  38. McCarthy, K., Reilly, J., McGinty, L., Smyth, B.: Experiments in dynamic critiquing. In: The 10th International Conference on Intelligent User Interfaces (IUI ’05), pp. 175–182. ACM, New York (2005)Google Scholar
  39. McGinty, L., Smyth, B.: On the role of diversity in conversational recommender systems. In: The Fifth International Conference on Case-Based Reasoning, pp. 276–290. Springer, Berlin (2003)Google Scholar
  40. McNee, M.S., Albert, I., Cosley, D., Gopalkrishnan, P., Lam, S.K., Rashid, A.M., Konstan, J.A., Riedl, J.: On the recommending of citations for research papers. In: 2002 ACM Conference on Computer Supported Cooperative Work (CSCW ’02), pp. 116–125. ACM, New York (2002)Google Scholar
  41. McNee, S.M., Lam, S.K., Konstan, J.A., Riedal, J.: 2003. Interfaces for eliciting new user preferences in recommender systems. In: User Modeling 2003, pp. 178–187 (2003)Google Scholar
  42. McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI Extended Abstracts, pp. 1097–1101 (2006a)Google Scholar
  43. McNee, S.M., Riedl, J., Konstan, J.A.: Making recommendations better: an analytic model for human-recommender interaction. In: CHI Extended Abstracts, pp. 1103–1108 (2006b)Google Scholar
  44. McSherry D.: Explanation in recommender systems. Artif. Intell. Rev. 24(2), 179–197 (2005)zbMATHCrossRefGoogle Scholar
  45. Ochi P., Rao S., Takayama L., Nass C.: Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int. J. Hum. Comput. Stud. 68(8), 472–482 (2010)CrossRefGoogle Scholar
  46. Paramythis A., Weibelzahl S., Masthoff J.: Layered evaluation of interactive adaptive systems: framework and formative methods. User Model. User Adapt. Interact. 20(5), 383–453 (2010)CrossRefGoogle Scholar
  47. Payne J.W., Bettman J.R., Schkade D.A.: Measuring constructed preference: towards a building code. J. Risk Uncertain. 19(1–3), 243–270 (1999)zbMATHCrossRefGoogle Scholar
  48. Pu, P., Chen, L.: Integrating tradeoff support in product search tools for e-commerce sites. In: The 6th ACM Conference on Electronic Commerce (EC ’05), pp. 269–278. ACM, New York (2005)Google Scholar
  49. Pu, P., Chen, L.: Trust building with explanation interfaces. In: The 11th International Conference on Intelligent User Interfaces (IUI ’06), pp. 93–100. ACM, New York (2006)Google Scholar
  50. Pu P., Chen L.: Trust-inspiring explanation interfaces for recommender systems. Knowl-Based. Syst. 20(6), 542–556 (2007)CrossRefGoogle Scholar
  51. Pu, P., Chen, L.: A user-centric evaluation framework of recommender systems. In: Workshop on User-Centric Evaluation of Recommender Systems and Their Interfaces (UCERSTI’10), ACM Conference on Recommender Systems (RecSys’10), Barcelona, Spain, pp. 14–21 (2010)Google Scholar
  52. Pu P., Faltings B.: Decision tradeoff using example-critiquing and constraint programming. Constraints Int. J. 9(4), 289–310 (2004)CrossRefGoogle Scholar
  53. Pu, P., Kumar, P.: Evaluating example-based search tools. In: The 5th ACM Conference on Electronic Commerce (EC ’04), pp. 208–217. ACM, New York (2004)Google Scholar
  54. Pu, P., Viappiani, P., Faltings, B.: Increasing user decision accuracy using suggestions. In: Grinter, R., Rodden, T., Aoki, P., Cutrell, E., Jeffries, R., Olson, G. (eds.) The SIGCHI Conference on Human Factors in Computing Systems (CHI ’06), ACM, New York (2006)Google Scholar
  55. Pu, P., Zhou, M., Castagnos, S.: Critiquing recommenders for public taste products. In: The Third ACM Conference on Recommender Systems (RecSys ’09), pp. 249–252. ACM, New York (2009)Google Scholar
  56. Pu, P., Faltings, B., Chen, L., Zhang, J.Y., Viappiani, P.: Usability guidelines for product recommenders based on example critiquing research. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, Chapter 16, pp. 511–546. Springer (2010)Google Scholar
  57. Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: The 5th ACM Conference on Recommender Systems (RecSys’11), Chicago, IL, USA, 23–27 October (2011)Google Scholar
  58. Rashid, A.M., Albert, I., Cosley, D., Lam, S.K., McNee, S.M., Konstan, J.A., Riedl, J.: Getting to know you: learning new user preferences in recommender systems. In: The 7th International Conference on Intelligent User Interfaces (IUI ’02), pp. 127–134. ACM, New York (2002)Google Scholar
  59. Reilly, J., McCarthy, K., McGinty, L., Smyth, B.: Dynamic critiquing. In: Funk, P., González Calero, P.A. (eds.) Advances in Case-Based Reasoning (ECCBR 2004), LNAI 3155, pp. 763–777. Springer, Heidelberg (2004)Google Scholar
  60. Resnick P., Varian H.R.: Recommender systems. Commun. ACM 40, 56–58 (1997)CrossRefGoogle Scholar
  61. Riedl J.: Personalization and privacy. IEEE Internet Comput. 5(6), 29–31 (2001)CrossRefGoogle Scholar
  62. Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Analysis of recommendation algorithms for e commerce. In: The 2nd ACM Conference on Electronic Commerce (EC ’00), pp. 158–167. ACM, New York (2000)Google Scholar
  63. Sarwar, B., Karypis, G., Konstan, J., Riedl, J. Item-based collaborative filtering recommendation algorithms. In: WWW’01, pp. 285–295 (2001)Google Scholar
  64. Shearin, S., Lieberman, H.: Intelligent profiling by example. In: The 6th International Conference on Intelligent User Interfaces (IUI ’01), pp. 145–151. ACM, New York (2001)Google Scholar
  65. Simonson I.: Determinants of customers’ responses to customized offers: conceptual framework and research propositions. J. Mark. 69, 32–45 (2005)CrossRefGoogle Scholar
  66. Sinha, R., Swearingen, K.: Comparing recommendations made by online systems and friends. In: DELOS-NSF Workshop on Personalization and Recommender Systems in Digital Libraries (2001)Google Scholar
  67. Sinha, R., Swearingen, K.: The role of transparency in recommender systems. In: CHI Extended Abstracts, pp. 830–831(2002)Google Scholar
  68. Smyth, B.: Case-based recommendation. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web: Methods and Strategies of Web Personalization, Lecture Notes in Computer Science, vol. 4321. Springer-Verlag, Berlin (2007)Google Scholar
  69. Smyth, B., McClave, P.: Similarity vs. diversity. In: Aha, D.W., Watson, I. (eds.) The 4th International Conference on Case-Based Reasoning: Case-Based Reasoning Research and Development (ICCBR ’01), pp. 347–361. Springer-Verlag, London (2001)Google Scholar
  70. Spiekermann S., Parachiv C.: Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. J. Electron. Commer. Res. 2(3), 255–285 (2002)zbMATHCrossRefGoogle Scholar
  71. Spiekermann, S., Grossklags, J., Berendt, B.: E-privacy in 2nd generation E-commerce: privacy preferences versus actual behavior. In: The 3rd ACM Conference on Electronic Commerce, pp. 38–47. ACM, New York (2001)Google Scholar
  72. Swearingen, K., Sinha, R.: Interaction design for recommender systems. In: Designing Interactive Systems (DIS’02), London, 25–28 June (2002)Google Scholar
  73. Tintarev, N., Masthoff, J.: Effective explanations of recommendations: user-centered design. In: 2007 ACM Conference on Recommender Systems (RecSys ’07), pp. 153–156. ACM, New York (2007a)Google Scholar
  74. Tintarev, N., Masthoff, J.: A survey of explanations in recommender systems. In: The 23rd IEEE International Conference on Data Engineering Workshop, pp. 801–810 (2007b)Google Scholar
  75. Viappiani, P., Faltings, B., Pu, P.: Evaluating preference-based search tools: a tale of two approaches. In: The Twenty-First National Conference on Artificial Intelligence (AAAI-06), Boston, USA, 16–20 July, pp. 205–210 (2006)Google Scholar
  76. Viappiani P., Faltings B., Pu P.: Preference-based search using example-critiquing with suggestions. J. Artif. Intell. Res. 27, 465–503 (2007)Google Scholar
  77. Vig, J., Sen, S., Riedl, J.: Tagsplanations: explaining recommendations using tags. In: The 13th International Conference on Intelligent User Interfaces (IUI’09), pp. 47–56. ACM, New York (2009)Google Scholar
  78. Williams, M.D., Tou, F.N.: RABBIT: an interface for database access. In: ACM ’82 Conference (ACM ’82), pp. 83–87. ACM, New York (1982)Google Scholar
  79. Xiao B., Benbasat I.: Ecommerce product recommendation agents: use, characteristics, and impact. MIS Q. 31(1), 137–209 (2007)Google Scholar
  80. Zanker M., Jessenitschnig M.: Case-studies on exploiting explicit customer requirements in recommender systems. User Model. User Adapt. Interact. 19(1–2), 133–166 (2009)CrossRefGoogle Scholar
  81. Ziegler, C., McNee, S.M., Konstan, J.A., Lausen, G.: Improving recommendation lists through topic diversification. In: The 14th International Conference on World Wide Web, pp. 22–32 (2005)Google Scholar
  82. Zukerman I., Albrecht D.W.: Predictive statistical models for user modeling. User Model. User Adapt. Interact. 11(1–2), 5–18 (2001)zbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.Human Computer Interaction Group, School of Computer and Communication SciencesSwiss Federal Institute of Technology (EPFL)LausanneSwitzerland
  2. 2.Department of Computer ScienceHong Kong Baptist UniversityKowloonHong Kong

Personalised recommendations