Meta-analysis of evaluation methods and metrics used in context-aware scholarly recommender systems

  • Zohreh Dehghani ChampiriEmail author
  • Adeleh AsemiEmail author
  • Salim Siti Salwah BintiEmail author
Regular Paper


With the current growth of the proposed contextual recommending algorithms, evaluating them becomes more critical. Researchers of recommender systems have expressed concerns that the evaluation quality cannot be properly judged. We carried out meta-analyses of the evaluation methods and metrics of 67 studies related to context-aware scholarly recommender systems, from the years 2000 to 2014. The analysis of variance results shows that offline evaluation methods are more commonly used compared to online and user studies, with the maximum rate of success. It also reveals the popularity order of accuracy metrics (31%) including “Recall, Precision, F-Measure”, “Mean Absolute Error, and Questionnaire studies, Reliability, Accessibility, Feasibility, Usability, Applicability and Performance”. By using factor analysis, 28 different evaluation metrics were classified into eight groups. The results of analysis have shown the difference in evaluation methods in applying different groups of metrics. This study highlights the importance of how an evaluation method should be adequately designed and implemented. Additionally, a few recommendations for future investigations on recommending evaluation are proposed.


Scholarly recommender systems Research paper recommender systems Evaluation Method Metric Context-aware Contextual information 


  1. 1.
    Champiri ZD, Shahamiri SR, Salim SSB (2015) A systematic review of scholar context-aware recommender systems. Expert Syst Appl 42(3):1743–1758CrossRefGoogle Scholar
  2. 2.
    Riboni D, Bettini C (2012) Private context-aware recommendation of points of interest: an initial investigation. In: IEEE international conference on pervasive computing and communications workshops (PERCOM Workshops). IEEEGoogle Scholar
  3. 3.
    Gediminas Adomavicius BM, Francesco R, Alex T (2011) Context-aware recommender systems. Association for the Advancement of Artificial Intelligence. All rights reserved. ISSN 0738-4602Google Scholar
  4. 4.
    Dehghani Champiri Z et al (2011) A multi-layer contextual model for recommender systems in digital libraries. In: Aslib proceedings. Emerald Group Publishing LimitedGoogle Scholar
  5. 5.
    Panniello U, Tuzhilin A, Gorgoglione M (2014) Comparing context-aware recommender systems in terms of accuracy and diversity. User Model User Adapt Interact 24(1–2):35–65CrossRefGoogle Scholar
  6. 6.
    Lim BY, Dey AK, Avrahami D (2009) Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACMGoogle Scholar
  7. 7.
    Gunawardana A, Shani G (2009) A survey of accuracy evaluation metrics of recommendation tasks. J Mach Learn Res 10:2935–2962MathSciNetzbMATHGoogle Scholar
  8. 8.
    Beel J, Langer S (2014) A comparison of offline evaluations, online evaluations, and user studies in the context of research paper recommender systems. Under Review. Pre-print available at
  9. 9.
    Said A, Bellogín A (2014) Comparative recommender system evaluation: benchmarking recommendation frameworks. In: Proceedings of the 8th ACM conference on recommender systems. ACMGoogle Scholar
  10. 10.
    Ekstrand MD (2014) Towards recommender engineering tools and experiments for identifying recommender differences. University of Minnesota, MinneapolisGoogle Scholar
  11. 11.
    Kluver D, Konstan JA (2014) Evaluating recommender behavior for new users. In: Proceedings of the 8th ACM conference on recommender systems. ACMGoogle Scholar
  12. 12.
    Manouselis N, Karagiannidis C, Sampson D (2014) Layered evaluation in recommender systems: a retrospective assessment. J e-Learn Knowl Soc 10(1)Google Scholar
  13. 13.
    Champiri ZD, Salim SSB, Shahamiri SR (2015) The role of context for recommendations in digital libraries. Int J Soc Sci Humanity 5(11):948CrossRefGoogle Scholar
  14. 14.
    Baltrunas L et al (2012) Context relevance assessment and exploitation in mobile recommender systems. Pers Ubiquit Comput 16(5):507–526CrossRefGoogle Scholar
  15. 15.
    Adomavicius G, Tuzhilin A (2005) Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749CrossRefGoogle Scholar
  16. 16.
    Panniello U, Gorgoglione M (2011) Context-aware recommender systems: a comparison of three approaches. In DART@ AI* IAGoogle Scholar
  17. 17.
    Adomavicius G, Tuzhilin A (2011) Context-aware recommender systems. In: Recommender systems handbook. Springer, pp 217–253Google Scholar
  18. 18.
    Liu L (2013) The implication of context and criteria information in recommender systems as applied to the service domain. University of Manchester, ManchesterGoogle Scholar
  19. 19.
    Adomavicius G et al (2005) Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans Inf Syst 23(1):103–145CrossRefGoogle Scholar
  20. 20.
    Adomavicius G, Tuzhilin A (2011) Context-aware recommender systems. In: Recommender systems handbook. Springer, Boston, MA, pp 217–253Google Scholar
  21. 21.
    Kantor PB et al (2011) Recommender systems handbook. Springer, New YorkzbMATHGoogle Scholar
  22. 22.
    Baltrunas L, Ricci F (2009) Context-based splitting of item ratings in collaborative filtering. In: Proceedings of the third ACM conference on recommender systems. ACMGoogle Scholar
  23. 23.
    Panniello U et al (2009) Experimental comparison of pre-versus post-filtering approaches in context-aware recommender systems. In: Proceedings of the third ACM conference on recommender systems. ACMGoogle Scholar
  24. 24.
    Yujie Z, Licai W (2010) Some challenges for context-aware recommender systems. In: 5th International conference on computer science and education (ICCSE). IEEEGoogle Scholar
  25. 25.
    Shani G, Gunawardana A (2011) Evaluating recommendation systems. In: Recommender systems handbook. Springer, pp 257–297Google Scholar
  26. 26.
    Herlocker JL et al (2004) Evaluating collaborative filtering recommender systems. ACM Trans Inf Syst 22(1):5–53CrossRefGoogle Scholar
  27. 27.
    Beel J et al (2013) A comparative analysis of offline and online evaluations and discussion of research paper recommender system evaluation. In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACMGoogle Scholar
  28. 28.
    Said A (2013) Evaluating the accuracy and utility of recommender systems. Doctoral dissertation, Universitätsbibliothek der Technischen Universität, BerlinGoogle Scholar
  29. 29.
    de Wit J (2008) Evaluating recommender systems. In: An evaluation framework to predict user satisfaction for recommender systems in an electronic program guide contextGoogle Scholar
  30. 30.
    Jannach D et al (2013) What recommenders recommend—an analysis of accuracy, popularity, and sales diversity effects. In: User modeling, adaptation, and personalization. Springer, pp 25–37Google Scholar
  31. 31.
    Zaier Z, Godin R, Faucher L (2008) Evaluating recommender systems. In: Automated solutions for cross media content and multi-channel distribution, AXMEDIS’08 international conference. IEEEGoogle Scholar
  32. 32.
    Pu P, Chen L, Hu R (2012) Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model User Adapt Interact 22(4–5):317–355CrossRefGoogle Scholar
  33. 33.
    Murakami T, Mori K, Orihara R (2007) Metrics for evaluating the serendipity of recommendation lists. In: Annual conference of the Japanese society for artificial intelligence. SpringerGoogle Scholar
  34. 34.
    Adamopoulos P, Tuzhilin A (2011) On unexpectedness in recommender systems: or how to expect the unexpected. In: Workshop on novelty and diversity in recommender systems (DiveRS 2011), at the 5th ACM international conference on recommender systems (RecSys’11). ACM, ChicagoGoogle Scholar
  35. 35.
    Ge M, Delgado-Battenfeld C, Jannach D (2010) Beyond accuracy: evaluating recommender systems by coverage and serendipity. In: Proceedings of the fourth ACM conference on recommender systems. ACMGoogle Scholar
  36. 36.
    Parra D, Sahebi S (2013) Recommender systems: sources of knowledge and evaluation metrics. In: Advanced techniques in web intelligence-2. Springer, pp 149–175Google Scholar
  37. 37.
    Schröder G, Thiele M, Lehner W (2011) Setting goals and choosing metrics for recommender system evaluations. In: UCERSTI2 workshop at the 5th ACM conference on recommender systems, Chicago, USAGoogle Scholar
  38. 38.
    Beel J et al (2013) Research paper recommender system evaluation: a quantitative literature survey. In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACMGoogle Scholar
  39. 39.
    Erdt M, Fernández A, Rensing C (2015) Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Trans Learn Technol 8(4):326–344CrossRefGoogle Scholar
  40. 40.
    Bobadilla J et al (2013) Recommender systems survey. Knowl Based Syst 46:109–132CrossRefGoogle Scholar
  41. 41.
    Tintarev N, Masthoff J (2007) A survey of explanations in recommender systems. In: IEEE 23rd international conference on data engineering workshop. IEEEGoogle Scholar
  42. 42.
    McNee SM, Riedl J, Konstan JA (2006) Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI’06 extended abstracts on human factors in computing systems. ACMGoogle Scholar
  43. 43.
    Konstan JA (2004) Introduction to recommender systems: algorithms and evaluation. ACM Trans Inf Syst 22(1):1–4CrossRefGoogle Scholar
  44. 44.
    Ferrier L et al (1995) Dysarthric speakers’ intelligibility and speech characteristics in relation to computer speech recognition. Augment Altern Commun 11(3):165–175CrossRefGoogle Scholar
  45. 45.
    Tenenhaus M, Amato S, Esposito Vinzi V (2004) A global goodness-of-fit index for PLS structural equation modelling. In: Proceedings of the XLII SIS scientific meetingGoogle Scholar
  46. 46.
    Kitchenham BA, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. EBSE technical report EBSE, pp 1–57Google Scholar
  47. 47.
    Knijnenburg BP et al (2012) Explaining the user experience of recommender systems. User Model User Adapt Interact 22(4–5):441–504CrossRefGoogle Scholar
  48. 48.
    Porcel C, Herrera-Viedma E (2010) Dealing with incomplete information in a fuzzy linguistic recommender system to disseminate information in university digital libraries. Knowl Based Syst 23(1):32–39CrossRefGoogle Scholar
  49. 49.
    Reuters T (2013) EndNote X7. Thomson Reuters, PhiladelphiaGoogle Scholar
  50. 50.
    Dey AK (2001) Understanding and using context. Pers Ubiquit Comput 5(1):4–7CrossRefGoogle Scholar
  51. 51.
    Harman HH (1976) Modern factor analysis. University of Chicago PressGoogle Scholar
  52. 52.
    Polit DF, Beck CT (2008) Nursing research: generating and assessing evidence for nursing practice. Lippincott Williams & Wilkins, PhiladelphiaGoogle Scholar
  53. 53.
    Croxton FE, Cowden DJ, Klein S, Prentice-Hall Inc, Englewood Cliffs NJ (1967) Applied General Statistics. J Am Stat Assoc 63(322):738Google Scholar
  54. 54.
    Geisler G, McArthur D, Giersch S (2001) Developing recommendation services for a digital library with uncertain and changing data. In: Proceedings of the 1st ACM/IEEE-CS joint conference on digital libraries. ACMGoogle Scholar
  55. 55.
    De Giusti MR et al (2010) An ontology-based context aware system for selective dissemination of information in a digital library. arXiv preprint arXiv:1005.4008
  56. 56.
    Torres R et al (2004) Enhancing digital libraries with TechLens+. In: Proceedings of the 4th ACM/IEEE-CS joint conference on digital libraries. ACMGoogle Scholar
  57. 57.
    Gantner Z et al (2011) MyMediaLite: a free recommender system library. In: Proceedings of the fifth ACM conference on recommender systems. ACMGoogle Scholar
  58. 58.
    Hwang S-Y, Hsiung W-C, Yang W-S (2003) A prototype WWW literature recommendation system for digital libraries. Online Inf Rev 27(3):169–182CrossRefGoogle Scholar
  59. 59.
    Sugiyama K, Kan MY (2010) Scholarly paper recommendation via user’s recent research interests. In: Proceedings of the 10th annual joint conference on digital libraries. ACMGoogle Scholar
  60. 60.
    Wang CY et al (2004) Extending e-books with contextual knowledge recommenders by analyzing personal portfolio and annotation to help learners solve problems in time. In: Proceedings of IEEE international conference on advanced learning technologies. IEEEGoogle Scholar
  61. 61.
    Wang F-H, Shao H-M (2004) Effective personalized recommendation based on time-framed navigation clustering and association mining. Expert Syst Appl 27(3):365–377MathSciNetCrossRefGoogle Scholar
  62. 62.
    Konstan JA et al (2005) Techlens: exploring the use of recommenders to support users of digital libraries. In: CNI fall task force meeting project briefing. Coalition for networked information, PhoenixGoogle Scholar
  63. 63.
    Liao I-E et al (2010) A library recommender system based on a personal ontology model and collaborative filtering technique for English collections. Electron Lib 28(3):386–400CrossRefGoogle Scholar
  64. 64.
    Wu D et al (2012) Temporal social tagging based collaborative filtering recommender for digital library. In: The outreach of digital libraries: a globalized resource network. Springer, pp 199–208Google Scholar
  65. 65.
    Yuan Z, Yu T, Zhang J (2011) A social tagging based collaborative filtering recommendation algorithm for digital library. In: Digital libraries: for cultural heritage, knowledge dissemination, and future creation. Springer, pp 192–201Google Scholar
  66. 66.
    Trujillo M, Millan M, Ortiz E (2007) A recommender system based on multi-features. In: Computational science and its applications—ICCSA 2007. Springer, pp 370–382Google Scholar
  67. 67.
    He Q et al (2010) Context-aware citation recommendation. In: Proceedings of the 19th international conference on world wide web. ACMGoogle Scholar
  68. 68.
    Porcel C, Moreno JM, Herrera-Viedma E (2009) A multi-disciplinar recommender system to advice research resources in university digital libraries. Expert Syst Appl 36(10):12520–12528CrossRefGoogle Scholar
  69. 69.
    Porcel C, Herrera-Viedma E (2010) Dealing with incomplete information in a fuzzy linguistic recommender system to disseminate information in university digital libraries. Knowl Based Syst 23(1):32–39CrossRefGoogle Scholar
  70. 70.
    Sun Y, Ni W, Men R (2009) A personalized paper recommendation approach based on web paper mining and reviewer’s interest modeling. In: ICRCCS’09 international conference on research challenges in computer science. IEEEGoogle Scholar
  71. 71.
    Morales-del-Castillo JM, Peis E, Herrera-Viedma E (2009) A filtering and recommender system prototype for scholarly users of digital libraries. Springer, New YorkCrossRefGoogle Scholar
  72. 72.
    Hwang S-Y, Wei C-P, Liao Y-F (2010) Coauthorship networks and academic literature recommendation. Electron Commer Res Appl 9(4):323–334CrossRefGoogle Scholar
  73. 73.
    Rocha LM (2001) TalkMine: a soft computing approach to adaptive knowledge recommendation. In: Soft computing agents. Springer, pp 89–116Google Scholar
  74. 74.
    Rao KN, Talwar VG (2011) Content-based document recommender system for aerospace grey literature: system design. DESIDOC J Lib Inf Technol.
  75. 75.
    Pagonis J, Clark AF (2010) Engene: a genetic algorithm classifier for content-based recommender systems that does not require continuous user feedback. In: 2010 UK workshop on computational intelligence (UKCI)Google Scholar
  76. 76.
    Serrano-Guerrero J et al (2011) A google wave-based fuzzy recommender system to disseminate information in University Digital Libraries 2.0. Inf Sci 181(9):1503–1516CrossRefGoogle Scholar
  77. 77.
    Will T et al (2009) Search personalization: Knowledge-based recommendation in digital libraries. In: AMCIS 2009 proceedings, p 728Google Scholar
  78. 78.
    Tsai C-S, Chen M-Y (2008) Using adaptive resonance theory and data-mining techniques for materials recommendation based on the e-library environment. Electron Lib 26(3):287–302CrossRefGoogle Scholar
  79. 79.
    Rodriguez MA et al (2009) A recommender system to support the scholarly communication process. arXiv preprint arXiv:0905.1594
  80. 80.
    Middleton SE, Shadbolt NR, De Roure DC (2004) Ontological user profiling in recommender systems. ACM Trans Inf Syst 22(1):54–88CrossRefGoogle Scholar
  81. 81.
    Amini B et al (2011) Incorporating scholar’s background knowledge into recommender system for digital libraries. In: 5th Malaysian conference in software engineering (MySEC). IEEEGoogle Scholar
  82. 82.
    Lopes GR et al (2008) A personalized recommender system for digital libraries. In: Proceedings of the 14th Brazilian symposium on multimedia and the web. ACMGoogle Scholar
  83. 83.
    Gipp B, Beel J, Hentschel C (2009) Scienstein: a research paper recommender system. In: Proceedings of the international conference on emerging trends in computing (ICETiC’09)Google Scholar
  84. 84.
    Vellino A, Zeber D (2007) A hybrid, multi-dimensional recommender for journal articles in a scientific digital library. In: Proceedings of the 2007 IEEE/WIC/ACM international conference on web intelligence and international conference on intelligent agent technologyGoogle Scholar
  85. 85.
    McNee SM, Riedl J, Konstan JA (2006) Making recommendations better: an analytic model for human-recommender interaction. In: CHI’06 extended abstracts on human factors in computing systems. ACMGoogle Scholar
  86. 86.
    Pham MC et al (2011) A clustering approach for collaborative filtering recommendation using social network analysis. J UCS 17(4):583–604Google Scholar
  87. 87.
    Yang Y, Yun L (2010) Literature recommendation based on reference graph. In: 3rd International conference on advanced computer theory and engineering (ICACTE). IEEEGoogle Scholar
  88. 88.
    Yang W-S, Lin Y-R (2013) A task-focused literature recommender system for digital libraries. Online Inf Rev 37(4):581–601CrossRefGoogle Scholar
  89. 89.
    Patton RM, Potok TE, Worley BA (2012) Discovery & refinement of scientific information via a recommender system. In: The second international conference on advanced communications and computationGoogle Scholar
  90. 90.
    Herlocker J, Jung S, Webster JG (2012) Collaborative filtering for digital librariesGoogle Scholar
  91. 91.
    Nakagawa A, Ito T (2002) An implementation of a knowledge recommendation system based on similarity among users’ profiles. In: Proceedings of the 41st SICE annual conference on SICE 2002. IEEEGoogle Scholar
  92. 92.
    Renda ME, Straccia U (2005) A personalized collaborative digital library environment: a model and an application. Inf Process Manag 41(1):5–21zbMATHCrossRefGoogle Scholar
  93. 93.
    Webster J, Jung S, Herlocker J (2004) Collaborative filtering: a new approach to searching digital libraries. N Rev Inf Netw 10(2):177–191CrossRefGoogle Scholar
  94. 94.
    Tejeda-Lorente Á et al (2014) A quality based recommender system to disseminate information in a university digital library. Inf Sci 261:52–69CrossRefGoogle Scholar
  95. 95.
    Mcnee SM (2006) Meeting user information needs in recommender systems. Doctoral degree. University of MinnesotaGoogle Scholar
  96. 96.
    Pennock DM et al (2000) Collaborative filtering by personality diagnosis: a hybrid memory-and model-based approach. In: Proceedings of the sixteenth conference on uncertainty in artificial intelligence. Morgan Kaufmann Publishers IncGoogle Scholar
  97. 97.
    Steinberg RM et al (2010) SmartSearch: automated recommendations using librarian expertise and the National Center for Biotechnology Information’s Entrez Programming Utilities. J Med Lib Assoc 98(2):171CrossRefGoogle Scholar
  98. 98.
    Franke M, Geyer-Schulz A, Neumann AW (2008) Recommender services in scientific digital libraries. In: Multimedia services in intelligent environments. Springer, pp 377–417Google Scholar
  99. 99.
    Zhang M, Wang W, Li X (2008) A paper recommender for scientific literatures based on semantic concept similarity. In: Digital libraries: universal and ubiquitous access to information. Springer, pp 359–362Google Scholar
  100. 100.
    Kang J, Choi J (2011) An ontology-based recommendation system using long-term and short-term preferences. In: International conference on information science and applications (ICISA). IEEEGoogle Scholar
  101. 101.
    Wakeling S (2012) The user-centered design of a recommender system for a universal library catalogue. In: Proceedings of the sixth ACM conference on recommender systems. ACMGoogle Scholar
  102. 102.
    De Nart D, Ferrara F, Tasso C (2013) Personalized access to scientific publications: from recommendation to explanation. In: User modeling, adaptation, and personalization. Springer, pp 296–301Google Scholar
  103. 103.
    Li QC, Dong ZH, Li T (2008) Research of information recommendation system based on reading behavior. In: International conference on machine learning and cybernetics. IEEEGoogle Scholar
  104. 104.
    Kuo JJ, Zhang YJ (2012) A library recommender system using interest change over time and matrix clustering. In: The outreach of digital libraries: a globalized resource network. Springer, pp 259–268Google Scholar
  105. 105.
    Tsuji K et al (2012) Use of library loan records for book recommendation. In: IIAI international conference on advanced applied informatics (IIAIAAI). IEEEGoogle Scholar
  106. 106.
    Herlocker JL, Konstan J (2001) Content-independent task-focused recommendation. Internet Comput IEEE 5(6):40–47CrossRefGoogle Scholar
  107. 107.
    Chandrasekaran K et al (2008) Concept-based document recommendations for citeseer authors. In: Adaptive hypermedia and adaptive web-based systems. SpringerGoogle Scholar
  108. 108.
    Whitney C, Schiff LR (2006) The Melvyl recommender project: developing library recommendation services. California Digital Library, OaklandCrossRefGoogle Scholar
  109. 109.
    Aittola M, Ryhänen T, Ojala T (2003) SmartLibrary—location-aware mobile library service, in human–computer interaction with mobile devices and services. Springer, pp 411–416Google Scholar
  110. 110.
    Zarrinkalam F, Kahani M (2013) SemCiR: a citation recommendation system based on a novel semantic distance measure. Program 47(1):92–112CrossRefGoogle Scholar
  111. 111.
    Middleton SE, Alani H, De Roure DC (2002) Exploiting synergy between ontologies and recommender systems. arXiv preprint cs/0204012Google Scholar
  112. 112.
    Middleton SE, Shadbolt NR, De Roure DC (2003) Capturing interest through inference and visualization: ontological user profiling in recommender systems. In: Proceedings of the 2nd international conference on knowledge capture. ACMGoogle Scholar
  113. 113.
    Geyer-Schulz A, Neumann A, Thede A (2003) Others also use: a robust recommender system for scientific libraries. In: Research and advanced technology for digital libraries. Springer, pp 113–125Google Scholar
  114. 114.
    Jung S et al (2004) SERF: integrating human recommendations with search. In: Proceedings of the thirteenth ACM international conference on information and knowledge management. ACMGoogle Scholar
  115. 115.
    Amini B, Ibrahim R, Othman MS (2013) Data sets for offline evaluation of scholar’s recommender system. In: Intelligent information and database systems. Springer, pp 158–167Google Scholar
  116. 116.
    McNee SM et al (2002) On the recommending of citations for research papers. In: Proceedings of the 2002 ACM conference on computer supported cooperative work. ACMGoogle Scholar
  117. 117.
    Konstan JA, Riedl J (1999) Research resources for recommender systems. In: CHI’99 workshop interacting with recommender systemsGoogle Scholar
  118. 118.
    Beel J et al (2016) Towards reproducibility in recommender-systems research. User Model User Adapt Interact 26(1):69–101CrossRefGoogle Scholar
  119. 119.
    del Olmo FH, Gaudioso E (2008) Evaluation of recommender systems: a new approach. Expert Syst Appl 35(3):790–804CrossRefGoogle Scholar
  120. 120.
    Pu P, Chen L, Hu R (2011) A user-centric evaluation framework for recommender systems. In: Proceedings of the fifth ACM conference on recommender systems. ACMGoogle Scholar
  121. 121.
    Hayes C, Cunningham P (2002) An on-line evaluation framework for recommender systems. Trinity College Dublin, Department of Computer Science, DublinGoogle Scholar
  122. 122.
    Hanson EM (2014) A beginner’s guide to creating library linked data: lessons from NCSU’s organization name linked data project. Ser Rev 40(4):251–258CrossRefGoogle Scholar
  123. 123.
    Figueroa C, Vagliano I, Rocha OR, Morisio M (2015) A systematic literature review of linked data-based recommender systems. Concurr Comput Pract Exp 27(17):4659–4684CrossRefGoogle Scholar
  124. 124.
    Cremonesi P, Garzotto F, Turrin R (2013) User-centric versus system-centric evaluation of recommender systems. In: Human–Computer interaction–INTERACT 2013. Springer, pp 334–351Google Scholar
  125. 125.
    Levy M (2013) Offline evaluation of recommender systems: all pain and no gain? In: Proceedings of the international workshop on reproducibility and replication in recommender systems evaluation. ACMGoogle Scholar
  126. 126.
    Mobasher B et al (2001) Effective personalization based on association rule discovery from web usage data. In: Proceedings of the 3rd international workshop on web information and data management. ACMGoogle Scholar
  127. 127.
    Sinha R, Swearingen K (2002) The role of transparency in recommender systems. In: CHI’02 extended abstracts on human factors in computing systems. ACMGoogle Scholar
  128. 128.
    Wu W, He L, Yang J (2012) Evaluating recommender systems. In: Seventh international conference on digital information management (ICDIM). IEEEGoogle Scholar
  129. 129.
    Powers D (2007) Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation (Technical Report). Adelaide, AustraliaGoogle Scholar
  130. 130.
    Iaquinta L et al (2008) Introducing serendipity in a content-based recommender system. In: Hybrid intelligent systems. HIS’08 Eighth international conference. IEEEGoogle Scholar
  131. 131.
    Kotkov D, Wang S, Veijalainen J (2016) A survey of serendipity in recommender systems. Knowl Based Syst 111:180–192CrossRefGoogle Scholar
  132. 132.
    Konstan JA, Riedl J (2012) Recommender systems: from algorithms to user experience. User Model User Adapt Interact 22(1):101–123CrossRefGoogle Scholar
  133. 133.
    Ozok AA, Fan Q, Norcio AF (2010) Design guidelines for effective recommender system interfaces based on a usability criteria conceptual model: results from a college student population. Behav Inf Technol 29(1):57–83CrossRefGoogle Scholar
  134. 134.
    Hiesel P et al (2016) A user interface concept for context-aware recommender systems. Mensch und Computer, TagungsbandGoogle Scholar
  135. 135.
    di Sciascio C (2017) Advanced user interfaces and hybrid recommendations for exploratory search. In: Proceedings of the 22nd iaaanternational conference on intelligent user interfaces companion. ACMGoogle Scholar
  136. 136.
    Calero Valdez A, Ziefle M, Verbert K (2016) HCI for recommender systems: the past, the present and the future. In: Proceedings of the 10th ACM conference on recommender systems. ACMGoogle Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Software Engineering, Faculty of Computer Science and Information TechnologyUniversity of MalayaLembah Pantai, Kuala LumpurMalaysia
  2. 2.Safahan Institute of Higher EducationIsfahanIran

Personalised recommendations