Evaluation Metrics

  • Òscar CelmaEmail author


This chapter presents the different evaluation methods for a recommender system. We introduce the existing metrics, as well as the pros and cons of each method. This chapter is the background for the following Chaps. 6 and 7, where the proposed metrics are used in real, large size, recommendation datasets.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    J. S. Breese, D. Heckerman, and C. Kadie, “Empirical analysis of predictive algorithms for collaborative filtering,” tech. rep., Microsoft Research, 1998.Google Scholar
  2. 2.
    J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating collaborative filtering recommender systems,” ACM Transaction on Information Systems, vol. 22, no. 1, pp. 5–53, 2004.CrossRefGoogle Scholar
  3. 3.
    G. Shani and A. Gunawardana, “Evaluating recommender systems,” tech. rep., Microsoft Research, MSR-TR-2009–159, November 2009.Google Scholar
  4. 4.
    M. Balabanovic and Y. Shoham, “Fab: Content-based, collaborative recommendation,” Communications of the ACM, vol. 40, pp. 66–72, 1997.CrossRefGoogle Scholar
  5. 5.
    Y. Y. Yao, “Measuring retrieval effectiveness based on user preference of documents,” Journal of the American Society for Information Science, vol. 46, no. 2, pp. 133–145, 1995.CrossRefGoogle Scholar
  6. 6.
    G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” IEEE Transactions on Knowledge and Data Engineering, vol. 17, no. 6, pp. 734–749, 2005.CrossRefGoogle Scholar
  7. 7.
    S. M. McNee, J. Riedl, and J. A. Konstan, “Being accurate is not enough: How accuracy metrics have hurt recommender systems,” in Computer Human Interaction. Human factors in computing systems, (New York, NY), pp. 1097–1101, ACM, 2006.Google Scholar
  8. 8.
    P. Erdös and A. Réyi, “On random graphs,” Science, vol. 6, no. 290, pp. 290–298, 1959.zbMATHGoogle Scholar
  9. 9.
    A. L. Barabási and R. Albert, “Emergence of scaling in random networks,” Science, vol. 286, pp. 509–512, October 1999.CrossRefMathSciNetGoogle Scholar
  10. 10.
    M. E. J. Newman, “Assortative mixing in networks,” Physical Review Letters, vol. 89, no. 20, 2002.Google Scholar
  11. 11.
    M. E. J. Newman, “Mixing patterns in networks,” Physical Review E, vol. 67, 2003.Google Scholar
  12. 12.
    D. J. Watts and S. H. Strogatz, “Collective dynamics of ‘small-world’ networks,” Nature, vol. 393, pp. 440–442, June 1998.CrossRefGoogle Scholar
  13. 13.
    E. Ravasz and A. L. Barabási, “Hierarchical organization in complex networks,” Physical Review. E, Statistical, Nonlinear, and Soft Matter Physics, vol. 67, February 2003.Google Scholar
  14. 14.
    E. Ravasz, A. L. Somera, D. A. Mongru, Z. N. Oltvai, and A. L. Barabási, “Hierarchical organization of modularity in metabolic networks,” Science, vol. 297, no. 5586, pp. 1551–1555, 2002.CrossRefGoogle Scholar
  15. 15.
    M. Newman, “A measure of betweenness centrality based on random walks,” Social Networks, vol. 27, pp. 39–54, January 2005.CrossRefGoogle Scholar
  16. 16.
    G. Sabidussi, “The centrality index of a graph,” Psychometrika, vol. 31, pp. 581–603, December 1966.zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    L. C. Freeman, “Centrality in social networks: Conceptual clarification,” Social Networks, vol. 1, no. 3, pp. 215–239, 1979.CrossRefGoogle Scholar
  18. 18.
    P. Cano, O. Celma, M. Koppenberger, and J. Martin-Buldú, “Topology of music recommendation networks,” Chaos: An Interdisciplinary Journal of Nonlinear Science, vol. 16, no. 013107, 2006.Google Scholar
  19. 19.
    J. Martin-Buldú, P. Cano, M. Koppenberger, J. Almendral, and S. Boccaletti, “The complex network of musical tastes,” New Journal of Physics, vol. 9, no. 172, 2007.Google Scholar
  20. 20.
    J. Park, O. Celma, M. Koppenberger, P. Cano, and J. Martin-Buldú, “The social network of contemporary popular musicians,” International Journal of Bifurcation and Chaos, vol. 17, no. 7, pp. 2281–2288, 2007.zbMATHCrossRefGoogle Scholar
  21. 21.
    A. Anglade, M. Tiemann, and F. Vignoli, “Complex-network theoretic clustering for identifying groups of similar listeners in p2p systems,” in Proceedings of the ACM conference on Recommender systems, (Minneapolis, MN), pp. 41–48, ACM, 2007.Google Scholar
  22. 22.
    J.-J. Aucouturier and F. Pachet, “A scale-free distribution of false positives for a large class of audio similarity measures,” Pattern Recognition, vol. 41, no. 1, pp. 272–284, 2008.zbMATHCrossRefGoogle Scholar
  23. 23.
    K. Jacobson and M. Sandler, “Musically meaningful or just noise? an analysis of on-line artist networks,” in Proceedings of the 6th International Symposium on Computer Music Modeling and Retrieval, (Copenhagen, Denmark), 2008.Google Scholar
  24. 24.
    R. Lambiotte and M. Ausloos, “Uncovering collective listening habits and music genres in bipartite networks,” Physical Review E, vol. 72, 2005.Google Scholar

Copyright information

© Springer Berlin Heidelberg 2010

Authors and Affiliations

  1. 1.BMATBarcelonaSpain

Personalised recommendations