Recommender Systems in Industry: A Netflix Case Study

Abstract

The Netflix Prize put a spotlight on the importance and use of recommender systems in real-world applications. Many the competition provided many lessons about how to approach recommendation and many more have been learned since the Grand Prize was awarded in 2009. The evolution of industrial applications of recommender systems has been driven by the availability of different kinds of user data and the level of interest for the area within the research community. The goal of this chapter is to give an up-to-date overview of recommender systems techniques used in an industrial setting. We will give a high-level description the practical use of recommendation and personalization techniques. We will highlight some of the main lessons learned from the Netflix Prize. We will then use Netflix personalization as a case study to describe several approaches and techniques used in a real-world recommendation system. Finally, we will pinpoint what we see as some promising current research avenues and unsolved problems that deserve attention in this domain from an industry perspective.

References

  1. 1.
    Agarwal, D., Chen, B.C., Elango, P., Ramakrishnan, R.: Content recommendation on web portals. Commun. ACM 56(6), 92–101 (2013). DOI 10.1145/2461256.2461277. URL http://doi.acm.org/10.1145/2461256.2461277
  2. 2.
    Agarwal, D., Chen, B.C., Pang, B.: Personalized recommendation of user comments via factor models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ‘11, pp. 571–582. Association for Computational Linguistics, Stroudsburg, PA, USA (2011). URL http://dl.acm.org/citation.cfm?id=2145432.2145499
  3. 3.
    Ahmed, A., Teo, C.H., Vishwanathan, S., Smola, A.: Fair and balanced: Learning to present news stories. In: Proceedings of the Fifth ACM International Conference on Web Search and Data Mining, WSDM ‘12, pp. 333–342. ACM, New York, NY, USA (2012). DOI 10.1145/2124295.2124337. URL http://doi.acm.org/10.1145/2124295.2124337
  4. 4.
    Amatriain, X., Lathia, N., Pujol, J.M., Kwak, H., Oliver, N.: The wisdom of the few: a collaborative filtering approach based on expert opinions from the web. In: Proc. of 32nd ACM SIGIR, SIGIR ‘09, pp. 532–539. ACM, New York, NY, USA (2009). DOI 10.1145/1571941.1572033. URL http://dx.doi.org/10.1145/1571941.1572033
  5. 5.
    Amatriain, X., Pujol, J.M., Oliver, N.: I Like It…I Like It Not: Evaluating User Ratings Noise in Recommender Systems. In: G.J. Houben, G. McCalla, F. Pianesi, M. Zancanaro (eds.) User Modeling, Adaptation, and Personalization, vol. 5535, chap. 24, pp. 247–258. Springer Berlin (2009). DOI 10.1007/978-3-642-02247-0_24. URL http://dx.doi.org/10.1007/978-3-642-02247-0_24
  6. 6.
    Andersen, R., Borgs, C., Chayes, J., Feige, U., Flaxman, A., Kalai, A., Mirrokni, V., Tennenholtz, M.: Trust-based recommendation systems: an axiomatic approach. In: Proc. of the 17th WWW, WWW ‘08, pp. 199–208. ACM, New York, NY, USA (2008). DOI 10.1145/1367497.1367525. URL http://doi.acm.org/10.1145/1367497.1367525
  7. 7.
    Basu, C., Hirsh, H., Cohen, W.: Recommendation as classification: using social and content-based information in recommendation. In: Proc. of AAAI ‘98, AAAI ‘98/IAAI ‘98, pp. 714–720. American Association for Artificial Intelligence, Menlo Park, CA, USA (1998). URL http://dl.acm.org/citation.cfm?id=295240.295795
  8. 8.
    Bell, R.M., Koren, Y.: Lessons from the Netflix Prize Challenge. SIGKDD Explor. Newsl. 9(2), 75–79 (2007). DOI 10.1145/1345448.1345465. URL http://dx.doi.org/10.1145/1345448.1345465
  9. 9.
    Berndhardsson, E.: Music recommendations at spotify (2013)Google Scholar
  10. 10.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003). URL http://dl.acm.org/citation.cfm?id=944919.944937
  11. 11.
    Bourke, S., McCarthy, K., Smyth, B.: Power to the people: exploring neighbourhood formations in social recommender system. In: Proc. of Recsys ‘11, RecSys ‘11, pp. 337–340. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043997. URL http://doi.acm.org/10.1145/2043932.2043997
  12. 12.
    Breiman, L.: Random forests. Machine learning 45(1), 5–32 (2001)CrossRefMATHGoogle Scholar
  13. 13.
    Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.: Learning to rank using gradient descent. In: Proceedings of the 22nd ICML, ICML ‘05, pp. 89–96. ACM, New York, NY, USA (2005). DOI 10.1145/1102351.1102363. URL http://dx.doi.org/10.1145/1102351.1102363
  14. 14.
    Burke, R.: The adaptive web. chap. Hybrid Web Recommender Systems, pp. 377–408 (2007). DOI 10.1007/978-3-540-72079-9_12. URL http://dx.doi.org/10.1007/978-3-540-72079-9_12
  15. 15.
    Cao, Z., Liu, T.: Learning to rank: From pairwise approach to listwise approach. In: In Proceedings of the 24th ICML, pp. 129–136 (2007). URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.64.1518
  16. 16.
    Celma, O.: Music Recommendation and Discovery: The Long Tail, Long Fail, and Long Play in the Digital Music Space. Springer (2010)Google Scholar
  17. 17.
    Chapelle, O., Keerthi, S.S.: Efficient algorithms for ranking with SVMs. Information Retrieval 13, 201–215 (2010). DOI 10.1007/s10791-009-9109-9. URL http://dx.doi.org/10.1007/s10791-009-9109-9
  18. 18.
    Chen, W.Y., Chu, J.C., Luan, J., Bai, H., Wang, Y., Chang, E.Y.: Collaborative filtering for orkut communities: Discovery of user latent behavior. In: Proceedings of the 18th International Conference on World Wide Web, WWW ‘09, pp. 681–690. ACM, New York, NY, USA (2009). DOI 10.1145/1526709.1526801. URL http://doi.acm.org/10.1145/1526709.1526801
  19. 19.
    Das, A.S., Datar, M., Garg, A., Rajaram, S.: Google news personalization: Scalable online collaborative filtering. In: Proceedings of the 16th International Conference on World Wide Web, WWW ‘07, pp. 271–280. ACM, New York, NY, USA (2007). DOI 10.1145/1242572.1242610. URL http://doi.acm.org/10.1145/1242572.1242610
  20. 20.
    Davidson, J., Liebald, B., Liu, J., Nandy, P., Van Vleet, T., Gargi, U., Gupta, S., He, Y., Lambert, M., Livingston, B., Sampath, D.: The youtube video recommendation system. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ‘10, pp. 293–296. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864770. URL http://doi.acm.org/10.1145/1864708.1864770
  21. 21.
    Diaz-Aviles, E., Georgescu, M., Nejdl, W.: Swarming to rank for recommender systems. In: Proc. of Recsys ‘12, RecSys ‘12, pp. 229–232. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2366001. URL http://doi.acm.org/10.1145/2365952.2366001
  22. 22.
    Elkan, C., Noto, K.: Learning classifiers from only positive and unlabeled data. In: Proc. of the 14th ACM SIGKDD, KDD ‘08, pp. 213–220. ACM, New York, NY, USA (2008). DOI 10.1145/1401890.1401920. URL http://dx.doi.org/10.1145/1401890.1401920
  23. 23.
    Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An efficient boosting algorithm for combining preferences. J. Mach. Learn. Res. 4, 933–969 (2003). URL http://portal.acm.org/citation.cfm?id=964285
  24. 24.
    Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 2007 (2007)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Annals of Statistics pp. 1189–1232 (2001)Google Scholar
  26. 26.
    Fujiwara, Y., Nakatsuji, M., Yamamuro, T., Shiokawa, H., Onizuka, M.: Efficient personalized pagerank with accuracy assurance. In: Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘12, pp. 15–23. ACM, New York, NY, USA (2012). DOI 10.1145/2339530.2339538. URL http://doi.acm.org/10.1145/2339530.2339538
  27. 27.
    Funk, S.: Netflix update: Try this at home. http://sifter.org/ simon/journal/20061211.html (2006). URL http://sifter.org/~simon/journal/20061211.html
  28. 28.
    Gorgoglione, M., Panniello, U., Tuzhilin, A.: The effect of context-aware recommendations on customer purchasing behavior and trust. In: Proc. of Recsys ‘11, RecSys ‘11, pp. 85–92. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043951. URL http://doi.acm.org/10.1145/2043932.2043951
  29. 29.
    Gupta, P., Goel, A., Lin, J., Sharma, A., Wang, D., Zadeh, R.: Wtf: The who to follow service at twitter. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 505–514. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488433
  30. 30.
    Herlocker, J.L., Konstan, J.A., Terveen, L.G., Riedl, J.T.: Evaluating collaborative filtering recommender systems. ACM Trans. Inf. Syst. 22(1), 5–53 (2004). DOI http://doi.acm.org/10.1145/963770.963772Google Scholar
  31. 31.
    Hu, Y., Koren, Y., Volinsky, C.: Collaborative Filtering for Implicit Feedback Datasets. In: Proc. of the 2008 Eighth ICDM, ICDM ‘08, vol. 0, pp. 263–272. IEEE Computer Society, Washington, DC, USA (2008). DOI 10.1109/ICDM.2008.22. URL http://dx.doi.org/10.1109/ICDM.2008.22
  32. 32.
    in the Industry, R.S.: Recommendation systems in the industry. Tutorial at Recsys 2009 (2009)Google Scholar
  33. 33.
    Jamali, M., Ester, M.: Trustwalker: a random walk model for combining trust-based and item-based recommendation. In: Proc. of KDD ‘09, KDD ‘09, pp. 397–406. ACM, New York, NY, USA (2009). DOI 10.1145/1557019.1557067. URL http://doi.acm.org/10.1145/1557019.1557067
  34. 34.
    Jeh, G., Widom, J.: Simrank: A measure of structural-context similarity. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘02, pp. 538–543. ACM, New York, NY, USA (2002). DOI 10.1145/775047.775126. URL http://doi.acm.org/10.1145/775047.775126
  35. 35.
    Karatzoglou, A., Amatriain, X., Baltrunas, L., Oliver, N.: Multiverse recommendation: n-dimensional tensor factorization for context-aware collaborative filtering. In: Proc. of the fourth ACM Recsys, RecSys ‘10, pp. 79–86. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864727. URL http://dx.doi.org/10.1145/1864708.1864727
  36. 36.
    Karimzadehgan, M., Li, W., Zhang, R., Mao, J.: A stochastic learning-to-rank algorithm and its application to contextual advertising. In: Proceedings of the 20th WWW, WWW ‘11, pp. 377–386. ACM, New York, NY, USA (2011). DOI 10.1145/1963405.1963460. URL http://doi.acm.org/10.1145/1963405.1963460
  37. 37.
    Karypis, G.: Evaluation of item-based top-n recommendation algorithms. In: CIKM ‘01: Proceedings of the tenth international conference on Information and knowledge management, pp. 247–254. ACM, New York, NY, USA (2001). DOI http://doi.acm.org/10.1145/ 502585.502627Google Scholar
  38. 38.
    Knijnenburg, B.P.: Conducting user experiments in recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 3–4. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365956. URL http://doi.acm.org/10.1145/2365952.2365956
  39. 39.
    Koenigstein, N., Nice, N., Paquet, U., Schleyen, N.: The xbox recommender system. In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ‘12, pp. 281–284. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2366015. URL http://doi.acm.org/10.1145/2365952.2366015
  40. 40.
    Kohavi, R., Deng, A., Frasca, B., Longbotham, R., Walker, T., Xu, Y.: Trustworthy online controlled experiments: five puzzling outcomes explained. In: Proceedings of KDD ‘12, pp. 786–794. ACM, New York, NY, USA (2012). DOI 10.1145/2339530.2339653. URL http://doi.acm.org/10.1145/2339530.2339653
  41. 41.
    Kohavi, R., Henne, R.M., Sommerfield, D.: Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. In: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘07, pp. 959–967. ACM, New York, NY, USA (2007). DOI 10.1145/1281192.1281295. URL http://doi.acm.org/10.1145/1281192.1281295
  42. 42.
    Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD, KDD ‘08, pp. 426–434. ACM, New York, NY, USA (2008). DOI 10.1145/1401890.1401944. URL http://dx.doi.org/10.1145/1401890.1401944
  43. 43.
    Koren, Y.: Collaborative filtering with temporal dynamics. In: Proceedings of the 15th ACM SIGKDD, KDD ‘09, pp. 447–456. ACM, New York, NY, USA (2009). DOI 10.1145/1557019.1557072. URL http://dx.doi.org/10.1145/1557019.1557072
  44. 44.
    Koren, Y., Bell, R., Volinsky, C.: Matrix Factorization Techniques for Recommender Systems. Computer 42(8), 30–37 (2009). DOI 10.1109/MC.2009.263. URL http://dx.doi.org/10.1109/MC.2009.263
  45. 45.
    Lagun, D., Hsieh, C.H., Webster, D., Navalpakkam, V.: Towards better measurement of attention and satisfaction in mobile search. In: Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ‘14, pp. 113–122. ACM, New York, NY, USA (2014). DOI 10.1145/2600428.2609631. URL http://doi.acm.org/10.1145/2600428.2609631
  46. 46.
    Lamere, P.B.: I’ve got 10 million songs in my pocket: Now what? In: Proceedings of the Sixth ACM Conference on Recommender Systems, RecSys ‘12, pp. 207–208. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365994. URL http://doi.acm.org/10.1145/2365952.2365994
  47. 47.
    Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web, WWW ‘10, pp. 661–670. ACM, New York, NY, USA (2010). DOI 10.1145/1772690.1772758. URL http://doi.acm.org/10.1145/1772690.1772758
  48. 48.
    Linden, G., Smith, B., York, J.: Amazon.com recommendations: Item-to-item collaborative filtering. IEEE Internet Computing 7(1), 76–80 (2003). DOI 10.1109/MIC.2003.1167344. URL http://dx.doi.org/10.1109/MIC.2003.1167344
  49. 49.
    Liu, J., Pedersen, E., Dolan, P.: Personalized news recommendation based on click behavior. In: 2010 International Conference on Intelligent User Interfaces (2010)Google Scholar
  50. 50.
    Liu, N.N., Meng, X., Liu, C., Yang, Q.: Wisdom of the better few: cold start recommendation via representative based rating elicitation. In: Proc. of RecSys ‘11, RecSys ‘11. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043943. URL http://doi.acm.org/10.1145/2043932.2043943
  51. 51.
    McLaughlin, M.R., Herlocker, J.L.: A collaborative filtering algorithm and evaluation metric that accurately model the user experience. In: Proc. of SIGIR ‘04 (2004)Google Scholar
  52. 52.
    Mcnee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI ‘06: CHI ‘06 extended abstracts on Human factors in computing systems, pp. 1097–1101. ACM Press, New York, NY, USA (2006). DOI 10.1145/ 1125451.1125659Google Scholar
  53. 53.
    Navalpakkam, V., Jentzsch, L., Sayres, R., Ravi, S., Ahmed, A., Smola, A.: Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 953–964. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488471
  54. 54.
    Navalpakkam, V., Jentzsch, L., Sayres, R., Ravi, S., Ahmed, A., Smola, A.: Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 953–964. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488471
  55. 55.
    Ning, X., Karypis, G.: Sparse linear methods with side information for top-n recommendations. In: Proc. of the 21st WWW, WWW ‘12 Companion, pp. 581–582. ACM, New York, NY, USA (2012). DOI 10.1145/2187980.2188137. URL http://doi.acm.org/10.1145/2187980.2188137
  56. 56.
    Noel, J., Sanner, S., Tran, K., Christen, P., Xie, L., Bonilla, E.V., Abbasnejad, E., Della Penna, N.: New objective functions for social collaborative filtering. In: Proc. of WWW ‘12, WWW ‘12, pp. 859–868. ACM, New York, NY, USA (2012). DOI 10.1145/2187836.2187952. URL http://doi.acm.org/10.1145/2187836.2187952
  57. 57.
    O’Donovan, J., Smyth, B.: Trust in recommender systems. In: Proc. of IUI ‘05, IUI ‘05, pp. 167–174. ACM, New York, NY, USA (2005). DOI 10.1145/1040830.1040870. URL http://doi.acm.org/10.1145/1040830.1040870
  58. 58.
    Oku, K., Nakajima, S., Miyazaki, J., Uemura, S.: Context-aware SVM for context-dependent information recommendation. In: Proc. of the 7th Conference on Mobile Data Management (2006)Google Scholar
  59. 59.
    Parra, D., Amatriain, X.: Walk the Talk: Analyzing the relation between implicit and explicit feedback for preference elicitation. In: J.A. Konstan, R. Conejo, J.L. Marzo, N. Oliver (eds.) User Modeling, Adaption and Personalization, Lecture Notes in Computer Science, vol. 6787, chap. 22, pp. 255–268. Springer, Berlin, Heidelberg (2011). DOI 10.1007/978-3-642-22362-4_22. URL http://dx.doi.org/10.1007/978-3-642-22362-4\_22
  60. 60.
    Parra, D., Karatzoglou, A., Amatriain, X., Yavuz, I.: Implicit feedback recommendation via implicit-to-explicit ordinal logistic regression mapping. In: Proc. of the 2011 CARS Workshop (2011)Google Scholar
  61. 61.
    Pizzato, L., Rej, T., Chung, T., Koprinska, I., Kay, J.: Recon: A reciprocal recommender for online dating. In: Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ‘10, pp. 207–214. ACM, New York, NY, USA (2010). DOI 10.1145/1864708.1864747. URL http://doi.acm.org/10.1145/1864708.1864747
  62. 62.
    Rabkin, A., Katz, R.: Chukwa: A system for reliable large-scale log collection. In: Proceedings of the 24th International Conference on Large Installation System Administration, LISA’10, pp. 1–15. USENIX Association, Berkeley, CA, USA (2010). URL http://dl.acm.org/citation.cfm?id=1924976.1924994
  63. 63.
    Radlinski, F., Kurup, M., Joachims, T.: How does clickthrough data reflect retrieval quality? In: Proc. of the 17th CIKM, CIKM ‘08, pp. 43–52. ACM, New York, NY, USA (2008). DOI 10.1145/1458082.1458092. URL http://dx.doi.org/10.1145/1458082.1458092
  64. 64.
    Reda, A., Park, Y., Tiwari, M., Posse, C., Shah, S.: Metaphor: A system for related search recommendations. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, CIKM ‘12, pp. 664–673. ACM, New York, NY, USA (2012). DOI 10.1145/2396761.2396847. URL http://doi.acm.org/10.1145/2396761.2396847
  65. 65.
    Rendle, S.: Factorization Machines. In: Proc. of 2010 IEEE ICDM, pp. 995–1000. IEEE (2010). DOI 10.1109/ICDM.2010.127. URL http://dx.doi.org/10.1109/ICDM.2010.127
  66. 66.
    Rendle, S., Freudenthaler, C., Gantner, Z., Thieme, L.S.: BPR: Bayesian personalized ranking from implicit feedback. In: Proceedings of the 25th UAI, UAI ‘09, pp. 452–461. AUAI Press, Arlington, Virginia, United States (2009). URL http://portal.acm.org/citation.cfm?id=1795167
  67. 67.
    Rendle, S., Freudenthaler, C., Thieme, L.S.: Factorizing personalized Markov chains for next-basket recommendation. In: Proc. of the 19th WWW, WWW ‘10, pp. 811–820. ACM, New York, NY, USA (2010). DOI 10.1145/1772690.1772773. URL http://dx.doi.org/10.1145/1772690.1772773
  68. 68.
    Rendle, S., Gantner, Z., Freudenthaler, C., Schmidt-Thieme, L.: Fast context-aware recommendations with factorization machines. In: Proc. of the 34th ACM SIGIR, SIGIR ‘11, pp. 635–644. ACM, New York, NY, USA (2011). DOI 10.1145/2009916.2010002. URL http://doi.acm.org/10.1145/2009916.2010002
  69. 69.
    Ribeiro, M.T., Lacerda, A., Veloso, A., Ziviani, N.: Pareto-efficient hybridization for multi-objective recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 19–26. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365962. URL http://doi.acm.org/10.1145/2365952.2365962
  70. 70.
    Rodriguez, M., Posse, C., Zhang, E.: Multiple objective optimization in recommender systems. In: Proceedings of the sixth ACM conference on Recommender systems, RecSys ‘12, pp. 11–18. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365961. URL http://doi.acm.org/10.1145/2365952.2365961
  71. 71.
    Salakhutdinov, R., Mnih, A., Hinton, G.E.: Restricted Boltzmann machines for collaborative filtering. In: Proc of ICML ‘07. ACM, New York, NY, USA (2007)Google Scholar
  72. 72.
    Science: Rockin’ to the Music Genome. Science 311(5765), 1223d– (2006). DOI 10.1126/science.311.5765.1223d. URL http://www.sciencemag.org
  73. 73.
    Sha, X., Quercia, D., Michiardi, P., Dell’Amico, M.: Spotting trends: the wisdom of the few. In: Proc. of the Recsys ‘12, RecSys ‘12, pp. 51–58. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365967. URL http://doi.acm.org/10.1145/2365952.2365967
  74. 74.
    Shardanand, U., Maes, P.: Social information filtering: algorithms for automating word of mouth. In: Proc. of SIGCHI ‘95, CHI ‘95, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., New York, NY, USA (1995). DOI 10.1145/223904.223931. URL http://dx.doi.org/10.1145/223904.223931
  75. 75.
    Shi, Y., Karatzoglou, A., Baltrunas, L., Larson, M., Hanjalic, A., Oliver, N.: TFMAP: optimizing MAP for top-n context-aware recommendation. In: Proc. of the 35th SIGIR, SIGIR ‘12, pp. 155–164. ACM, New York, NY, USA (2012). DOI 10.1145/2348283.2348308. URL http://doi.acm.org/10.1145/2348283.2348308
  76. 76.
    Shi, Y., Karatzoglou, A., Baltrunas, L., Larson, M., Oliver, N., Hanjalic, A.: CLiMF: learning to maximize reciprocal rank with collaborative less-is-more filtering. In: Proc. of the sixth Recsys, RecSys ‘12, pp. 139–146. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365981. URL http://dx.doi.org/10.1145/2365952.2365981
  77. 77.
    Sigurbjörnsson, B., van Zwol, R.: Flickr tag recommendation based on collective knowledge. In: Proceedings of the 17th International Conference on World Wide Web, WWW ‘08, pp. 327–336. ACM, New York, NY, USA (2008). DOI 10.1145/1367497.1367542. URL http://doi.acm.org/10.1145/1367497.1367542
  78. 78.
    Steck, H.: Training and testing of recommender systems on data missing not at random. In: Proc. of the 16th ACM SIGKDD, KDD ‘10, pp. 713–722. ACM, New York, NY, USA (2010). DOI 10.1145/1835804.1835895. URL http://dx.doi.org/10.1145/1835804.1835895
  79. 79.
    Steck, H.: Item popularity and recommendation accuracy. In: Proceedings of the fifth ACM conference on Recommender systems, RecSys ‘11, pp. 125–132. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043957. URL http://doi.acm.org/10.1145/2043932.2043957
  80. 80.
    Steck, H.: Evaluation of recommendations: Rating-prediction and ranking. In: Proceedings of the 7th ACM Conference on Recommender Systems, RecSys ‘13, pp. 213–220. ACM, New York, NY, USA (2013). DOI 10.1145/2507157.2507160. URL http://doi.acm.org/10.1145/2507157.2507160
  81. 81.
    Stern, D.H., Herbrich, R., Graepel, T.: Matchbox: large scale online bayesian recommendations. In: Proc. of the 18th WWW, WWW ‘09, pp. 111–120. ACM, New York, NY, USA (2009). DOI 10.1145/1526709.1526725. URL http://dx.doi.org/10.1145/1526709.1526725
  82. 82.
    Takács, G., Pilászy, I., Németh, B., Tikk, D.: Major components of the gravity recommendation system. SIGKDD Explor. Newsl. 9(2), 80–83 (2007). DOI 10.1145/1345448.1345466. URL http://doi.acm.org/10.1145/1345448.1345466
  83. 83.
    Takács, G., Tikk, D.: Alternating least squares for personalized ranking. In: Proc. of Recsys ‘12, RecSys ‘12, pp. 83–90. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365972. URL http://doi.acm.org/10.1145/2365952.2365972
  84. 84.
    Tan, M., Xia, T., Guo, L., Wang, S.: Direct optimization of ranking measures for learning to rank models. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘13, pp. 856–864. ACM, New York, NY, USA (2013). DOI 10.1145/2487575.2487630. URL http://doi.acm.org/10.1145/2487575.2487630
  85. 85.
    Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical dirichlet processes. Journal of the American Statistical Association 101 (2004)Google Scholar
  86. 86.
    Valizadegan, H., Jin, R., Zhang, R., Mao, J.: Learning to Rank by Optimizing NDCG Measure. In: Proc. of SIGIR ‘00, pp. 41–48 (2000). URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.154.8402
  87. 87.
    Vargas, S., Castells, P.: Rank and relevance in novelty and diversity metrics for recommender systems. In: Proceedings of the fifth ACM conference on Recommender systems, RecSys ‘11, pp. 109–116. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043955. URL http://doi.acm.org/10.1145/2043932.2043955
  88. 88.
    Wang, J., Sarwar, B., Sundaresan, N.: Utilizing related products for post-purchase recommendation in e-commerce. In: Proceedings of the Fifth ACM Conference on Recommender Systems, RecSys ‘11, pp. 329–332. ACM, New York, NY, USA (2011). DOI 10.1145/2043932.2043995. URL http://doi.acm.org/10.1145/2043932.2043995
  89. 89.
    Wang, J., Zhang, Y., Posse, C., Bhasin, A.: Is it time for a career switch? In: Proceedings of the 22Nd International Conference on World Wide Web, WWW ‘13, pp. 1377–1388. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland (2013). URL http://dl.acm.org/citation.cfm?id=2488388.2488509
  90. 90.
    Weston, J., Yee, H., Weiss, R.: Learning to rank recommendations with the k-order statistic loss. In: ACM International Conference on Recommender Systems (RecSys) (2013). URL http://dl.acm.org/citation.cfm?id=2507210
  91. 91.
    Xia, F., Liu, T.Y., Wang J.and Zhang, W., Li, H.: Listwise approach to learning to rank: theory and algorithm. In: Proc. of the 25th ICML, ICML ‘08, pp. 1192–1199. ACM, New York, NY, USA (2008). DOI 10.1145/1390156.1390306. URL http://dx.doi.org/10.1145/1390156.1390306
  92. 92.
    Xiong, L., Chen, X., Huang, T., J. Schneider, J.G.C.: Temporal collaborative filtering with bayesian probabilistic tensor factorization. In: Proceedings of SIAM Data Mining (2010)Google Scholar
  93. 93.
    Xu, J., Li, H.: AdaRank: a boosting algorithm for information retrieval. In: Proc. of SIGIR ‘07, SIGIR ‘07, pp. 391–398. ACM, New York, NY, USA (2007). DOI 10.1145/1277741.1277809. URL http://dx.doi.org/10.1145/1277741.1277809
  94. 94.
    Xu, J., Liu, T.Y., Lu, M., Li, H., Ma, W.Y.: Directly optimizing evaluation measures in learning to rank. In: Proc. of SIGIR ‘08, pp. 107–114. ACM, New York, NY, USA (2008). DOI 10.1145/1390334.1390355. URL http://dx.doi.org/10.1145/1390334.1390355
  95. 95.
    Y, K., Sill, J.: OrdRec: an ordinal model for predicting personalized item rating distributions. In: RecSys ‘11, pp. 117–124 (2011)Google Scholar
  96. 96.
    Yang, S., Long, B., Smola, A., Zha, H., Zheng, Z.: Collaborative competitive filtering: learning recommender using context of user choice. In: Proc. of the 34th ACM SIGIR, SIGIR ‘11, pp. 295–304. ACM, New York, NY, USA (2011). DOI 10.1145/2009916.2009959. URL http://dx.doi.org/10.1145/2009916.2009959
  97. 97.
    Yang, X., Steck, H., Guo, Y., Liu, Y.: On top-k recommendation using social networks. In: Proc. of RecSys ‘12, RecSys ‘12, pp. 67–74. ACM, New York, NY, USA (2012). DOI 10.1145/2365952.2365969. URL http://doi.acm.org/10.1145/2365952.2365969
  98. 98.
    Yi, J., Chen, Y., Li, J., Sett, S., Yan, T.W.: Predictive model performance: Offline and online evaluations. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ‘13, pp. 1294–1302. ACM, New York, NY, USA (2013). DOI 10.1145/2487575.2488215. URL http://doi.acm.org/10.1145/2487575.2488215

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.NetflixLos GatosUSA
  2. 2.QuoraMountain ViewUSA

Personalised recommendations