Making recommendations using transfer learning

Neural Computing and Applications (2021)Cite this article

Abstract

Deep learning-based recommender systems have gained much attention due to the advantage of encoding content-based information, such as user textual reviews and item descriptions, images, or videos, without the trouble of manually crafting feature vectors. However, those systems are trained from scratch with randomly initialized parameters, where the training process can take a long time to converge. With the most recent breakthroughs in Natural Language Processing using transfer learning, pre-trained transformer-based models now provide a better foundation for textual information encoding. This inspires us to propose a transformer-based recommender system using transfer learning. As the first core contribution in this work, we apply transfer learning to the system, by fine-tuning the pre-trained transformer models for information encoding. The experiment result shows that the proposed system outperforms several other deep learning-based recommender systems on multiple datasets. As the second core contribution, we propose a novel user vector encoding algorithm that assists all the models to achieve a better performance, when the user content information is not available.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

US$ 39.95

Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. 1.

    Alashkar T, Jiang S, Wang S, Fu Y (2017) Examples-rules guided deep neural network for makeup recommendation. In: Thirty-first AAAI conference on artificial intelligence

  2. 2.

    Bansal T, Belanger D, McCallum A (2016) Ask the GRU: multi-task learning for deep text recommendations. In: Proceedings of the 10th ACM conference on recommender systems, pp 107–114

  3. 3.

    Chen C, Zhao P, Li L, Zhou J, Li X, Qiu M (2017) Locally connected deep learning framework for industrial-scale recommender systems. In: Proceedings of the 26th international conference on World Wide Web companion, pp 769–770

  4. 4.

    Cheng HT, Koc L, Harmsen J, Shaked T, Chandra T, Aradhye H, Anderson G, Corrado G, Chai W, Ispir M et al (2016) Wide & deep learning for recommender systems. In: Proceedings of the 1st workshop on deep learning for recommender systems, pp 7–10

  5. 5.

    Chu WT, Tsai YL (2017) A hybrid recommendation system considering visual information for predicting favorite restaurants. World Wide Web 20(6):1313–1331

    Article  Google Scholar 

  6. 6.

    Covington P, Adams J, Sargin E (2016) Deep neural networks for Youtube recommendations. In: Proceedings of the 10th ACM conference on recommender systems, pp 191–198

  7. 7.

    Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  8. 8.

    Dziugaite GK, Roy DM (2015) Neural network matrix factorization. CoRR arXiv:1511.06443

  9. 9.

    Elkahky AM, Song Y, He X (2015) A multi-view deep learning approach for cross domain user modeling in recommendation systems. In: Proceedings of the 24th international conference on World Wide Web, pp 278–288

  10. 10.

    Fang X, Tao J (2019) A transfer learning based approach for aspect based sentiment analysis. In: 2019 sixth international conference on Social Networks Analysis, Management and Security (SNAMS), pp 478–483

  11. 11.

    Fang X, Xu M, Xu S, Zhao P (2019) A deep learning framework for predicting cyber attacks rates. EURASIP J Inf Secur 2019(1):5

    Article  Google Scholar 

  12. 12.

    Fang X, Yuan Z (2019) Performance enhancing techniques for deep learning models in time series forecasting. Eng Appl Artif Intell 85:533–542

    Article  Google Scholar 

  13. 13.

    Fang X, Zhan J (2015) Sentiment analysis using product review data. J Big Data 2(1):5

    Article  Google Scholar 

  14. 14.

    Guo H, Tang R, Ye Y, Li Z, He X (2017) Deepfm: a factorization-machine based neural network for CTR prediction. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia, pp 1725–1731

  15. 15.

    He R, McAuley J (2016) VBPR: visual Bayesian personalized ranking from implicit feedback. In: Thirtieth AAAI conference on artificial intelligence

  16. 16.

    Hidasi B, Karatzoglou A, Baltrunas L, Tikk D (2015) Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939

  17. 17.

    Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. In: Proceedings of the 56th annual meeting of the Association for Computational Linguistics (volume 1: long papers), vol 1, pp 328–339

  18. 18.

    Hu M, Liu B (2004) Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 168–177

  19. 19.

    Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2261–2269

  20. 20.

    Imdb api. https://github.com/jnwatson/py-lmdb

  21. 21.

    Imdb datasets. https://www.imdb.com/interfaces/

  22. 22.

    Kim D, Park C, Oh J, Lee S, Yu H (2016) Convolutional matrix factorization for document context-aware recommendation. In: Proceedings of the 10th ACM conference on recommender systems, pp 233–240

  23. 23.

    Kim D, Park C, Oh J, Yu H (2017) Deep hybrid recommender systems via exploiting document context and statistics of items. Inf Sci 417:72–87

    Article  Google Scholar 

  24. 24.

    Li S, Kawale J, Fu Y (2015) Deep collaborative filtering via marginalized denoising auto-encoder. In: Proceedings of the 24th ACM international on Conference on Information and Knowledge Management, CIKM ’15. Association for Computing Machinery, New York, pp 811–820. https://doi.org/10.1145/2806416.2806527

  25. 25.

    Ling G, Lyu MR, King I (2014) Ratings meet reviews, a combined approach to recommend. In: Proceedings of the 8th ACM conference on recommender systems, pp 105–112

  26. 26.

    Long M, Cao Y, Wang J, Jordan MI (2015) Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd international conference on international conference on machine learning, vol 37, pp 97–105. JMLR.org

  27. 27.

    Malkiel I, Barkan O, Caciularu A, Razin N, Katz O, Koenigstein N (2020) RecoBERT: a catalog language model for text-based recommendations. Findings of the Association for Computational Linguistics: EMNLP, pp 1704–1714

  28. 28.

    Merity S, Xiong C, Bradbury J, Socher R (2017) Pointer sentinel mixture models. In: Proceedings of the international conference on learning representations

  29. 29.

    Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119

  30. 30.

    Movielens dataset. https://grouplens.org/datasets/movielens/

  31. 31.

    Nguyen HT, Wistuba M, Grabocka J, Drumond LR, Schmidt-Thieme L (2017) Personalized deep learning for tag recommendation. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, pp 186–197

  32. 32.

    Okura S, Tagami Y, Ono S, Tajima A (2017) Embedding-based news recommendation for millions of users. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1933–1942

  33. 33.

    Purushotham S, Liu Y, Kuo CCJ (2012) Collaborative topic regression with social matrix factorization for recommendation systems. In: Proceedings of the 29th international conference on machine learning, Edinburgh, Scotland, UK

  34. 34.

    Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. Technical report, OpenAI

  35. 35.

    Řehůřek R, Sojka P (2010) Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks. ELRA, Valletta, Malta, pp 45–50

  36. 36.

    Sun F, Liu J, Wu J, Pei C, Lin X, Ou W, Jiang P (2019) BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM international conference on information and knowledge management, Beijing, China, pp 1441–1450

  37. 37.

    Tao J, Fang X (2020) Toward multi-label sentiment analysis: a transfer learning based approach. J Big Data 7(1):1–26

    Article  Google Scholar 

  38. 38.

    Tf-idf weighting. https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html

  39. 39.

    Vartak M, Thiagarajan A, Miranda C, Bratman J, Larochelle H (2017) A meta-learning perspective on cold-start recommendations for items. In: Advances in neural information processing systems, pp 6904–6914

  40. 40.

    Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  41. 41.

    Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR (2018) Glue: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP workshop BlackboxNLP: analyzing and interpreting neural networks for NLP, Brussels, Belgium, pp 353–355

  42. 42.

    Wang T, Fu Y (2020) Item-based collaborative filtering with BERT. In: Proceedings of the 3rd workshop on e-Commerce and NLP, Seattle, WA, USA, pp 54–58

  43. 43.

    Wang H, Wang N, Yeung DY (2015) Collaborative deep learning for recommender systems. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1235–1244

  44. 44.

    Wu C, Wang J, Liu J, Liu W (2016) Recurrent neural network based recommendation for time heterogeneous feedback. Knowl-Based Syst 109:90–103

    Article  Google Scholar 

  45. 45.

    Xie R, Liu Z, Yan R, Sun M (2016) Neural emoji recommendation in dialogue systems. arXiv preprint arXiv:1612.04609

  46. 46.

    Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNet: generalized autoregressive pretraining for language understanding, pp 1–18. arXiv:1906.08237

  47. 47.

    Yu W, Zhang H, He X, Chen X, Xiong L, Qin Z (2018) Aesthetic-based clothing recommendation. In: Proceedings of the 2018 World Wide Web conference, pp 649–658

  48. 48.

    Zhang S, Yao L, Sun A, Tay Y (2019) Deep learning based recommender system: a survey and new perspectives. ACM Comput Surv (CSUR) 52(1):1–38

    Google Scholar 

  49. 49.

    Zheng L, Lu CT, He L, Xie S, He H, Li C, Noroozi V, Dong B, Philip SY (2019) Mars: memory attention-aware recommender system. In: 2019 IEEE international conference on Data Science and Advanced Analytics (DSAA). IEEE, pp 11–20

Download references

Author information

Affiliations

  1. School of Information Technology, Illinois State University, Normal, IL, USA

    Xing Fang

Authors
  1. Xing Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xing Fang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fang, X. Making recommendations using transfer learning. Neural Comput & Applic (2021). https://doi.org/10.1007/s00521-021-05730-3

Download citation

Keywords

  • Recommender system
  • Deep learning
  • Transfer learning
  • User vector embedding