Deep learning-based recommender systems have gained much attention due to the advantage of encoding content-based information, such as user textual reviews and item descriptions, images, or videos, without the trouble of manually crafting feature vectors. However, those systems are trained from scratch with randomly initialized parameters, where the training process can take a long time to converge. With the most recent breakthroughs in Natural Language Processing using transfer learning, pre-trained transformer-based models now provide a better foundation for textual information encoding. This inspires us to propose a transformer-based recommender system using transfer learning. As the first core contribution in this work, we apply transfer learning to the system, by fine-tuning the pre-trained transformer models for information encoding. The experiment result shows that the proposed system outperforms several other deep learning-based recommender systems on multiple datasets. As the second core contribution, we propose a novel user vector encoding algorithm that assists all the models to achieve a better performance, when the user content information is not available.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Alashkar T, Jiang S, Wang S, Fu Y (2017) Examples-rules guided deep neural network for makeup recommendation. In: Thirty-first AAAI conference on artificial intelligence
Bansal T, Belanger D, McCallum A (2016) Ask the GRU: multi-task learning for deep text recommendations. In: Proceedings of the 10th ACM conference on recommender systems, pp 107–114
Chen C, Zhao P, Li L, Zhou J, Li X, Qiu M (2017) Locally connected deep learning framework for industrial-scale recommender systems. In: Proceedings of the 26th international conference on World Wide Web companion, pp 769–770
Cheng HT, Koc L, Harmsen J, Shaked T, Chandra T, Aradhye H, Anderson G, Corrado G, Chai W, Ispir M et al (2016) Wide & deep learning for recommender systems. In: Proceedings of the 1st workshop on deep learning for recommender systems, pp 7–10
Chu WT, Tsai YL (2017) A hybrid recommendation system considering visual information for predicting favorite restaurants. World Wide Web 20(6):1313–1331
Covington P, Adams J, Sargin E (2016) Deep neural networks for Youtube recommendations. In: Proceedings of the 10th ACM conference on recommender systems, pp 191–198
Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
Dziugaite GK, Roy DM (2015) Neural network matrix factorization. CoRR arXiv:1511.06443
Elkahky AM, Song Y, He X (2015) A multi-view deep learning approach for cross domain user modeling in recommendation systems. In: Proceedings of the 24th international conference on World Wide Web, pp 278–288
Fang X, Tao J (2019) A transfer learning based approach for aspect based sentiment analysis. In: 2019 sixth international conference on Social Networks Analysis, Management and Security (SNAMS), pp 478–483
Fang X, Xu M, Xu S, Zhao P (2019) A deep learning framework for predicting cyber attacks rates. EURASIP J Inf Secur 2019(1):5
Fang X, Yuan Z (2019) Performance enhancing techniques for deep learning models in time series forecasting. Eng Appl Artif Intell 85:533–542
Fang X, Zhan J (2015) Sentiment analysis using product review data. J Big Data 2(1):5
Guo H, Tang R, Ye Y, Li Z, He X (2017) Deepfm: a factorization-machine based neural network for CTR prediction. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia, pp 1725–1731
He R, McAuley J (2016) VBPR: visual Bayesian personalized ranking from implicit feedback. In: Thirtieth AAAI conference on artificial intelligence
Hidasi B, Karatzoglou A, Baltrunas L, Tikk D (2015) Session-based recommendations with recurrent neural networks. arXiv preprint arXiv:1511.06939
Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. In: Proceedings of the 56th annual meeting of the Association for Computational Linguistics (volume 1: long papers), vol 1, pp 328–339
Hu M, Liu B (2004) Mining and summarizing customer reviews. In: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 168–177
Huang G, Liu Z, van der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: 2017 IEEE conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp 2261–2269
Imdb api. https://github.com/jnwatson/py-lmdb
Imdb datasets. https://www.imdb.com/interfaces/
Kim D, Park C, Oh J, Lee S, Yu H (2016) Convolutional matrix factorization for document context-aware recommendation. In: Proceedings of the 10th ACM conference on recommender systems, pp 233–240
Kim D, Park C, Oh J, Yu H (2017) Deep hybrid recommender systems via exploiting document context and statistics of items. Inf Sci 417:72–87
Li S, Kawale J, Fu Y (2015) Deep collaborative filtering via marginalized denoising auto-encoder. In: Proceedings of the 24th ACM international on Conference on Information and Knowledge Management, CIKM ’15. Association for Computing Machinery, New York, pp 811–820. https://doi.org/10.1145/2806416.2806527
Ling G, Lyu MR, King I (2014) Ratings meet reviews, a combined approach to recommend. In: Proceedings of the 8th ACM conference on recommender systems, pp 105–112
Long M, Cao Y, Wang J, Jordan MI (2015) Learning transferable features with deep adaptation networks. In: Proceedings of the 32nd international conference on international conference on machine learning, vol 37, pp 97–105. JMLR.org
Malkiel I, Barkan O, Caciularu A, Razin N, Katz O, Koenigstein N (2020) RecoBERT: a catalog language model for text-based recommendations. Findings of the Association for Computational Linguistics: EMNLP, pp 1704–1714
Merity S, Xiong C, Bradbury J, Socher R (2017) Pointer sentinel mixture models. In: Proceedings of the international conference on learning representations
Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp 3111–3119
Movielens dataset. https://grouplens.org/datasets/movielens/
Nguyen HT, Wistuba M, Grabocka J, Drumond LR, Schmidt-Thieme L (2017) Personalized deep learning for tag recommendation. In: Pacific-Asia conference on knowledge discovery and data mining. Springer, pp 186–197
Okura S, Tagami Y, Ono S, Tajima A (2017) Embedding-based news recommendation for millions of users. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 1933–1942
Purushotham S, Liu Y, Kuo CCJ (2012) Collaborative topic regression with social matrix factorization for recommendation systems. In: Proceedings of the 29th international conference on machine learning, Edinburgh, Scotland, UK
Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training. Technical report, OpenAI
Řehůřek R, Sojka P (2010) Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 workshop on new challenges for NLP frameworks. ELRA, Valletta, Malta, pp 45–50
Sun F, Liu J, Wu J, Pei C, Lin X, Ou W, Jiang P (2019) BERT4Rec: sequential recommendation with bidirectional encoder representations from transformer. In: Proceedings of the 28th ACM international conference on information and knowledge management, Beijing, China, pp 1441–1450
Tao J, Fang X (2020) Toward multi-label sentiment analysis: a transfer learning based approach. J Big Data 7(1):1–26
Vartak M, Thiagarajan A, Miranda C, Bratman J, Larochelle H (2017) A meta-learning perspective on cold-start recommendations for items. In: Advances in neural information processing systems, pp 6904–6914
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Wang A, Singh A, Michael J, Hill F, Levy O, Bowman SR (2018) Glue: a multi-task benchmark and analysis platform for natural language understanding. In: Proceedings of the 2018 EMNLP workshop BlackboxNLP: analyzing and interpreting neural networks for NLP, Brussels, Belgium, pp 353–355
Wang T, Fu Y (2020) Item-based collaborative filtering with BERT. In: Proceedings of the 3rd workshop on e-Commerce and NLP, Seattle, WA, USA, pp 54–58
Wang H, Wang N, Yeung DY (2015) Collaborative deep learning for recommender systems. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp 1235–1244
Wu C, Wang J, Liu J, Liu W (2016) Recurrent neural network based recommendation for time heterogeneous feedback. Knowl-Based Syst 109:90–103
Xie R, Liu Z, Yan R, Sun M (2016) Neural emoji recommendation in dialogue systems. arXiv preprint arXiv:1612.04609
Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov R, Le QV (2019) XLNet: generalized autoregressive pretraining for language understanding, pp 1–18. arXiv:1906.08237
Yu W, Zhang H, He X, Chen X, Xiong L, Qin Z (2018) Aesthetic-based clothing recommendation. In: Proceedings of the 2018 World Wide Web conference, pp 649–658
Zhang S, Yao L, Sun A, Tay Y (2019) Deep learning based recommender system: a survey and new perspectives. ACM Comput Surv (CSUR) 52(1):1–38
Zheng L, Lu CT, He L, Xie S, He H, Li C, Noroozi V, Dong B, Philip SY (2019) Mars: memory attention-aware recommender system. In: 2019 IEEE international conference on Data Science and Advanced Analytics (DSAA). IEEE, pp 11–20
Conflict of interest
The authors declare that they have no conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Fang, X. Making recommendations using transfer learning. Neural Comput & Applic (2021). https://doi.org/10.1007/s00521-021-05730-3
- Recommender system
- Deep learning
- Transfer learning
- User vector embedding