Design and Implementation of the Cross-Harmonic Recommender System Based on Spark

  • Huang JieEmail author
  • Liu ChangSheng
  • Liu ChengLi
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 302)


With the rapid development of information technology, information overload has become an important challenge of Internet. In order to alleviate the growing contradiction between users and massive data, the researchers proposed the concept of the cross-harmonic recommender system. By analyzing characteristic of datasets, recommendation algorithms and method for weight calculation, we introduced a fast and general engine for large-scale data processing and implemented the cross-harmonic recommender system based on Spark, aiming at improving accuracy, diversity and efficiency of the recommender system.


Spark Hybrid recommendation Recommendation algorithm 


  1. 1.
    Shvachko, K., Kuang, H., Radia, S., et al.: The hadoop distributed file system. In: 2014 IEEE 26th Symposium on IEEE Mass Storage Systems and Technologies (MSST), pp. 1–20 (2014)Google Scholar
  2. 2.
    Zaharia, M., Das, T., Li, H., et al.: Discretized streams: an efficient and fault-tolerant model for stream processing on large cluster. HotCloud 12, 34–56 (2015)Google Scholar
  3. 3.
    Meng, X., Bradley, J., Yavuz, B., et al.: Mllib: machine learning in apache spark. J. Mach. Learn. Res. 12(34), 23–34 (2017)zbMATHGoogle Scholar
  4. 4.
    Wang, Z., Sun, L., Zhu, W., et al.: Joint social and content recommendation for user-generated videos in online social network. IEEE Trans. Multimed. 15(3), 698–709 (2017)CrossRefGoogle Scholar
  5. 5.
    Koren, Y.: The bellkor solution to the netfix grand prize. Netflix Prize Doc. 34, 32–45 (2016)Google Scholar
  6. 6.
    Zaharia, M., Chowdbury, M., Franklin, M.J., et al.: Spark: cluster computing with working sets. HotCloud 10(11), 80–98 (2015)Google Scholar
  7. 7.
    Li, H., Ghodsi, A., Zaharia, M., et al.: Tachyon: reliable, memory speed storage for cluster computing frameworks. In: Proceedings of the ACM Symposium on Cloud Computing, pp. 13–25. ACM (2016)Google Scholar
  8. 8.
    Engle, C., Lupher, A., Xin, R., et al.: Shark: first data analysis using coarse-grained distributed memory. In: Proceeding of the 2012 ACM SIGMOD International Conference on Management of Data (2012)Google Scholar
  9. 9.
    Armbrust, M., Xin, R.S., Lian, C., et al.: Spark SQL: relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1383–1394. ACM (2015)Google Scholar
  10. 10.
    Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1), 143–177 (2014)CrossRefGoogle Scholar
  11. 11.
    Yang, R., Hu, W., Qu, Y.: Using semantic technology to improve recommender systems based on slope one. In: Li, J., Qi, G., Zhao, D., Nejdl, W., Zheng, H.T. (eds.) Semantic Web and Web Science, pp. 11–23. Springer, New York (2013). Scholar
  12. 12.
    Takács, G., Pilászy, I., Németh, B., Tikk, D.: Major components of the gravity recommendation system. ACM SIGKDD Explor. Newsl. 9(2), 80–83 (2014)CrossRefGoogle Scholar
  13. 13.
    Nightingale, E.B., Chen, P.M., Flinn, J.: Speculative execution in a distributed file system. ACM (2010)Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2019

Authors and Affiliations

  1. 1.Department of Aviation Electronic Equipment MaintenanceAirforce Aviation Repair Institute of TechnologyChangshaChina
  2. 2.Hunan Key Laboratory of Intelligent Information Perception and Processing TechnologyZhuzhouChina
  3. 3.School of Engineering, Computer and AviationUniversity of LeónLeónSpain

Personalised recommendations