Performance Analysis of Machine Learning Techniques on Big Data Using Apache Spark

  • Garima Mogha
  • Khyati AhlawatEmail author
  • Amit Prakash SinghEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 799)


Applying Intelligence to the machines is a need in today’s world and this need leads to the evolution of machine learning. The analysis of data using machine learning algorithms is a trending research area and this analysis lead to some problems when the data comes out to be big data. This paper compares various classification based machine learning algorithms namely, Decision Tree Learning, Naïve Bayes, Random Forest and Support Vector Machines on big data using Apache Spark. The accuracy is evaluated to find out which classification based algorithm gives fast and better result.


Big data Apache spark Machine learning Apache hadoop 


  1. 1.
    Gupta, G.P., Kulariya, M.: A framework for fast and efficient cyber security network intrusion detection using apache spark. Procedia Comput. Sci. 93, 824–831 (2016)CrossRefGoogle Scholar
  2. 2.
    Shyam, R., Bharathi Ganesh, H.B., Kumar, S., Poornachandran, P., Soman, K.P.: Apache spark a big data analytics platform for smart grid. Procedia Technol. 21, 171–178 (2015)CrossRefGoogle Scholar
  3. 3.
    Kavakiotis, I., Tsave, O., Salifoglou, A., Maglaveras, N., Vlahavas, I., Chouvarda, I.: Machine learning and data mining methods in diabetes research. Comput. Struct. Biotechnol. J. 15, 104–116 (2017)CrossRefGoogle Scholar
  4. 4.
    Kumar, D., Singh, R., Kumar, A., Sharma, N.: An adaptive method of PCA for minimization of classification error using Naïve Bayes classifier. Procedia Comput. Sci. 70, 9–15 (2015)CrossRefGoogle Scholar
  5. 5.
    Zhang, P., Wu, X., Wang, X., Bi, S.: Short-term load forecasting based on big data technologies. CSEE J. Power Energy Syst. 1(3), 59–67 (2015)CrossRefGoogle Scholar
  6. 6.
    Liu, S., Wang, X., Liu, M., Zhu, J.: Towards better analysis of machine learning models: a visual analytics perspective. Vis. Inf. 1(1), 48–56 (2017)Google Scholar
  7. 7.
    Panigrahi, S., Lenka, R.K., Stitipragyan, A.: A hybrid distributed collaborative filtering recommender engine using apache spark. Procedia Comput. Sci. 83, 1000–1006 (2016)CrossRefGoogle Scholar
  8. 8.
    Alpaydin, E.: Introduction to Machine Learning, 3rd edn. The MIT Press, Cambridge, London (2014)zbMATHGoogle Scholar
  9. 9.
    Berthold, M.R., Borgelt, C., Höppner, F., Klawonn, F.: Guide to Intelligent Data Analysis, 2nd edn. Springer, London (2010). Scholar
  10. 10.
    Kelleher, J.D., Mac Namee, B., D’Arcy, A.: Fundamentals of Machine Learning for Predictive Data Analytics. The MIT Press, Cambridge, London (2015)zbMATHGoogle Scholar
  11. 11.
    Han, J., Kamber, M.: Data Mining: Concepts and Techniques, 2nd edn. Morgan Kaufman Publishers, Burlington (2011)zbMATHGoogle Scholar
  12. 12.
    Mitchell, T.M.: Machine Learning. Mcgraw Hill Education Private Limited, New York (1997)zbMATHGoogle Scholar
  13. 13.
    Scott, J.A.: Getting Started with Apache Spark: Inception to Production, 1st edn. MapR Technologies, San Jose (2015)Google Scholar
  14. 14.
    Sivarajah, U., Kamal, M.M., Irani, Z., Weerakkody, V.: Critical analysis of big data challenges and analytical methods. J. Bus. Res. 70, 263–286 (2017)CrossRefGoogle Scholar
  15. 15.
    Reyes-Ortiz, J.L., Oneto, L., Anguita, D.: Big data analytics in the cloud: spark on Hadoop vs MPI/OpenMP on Beowulf. Procedia Comput. Sci. 53, 121–130 (2015)CrossRefGoogle Scholar
  16. 16.
    Shafique, M.A., Hato, E.: Classification of travel data with multiple sensor information using random forest. Transp. Res. Procedia 22, 144–153 (2017)CrossRefGoogle Scholar
  17. 17.
    Swetapadma, A., Yadav, A.: Protection of parallel transmission lines including inter-circuit faults using Naïve Bayes classifier. Alexandria Eng. J. 55(2), 1411–1419 (2016)CrossRefGoogle Scholar
  18. 18.
    Jayasree, V., Balan, R.S.: Money laundering regulatory risk evaluation using bitmap index-based decision tree. J. Assoc. Arab Univ. Basic Appl. Sci. 23, 96–102 (2017)Google Scholar
  19. 19.
    Götz, M., Richerzhagen, M., Bodenstein, C., Cavallaro, G., Glock, P., Riedel, M., Benediktsson, J.A.: On scalable data mining techniques for earth science. Procedia Comput. Sci. 51, 2188–2197 (2015)CrossRefGoogle Scholar
  20. 20.
  21. 21.
  22. 22.
  23. 23.

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.University School of Information Communication and TechnologyGuru Gobind Singh Indraprastha UniversityNew DelhiIndia

Personalised recommendations