Advertisement

A Survey on Various Message Brokers for Real-Time Big Data

  • Spandana SrinivasEmail author
  • Viswavardhan Reddy Karna
Conference paper
Part of the Lecture Notes on Data Engineering and Communications Technologies book series (LNDECT, volume 39)

Abstract

In the current scenario, processing the huge information is very difficult in the data pipelines. The solution to overcome the above problem is by using message brokers. This will help in collecting and delivering the huge amount of real-time data. In this work, producer client is designed using java language to fetch the information of network elements using Kafka messaging scheme. The consumer application is configured to fetch information from kafka bus. Moreover, performance metrics of kafka bus is also collected. From the results, it is observed that kafka achieved 100% throughput and latency is 1.3 s for fetching 2 lakh messages. From this, it is understood that kafka is very fast, reliable and fault tolerant messaging scheme. Moreover, from the survey it is shown that kafka is the best messaging scheme.

Keywords

Kafka Zookeeper Topic Producer Consumer 

References

  1. 1.
    Hiraman, B.R., Chapte Viresh, M., Abhijeet, C.K.: A study of Apache Kafka in big data stream processing. In: International Conference on Information, Communication, Engineering and Technology (ICICET), pp. 1–3, Pune (2018)Google Scholar
  2. 2.
    Ionescu, V.M.: The analysis of the performance of RabbitMQ and ActiveMQ. In: 14th RoEduNet International Conference - Networking in Education and Research (RoEduNet NER), pp. 132–137, Craiova (2015)Google Scholar
  3. 3.
    Hong, X.J., Yang, H.S., Kim, Y.H.: Performance analysis of RESTful API and RabbitMQ for microservice web application. In: International Conference on Information and Communication Technology Convergence (ICTC), pp. 257–259, Jeju (2018)Google Scholar
  4. 4.
    Klein, A.F., Ştefãnescu, M., Saied, A., Swakhoven, K.: An experimental comparison of ActiveMQ and OpenMQ brokers in asynchronous cloud environment. In: Fifth International Conference on Digital Information Processing and Communications (ICDIPC), pp. 24–30, Sierre (2015)Google Scholar
  5. 5.
    He, D., Kang, Y., Su, X.: Research on data exchange platform based on JMS. In: 2nd IEEE International Conference on Computer and Communications (ICCC), pp. 110–113, Chengdu (2016)Google Scholar
  6. 6.
    Chen, G., Du, Y., Qin, P., Zhang, L.: Research of JMS based message oriented middleware for cluster. In: International Conference on Computational and Information Sciences, pp. 1628–1631, Shiyang (2013)Google Scholar
  7. 7.
    Shree, R., Choudhury, T., Gupta, S.C., Kumar, P.: KAFKA: the modern platform for data management and analysis in big data domain. In: 2nd International Conference on Telecommunication and Networks (TEL-NET), pp. 1–5, Noida (2017)Google Scholar
  8. 8.
    Wang, Z.: Kafka and ıts using in high-throughput and reliable message distribution. In: 8th International Conference on Intelligent Networks and Intelligent Systems (ICINIS), pp. 117–120, Tianjin (2015)Google Scholar
  9. 9.
    Bang, J., Son, S., Kim, H., Moon, Y., Choi, M.: Design and implementation of a load shedding engine for solving starvation problems in Apache Kafka. In: IEEE/IFIP Network Operations and Management Symposium, pp. 1–4, Taipei (2018)Google Scholar
  10. 10.
    Nguyen, C.N., Kim, J., Hwang, S.: KOHA: building a kafka-based distributed queue system on the fly in a hadoop cluster. In: IEEE 1st International Workshops on Foundations and Applications of Self* Systems (FAS*W), pp. 48–53, Augsburg (2016)Google Scholar
  11. 11.
    Versaci, F., Pireddu, L., Zanetti, G.: Kafka interfaces for composable streaming genomics pipelines. In: IEEE EMBS International Conference on Biomedical & Health Informatics (BHI), pp. 259–262, Las Vegas, NV (2018)Google Scholar
  12. 12.
    Kato, K., Takefusa, A., Nakada, H., Oguchi, M.: A study of a scalable distributed stream processing infrastructure using Ray and Apache Kafka. In: IEEE International Conference on Big Data (Big Data), pp. 5351–5353, Seattle, WA, USA (2018)Google Scholar
  13. 13.
    Le Noac’h, P., Costan, A., Bougé, L.: A performance evaluation of Apache Kafka in support of big data streaming applications. In: IEEE International Conference on Big Data, pp. 4803–4806, Boston, MA (2017)Google Scholar
  14. 14.
    Wang, M., Liu, J., Zhou, W.: Design and ımplementation of a high-performance stream-oriented big data processing system. In: 8th International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC), pp. 363–368, Hangzhou (2016)Google Scholar
  15. 15.
    Vargas, W.V., Munoz-Arcentales, A., Rodríguez, J.S.: A distributed system model for managing data ingestion in a wireless sensor network. In: IEEE 7th Annual Computing and Communication Workshop and Conference (CCWC), pp. 1–5, Las Vegas, NV (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of TCERVCEBengaluruIndia

Personalised recommendations