Encyclopedia of Big Data Technologies

Living Edition
| Editors: Sherif Sakr, Albert Zomaya

StreamMine3G: Elastic and Fault Tolerant Large Scale Stream Processing

  • André Martin
  • Andrey Brito
  • Christof Fetzer
Living reference work entry
DOI: https://doi.org/10.1007/978-3-319-63962-8_145-1

Introduction

During the past decade, we have been witnessing a massive growth of data. In particular the advent of new mobile devices such as smartphones, tablets and online services like Facebook and Twitter created a complete new era for data processing. Although there exist already well-established approaches such as MapReduce (Dean and Ghemawat 2008) and its open-source implementation Hadoop (2015) in order to cope with these large amounts of data, there is a recent trend of moving away from batch processing to low-latency online processing using event stream processing (ESP) systems. Inspired by the simplicity of the MapReduce programming paradigm, a number of open-source as well as commercial ESP systems have evolved over the past years such as Apache S4 (Neumeyer et al. 2010; Apache 2015) (originally pushed by Yahoo!), Apache Storm (2015) (Twitter), and Apache Samza (2015) (LinkedIn), addressing the strong need for data processing in near real time.

Since the amount of data...

This is a preview of subscription content, log in to check access.

References

  1. Apache s4 – distributed stream computing platform (2015). https://incubator.apache.org/s4/
  2. Apache samza – a distributed stream processing framework (2015). http://samza.incubator.apache.org/
  3. Apache storm – distributed and faulttolerant realtime computation (2015). https://storm.incubator.apache.org/
  4. Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113CrossRefGoogle Scholar
  5. Gu Y, Zhang Z, Ye F, Yang H, Kim M, Lei H, Liu Z (2009) An empirical study of high availability in stream processing systems. In: Proceedings of the 10th ACM/IFIP/USENIX international conference on middleware, Middleware ’09, New York, pp 23:1–23:9. Springer, New YorkGoogle Scholar
  6. Hadoop mapreduce open source implementation (2015). http://hadoop.apache.org/
  7. Hunt P, Konar M, Junqueira FP, Reed B (2010) Zookeeper: wait-free coordination for Internet-scale systems. In: Proceedings of the 2010 USENIX conference on USENIX annual technical conference, USENIXATC’10, Berkeley. USENIX Association, pp 1–14Google Scholar
  8. Hwang J-H, Balazinska M, Rasin A, Cetintemel U, Stonebraker M, Zdonik S (2005) High-availability algorithms for distributed stream processing. In: Proceedings of the 21st international conference on data engineering, ICDE ’05, Washington, DC. IEEE Computer Society, pp 779–790Google Scholar
  9. Martin A, Fetzer C, Brito A (2011) Active replication at (almost) no cost. In: Proceedings of the 2011 IEEE 30th international symposium on reliable distributed systems, SRDS ’11, Washington, DC. IEEE Computer Society, pp 21–30Google Scholar
  10. Martin A, Knauth T, Creutz S, Becker D, Weigert S, Fetzer C, Brito A (2011) Low-overhead fault tolerance for high-throughput data processing systems. In: Proceedings of the 2011 31st international conference on distributed computing systems, ICDCS ’11, Washington, DC. IEEE Computer Society, pp 689–699Google Scholar
  11. Martin A, Smaneoto T, Dietze T, Brito A, Fetzer C (2015) User-constraint and self-adaptive fault tolerance for event stream processing systems. In: Proceedings of The 45th annual IEEE/IFIP international conference on dependable systems and networks (DSN 2015), Los Alamitos. IEEE Computer SocietyGoogle Scholar
  12. Neumeyer L, Robbins B, Nair A, Kesari A (2010) S4: distributed stream computing platform. In: Proceedings of the 2010 IEEE international conference on data mining workshops, ICDMW ’10, Washington, DC. IEEE Computer Society, pp 170–177Google Scholar
  13. Schneider FB (1990) Implementing fault-tolerant services using the state machine approach: a tutorial. ACM Comput Surv 22(4):299–319CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.TU DresdenDresdenGermany
  2. 2.UFCGCampina GrandeBrazil

Section editors and affiliations

  • Asterios Katsifodimos
    • 1
  • Pramod Bhatotia
    • 2
  1. 1.Delft University of TechnologyDelftNetherlands
  2. 2.School of InformaticsUniversity of EdinburghEdinburghUnited Kingdom