Towards Hierarchical Autonomous Control for Elastic Data Stream Processing in the Fog

  • Valeria Cardellini
  • Francesco Lo Presti
  • Matteo Nardelli
  • Gabriele Russo Russo
Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10659)

Abstract

In the Big Data era, Data Stream Processing (DSP) applications should be capable to seamlessly process huge amount of data. Hence, they need to dynamically scale their execution on multiple computing nodes so to adjust to unpredictable data source rate. In this paper, we present a hierarchical and distributed architecture for the autonomous control of elastic DSP applications. It revolves around a two layered approach. At the lower level, distributed components issue requests for adapting the deployment of DSP operations as to adjust to changing workload conditions. At the higher level, a per-application centralized component works on a broader time scale; it oversees the application behavior and grants reconfigurations to control the application performance while limiting the negative effect of their enactment, i.e., application downtime. We have implemented the proposed solution in our distributed Storm prototype and evaluated its behavior adopting simple policies. The experimental results are promising and show that, even with simple policies, it is possible to limit the number of reconfigurations while at the same time guaranteeing an adequate level of application performance.

Keywords

Data Stream Processing Self adaptive Hierarchical control MAPE loop 
This is a preview of subscription content, log in to check access

References

  1. 1.
    Cardellini, V., Lo Presti, F., Nardelli, M., Russo Russo, G.: Optimal operator deployment and replication for elastic distributed data stream processing. Concurrency Comput.: Practice Exper. (2017).  https://doi.org/10.1002/cpe.4334
  2. 2.
    Cardellini, V., Grassi, V., Lo Presti, F., Nardelli, M.: Distributed QoS-aware scheduling in Storm. In: Proceedings of ACM DEBS 2015, pp. 344–347 (2015)Google Scholar
  3. 3.
    De Matteis, T., Mencagli, G.: Elastic scaling for distributed latency-sensitive data stream operators. In: Proceedings of PDP 2017, pp. 61–68 (2017)Google Scholar
  4. 4.
    Fernandez, R.C., Migliavacca, M., Kalyvianaki, E., Pietzuch, P.: Integrating scale out and fault tolerance in stream processing using operator state management. In: Proceedings of ACM SIGMOD 2013, pp. 725–736 (2013)Google Scholar
  5. 5.
    Gedik, B., Schneider, S., Hirzel, M., Wu, K.L.: Elastic scaling for data stream processing. IEEE Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2014)CrossRefGoogle Scholar
  6. 6.
    Gulisano, V., Jiménez-Peris, R., Patiño Martínez, M., Soriente, C., Valduriez, P.: StreamCloud: an elastic and scalable data streaming system. IEEE Trans. Parallel Distrib. Syst. 23(12), 2351–2365 (2012)CrossRefGoogle Scholar
  7. 7.
    Heinze, T., Pappalardo, V., Jerzak, Z., Fetzer, C.: Auto-scaling techniques for elastic data stream processing. In: Proceedings of IEEE ICDEW 2014, pp. 296–302 (2014)Google Scholar
  8. 8.
    Heinze, T., Roediger, L., Meister, A., Ji, Y., et al.: Online parameter optimization for elastic data stream processing. In: Proceedings of ACM SoCC 2015, pp. 276–287 (2015)Google Scholar
  9. 9.
    Jerzak, Z., Ziekow, H.: The DEBS 2015 grand challenge. In: Proceedings of ACM DEBS 2015, pp. 266–268 (2015)Google Scholar
  10. 10.
    Lohrmann, B., Janacik, P., Kao, O.: Elastic stream processing with latency guarantees. In: Proceedings of IEEE ICDCS 2015, pp. 399–410 (2015)Google Scholar
  11. 11.
    Mencagli, G.: A game-theoretic approach for elastic distributed data stream processing. ACM Trans. Auton. Adapt. Syst. 11(2), 13:1–13:34 (2016)CrossRefGoogle Scholar
  12. 12.
    Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., et al.: Network-aware operator placement for stream-processing systems. In: Proceedings of IEEE ICDE 2006 (2006)Google Scholar
  13. 13.
    Sajjad, H.P., Danniswara, K., Al-Shishtawy, A., Vlassov, V.: Spanedge: towards unifying stream processing over central and near-the-edge data centers. In: Proceedings of 2016 IEEE/ACM Symposium on Edge Computing, pp. 168–178 (2016)Google Scholar
  14. 14.
    Saurez, E., Hong, K., Lillethun, D., Ramachandran, U., et al.: Incremental deployment and migration of geo-distributed situation awareness applications in the fog. In: Proceedings of ACM DEBS 2016, pp. 258–269 (2016)Google Scholar
  15. 15.
    Weyns, D., et al.: On patterns for decentralized control in self-adaptive systems. In: de Lemos, R., Giese, H., Müller, H.A., Shaw, M. (eds.) Software Engineering for Self-Adaptive Systems II. LNCS, vol. 7475, pp. 76–107. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-35813-5_4 CrossRefGoogle Scholar
  16. 16.
    Xu, L., Peng, B., Gupta, I.: Stela: enabling stream processing systems to scale-in and scale-out on-demand. In: Proceedings of IEEE IC2E 2016, pp. 22–31 (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Civil Engineering and Computer Science EngineeringUniversity of Rome Tor VergataRomeItaly

Personalised recommendations