Abstract
Distributed Stream Processing (DSP) systems are well acknowledged to be potent in processing huge volume of real-time stream data with low latency and high throughput. Recently, the edge computing paradigm shows great potentials in supporting and boosting the DSP applications, especially the time-critical and latency-sensitive ones, over the Internet of Things (IoT) or mobile devices by means of offloading the computation from remote cloud to edge servers for further reduced communication latencies. Nevertheless, various challenges, especially the joint operator scaling and placement, are yet to be properly explored and addressed. Traditional efforts in this direction usually assume that the data-flow graph of a DSP application is pre-given and static. The resulting models and methods can thus be ineffective and show bad user-perceived quality-of-service (QoS) when dealing with real-world scenarios with reconfigurable data-flow graphs and scalable operator placement. In contrast, in this paper, we consider that the data-flow graphs are configurable and hence propose the joint operator scaling and placement problem. To address this problem, we first build a queuing-network-based QoS estimation model, then formulate the problem into an integer-programming one, and finally propose a two-stage approach for finding the near-optimal solution. Experiments based on real-world DSP test cases show that our method achieves higher cost effectiveness than traditional ones while meeting the user-defined QoS constraints.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amarasinghe, G., de Assuno, M.D., Harwood, A., Karunasekera, S.: A data stream processing optimisation framework for edge computing applications. In: 2018 IEEE 21st International Symposium on Real-Time Distributed Computing (ISORC), pp. 91–98. IEEE (2018)
Benoit, A., Dobrila, A., Nicod, J.M., Philippe, L.: Scheduling linear chain streaming applications on heterogeneous systems with failures. Future Gener. Comput. Syst. 29(5), 1140–1151 (2013)
Cai, X., Kuang, H., Hu, H., Song, W., Lü, J.: Response time aware operator placement for complex event processing in edge computing. In: Pahl, C., Vukovic, M., Yin, J., Yu, Q. (eds.) ICSOC 2018. LNCS, vol. 11236, pp. 264–278. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03596-9_18
Cardellini, V., Grassi, V., Lo Presti, F., Nardelli, M.: Optimal operator replication and placement for distributed stream processing systems. ACM SIGMETRICS Perform. Eval. Rev. 44(4), 11–22 (2017)
Cardellini, V., Lo Presti, F., Nardelli, M., Russo Russo, G.: Optimal operator deployment and replication for elastic distributed data stream processing. Concurr. Comput. Pract. Exp. 30(9), e4334 (2018)
Gen, M., Lin, L.: Genetic algorithms. In: Wiley Encyclopedia of Computer Science and Engineering, pp. 1–15 (2007)
Gibert Renart, E., da Silva Veith, A., Balouek-Thomert, D., Dias de Assuncao, M., Lefèvre, L., Parashar, M.: Distributed operator placement for IoT data analytics across edge and cloud resources. In: CCGrid 2019 - 19th Annual IEEE/ACM International Symposium in Cluster, Cloud, and Grid Computing, pp. 1–10. IEEE/ACM (2019)
Hidalgo, N., Rosas, E.: Self-adaptive processing graph with operator fission for elastic stream processing. J. Syst. Softw. 127, 205–216 (2017)
Hiessl, T., Karagiannis, V., Hochreiner, C., Schulte, S., Nardelli, M.: Optimal placement of stream processing operators in the fog. In: 2019 IEEE 3rd International Conference on Fog and Edge Computing (ICFEC), pp. 1–10. IEEE (2019)
Hu, W., et al.: quantifying the impact of edge computing on mobile applications. In: Proceedings of the 7th ACM SIGOPS Asia-Pacific Workshop on Systems, p. 5. ACM (2016)
Kaur, N., Sood, S.K.: Efficient resource management system based on 4vs of big data streams. Big Data Res. 9, 98–106 (2017)
Mai, L., et al.: Chi: a scalable and programmable control plane for distributed stream processing systems. Proc. VLDB Endow. 11(10), 1303–1316 (2018)
Myrvold, W., Ruskey, F.: Ranking and unranking permutations in linear time. Inf. Process. Lett. 79(6), 281–284 (2001)
Nardelli, M., Cardellini, V., Grassi, V., Presti, F.L.: Efficient operator placement for distributed data stream processing applications. IEEE Trans. Parallel Distrib. Syst. 30(8), 1753–1767 (2019)
Networking, C.V.: Cisco Global Cloud Index: Forecast and Methodology, 2016–2021. Cisco Public, San Jose (2018). White paper
Pietzuch, P., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.: Network-aware operator placement for stream-processing systems. In: 22nd International Conference on Data Engineering (ICDE 2006), pp. 49–49. IEEE (2006)
Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)
da Silva Veith, A., de Assunção, M.D., Lefèvre, L.: Latency-aware placement of data stream analytics on edge computing. In: Pahl, C., Vukovic, M., Yin, J., Yu, Q. (eds.) ICSOC 2018. LNCS, vol. 11236, pp. 215–229. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03596-9_14
Taneja, M., Davy, A.: Resource aware placement of iot application modules in fog-cloud computing paradigm. In: 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pp. 1222–1228. IEEE (2017)
Yang, S.: Iot stream processing and analytics in the fog. IEEE Commun. Mag. 55(8), 21–27 (2017)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, Q., Xia, Y., Wang, Y., Wu, C., Luo, X., Lee, J. (2019). Joint Operator Scaling and Placement for Distributed Stream Processing Applications in Edge Computing. In: Yangui, S., Bouassida Rodriguez, I., Drira, K., Tari, Z. (eds) Service-Oriented Computing. ICSOC 2019. Lecture Notes in Computer Science(), vol 11895. Springer, Cham. https://doi.org/10.1007/978-3-030-33702-5_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-33702-5_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-33701-8
Online ISBN: 978-3-030-33702-5
eBook Packages: Computer ScienceComputer Science (R0)