Abstract
Distributed stream processing architecture has emerged as appealing solution to coping with the analysis of large amount of data from dispersed sources. A fundamental problem in such stream processing systems is how to best utilize the available resources so that the overall system performance is optimized. We consider a distributed stream processing system that consists of a network of cooperating servers, collectively providing processing services for multiple data streams. Each stream is required to complete a series of operations on various servers. We assume all servers have finite computing resources and all communication links have finite available bandwidth. The problem is to find distributed schemes to allocate the limited computing resources as well as the communication bandwidth in the system so as to achieve a maximum concurrent throughput for all output streams. We present a generalized multicommodity flow model for the above problem. We develop a distributed resource allocation algorithm that guarantees the optimality. We also provide detailed analysis on the complexity of the algorithm and demonstrate the performance using numerical experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abadi, et al.: Aurora: A new model and architecture for data stream management. VLDB Journal 12(2) (September 2003)
Awerbuch, B., Leighton, F.: A simple local-control approximation algorithm for multicommodity flow. In: Proc. of the 34th IEEE Symp. on Foundations of Computer Science (FOCS), pp. 459–468 (1993)
Awerbuch, B., Leighton, F.: Improved approximation algorithms for the multi-commodity flow problem and local competitive routing in dynamic networks. In: Proc. of the 26th ACM Symp. on Theory of Computing (STOC), pp. 487–496 (1994)
Ahmad, Y., et al.: Distributed Operation in the Borealis Stream Processing Engine. In: SIGMOD 2005 (2005)
Babcock, B., Babu, S., Datar, M., Motwani, R.: Chain: Operator scheduling for memory minimization in data stream systems. In: SIGMOD (June 2003)
Bazaraa, M.S., Jarvis, J.J., Sherali, H.D.: Linear Programming and Network Flows. John Wiley & Sons, Chichester (1977)
Broberg, J.A., Liu, Z., Xia, C.H., Zhang, L.: A Multicommodity Flow Model for Distributed Streaming Processing. Poster in SIGMETRICS (2006)
Carney, D., Cÿetintemel, U., Rasin, A., Zdonik, S., Cherniack, M., Stonebraker, M.: Operator scheduling in a data stream manager. In: 29th VLDB (September 2003)
Chandrasekaran, S., Franklin, M.J.: Remembrance of streams past: Overload-sensitive management of archived streams. In: 30th VLDB (September 2004)
Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to Algorithms. MIT Press/ McGraw-Hill Book Company, Cambridge/ Boston (1990)
Cranor, C., Johnson, T., Shkapenyuk, V., Spatscheck, O.: Gigascope: A stream database for network applications. In: SIGMOD (June 2003)
Approximating Fractional Multicommodity Flows Independent of the Number of Commodities. SIAM J. Discrete Math. 13(4), 505–520 (2000)
Hu, T.C.: Multi-Commodity Network Flows. Operations Research 11, 344–360 (1963)
Motwani, et al.: Query processing, approximation, and resource management in a data stream management system. In: CIDR (January 2003)
Pietzuch, P., Shneidman, J., Ledlie, J., Welsh, M., Seltzer, M., Roussopoulos, M.: Hourglass: A Stream-Based Overlay Network for Sensor Applications. In: HIP 2004 (2004)
Shahrokhi, F., Matula, D.W.: The maximum concurrent flow problem. J. Assoc. Comput. Mach. 37, 318–334 (1990)
Stein, C.: Approximation algorithms for multicommodity flow and shop scheduling problems, Ph.D thesis. MIT (1992)
Srivastava, U., Munagala, K., Widom, J.: Operator placement for in-network stream query processing. In: Proceedings of the 24th ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pp. 250–258 (2005)
Tatbul, N., Etintemel, U.C., Zdonik, S., Cherniack, M., Stonebraker, M.: Load shedding in a data stream manager. In: 29th VLDB (September 2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Xia, C.H., Broberg, J.A., Liu, Z., Zhang, L. (2006). Distributed Resource Allocation in Stream Processing Systems. In: Dolev, S. (eds) Distributed Computing. DISC 2006. Lecture Notes in Computer Science, vol 4167. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11864219_34
Download citation
DOI: https://doi.org/10.1007/11864219_34
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44624-8
Online ISBN: 978-3-540-44627-9
eBook Packages: Computer ScienceComputer Science (R0)