Abstract
In the recent years, many emerging on-line data analysis applications require real-time delivery of the streaming data while dealing with unpredictable increase in the volume of data. In this paper we propose a novel approach for efficient stream processing of bursts in the Cloud. Our approach uses two queues to schedule requests pending execution. When bursts occur, incoming requests that exceed maximum processing capacity of the node, instead of being dropped, are diverted to a secondary queue. Requests in the secondary queue are concurrently scheduled with the primary queue, so that they can be immediately executed whenever the node has any processing power unused as the results of burst fluctuations. With this mechanism, processing power of nodes is fully utilized and the bursts are efficiently accommodated. Our experimental results illustrate the efficiency of our approach.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Chandrasekaran, S., Cooper, O., Deshpande, A., Franklin, M.J., Hellerstein, J.M., Hong, W., Krishnamurthy, S., Madden, S., Raman, V., Reiss, F., Shah, M.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: CIDR, Asilomar, CA (January 2003)
Tatbul, N., Çetintemel, U., Zdonik, S.B., Cherniack, M., Stonebraker, M.: Load Shedding in a Data Stream Manager. In: VLDB 2003, Berlin, Germany, pp. 309–320 (2003)
Arasu, A., Babcock, B., Babu, S., Cieslewicz, J., Datar, M., Ito, K., Motwani, R., Srivastava, U., Widom, J.: STREAM: The Stanford Data Stream Management System (March 2005)
Madden, S., Gehrke, J.: Query Processing in Sensor Networks. IEEE Pervasive Computing, vol 3(1) (March 2004)
Lu, L., Varman, P., Doshi, K.: Graduated QoS by Decomposing Bursts: Don’t Let the Tail Wag Your Server. In: ICDCS 2009, Montreal, QC, Canada, pp. 12–21 (June 2009)
Repantis, T., Gu, X., Kalogeraki, V.: Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems. In: van Steen, M., Henning, M. (eds.) Middleware 2006. LNCS, vol. 4290, pp. 322–341. Springer, Heidelberg (2006)
Kalogeraki, V., Melliar-Smith, P.M., Moser, L.E.: Dynamic Scheduling of Distributed Method Invocations. In: IEEE Real-Time Systems Symposium (RTSS), Orlando, FL (December 2000)
FreePastry (2006), http://freepastry.org/FreePastry
Drougas, Y., Kalogeraki, V.: RASC: Dynamic Rate Allocation for Distributed Stream Processing Applications. In: International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, CA (March 2007)
Chen, F., Kalogeraki, V.: RUBEN: A Technique for Scheduling Multimedia Applications in Overlay Networks. In: Globecom 2004, Dalas, TX (November 2004)
Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive Control of Extreme-scale Stream Processing Systems. In: ICDCS 2006, Lisboa, Portugal (2006)
Tatbul, N., Çetintemel, U., Zdonik, S.: Staying FIT: Efficient Load Shedding Techniques for Distributed Stream Processing. In: VLDB 2007, Vienna, Austria, pp. 159–170 (September 2007)
Chen, Y., Lu, C., Koutsoukos, X.: Optimal Discrete Rate Adaptation for Distributed Real-Time Systems. In: Real Time Systems Symposium (RTSS), Tucson, AZ (December 2007)
Drougas, Y., Kalogeraki, V.: Accommodating Bursts in Distributed Stream Processing Systems. In: 23rd International Parallel and Distributed Processing Symposium (IPDPS), Rome, Italy (May 2009)
Amazon Elastic Computer Cloud (Amazon EC2), http://aws.amazon.com/ec2/
IBM Cloud Computing, http://www.ibm.com/ibm/cloud/
Drougas, Y., Kalogeraki, V.: RASC: Dynamic Rate Allocation for Distributed Stream Processing Applications. In: International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, CA (March 2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 ICST Institute for Computer Science, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Vu, D., Kalogeraki, V., Drougas, Y. (2012). Efficient Stream Processing in the Cloud. In: Zhang, X., Qiao, D. (eds) Quality, Reliability, Security and Robustness in Heterogeneous Networks. QShine 2010. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 74. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-29222-4_19
Download citation
DOI: https://doi.org/10.1007/978-3-642-29222-4_19
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-29221-7
Online ISBN: 978-3-642-29222-4
eBook Packages: Computer ScienceComputer Science (R0)