Data Stream Analytics as Cloud Service for Mobile Applications
Many mobile applications are based on cloud services such as location service, messaging service, etc. Currently most cloud services are based on statically prepared information rather than the real-time analytics results of dynamically captured events. A paradigm shift is to take Continuous Stream Analytics (CSA) as a cloud service, which, however, poses several specific challenges in scalability, latency, time-window semantics and transaction control.
In this work we extend the SQL query engine to unify the processing of static relations and dynamic streams for providing the platform support of CSA service. This platform is significantly differentiated from the current generation of stream processing systems which are in general built separately from the database engine thus unable to take advantage of the functionalities already offered by the existing data management technology, and suffer from the overhead of inter-platform data access and movement.
To capture the window semantics in CSA, we introduce the cycle-based query model and support it in terms of the cut-and-rewind query execution mechanism. This mechanism allows a SQL query to run cycle by cycle for processing the unbounded stream data chunk by chunk, but without shutting the query instance down between chunks for continuously maintaining the application state across the execution cycles, as required by sliding-window oriented operations. We also propose the cycle-based transaction model with cycle-based isolation and visibility. To scale-up analytics computation, we introduce the parallel infrastructure with multi-engines cooperated and synchronized based the common data chunking criteria without centralized coordination. To scale-up service provisioning, we investigate how to stage the continuously generated analytics results efficiently through metadata manipulation without physical data moving and copying.
We have prototyped our approach by extending the PostgreSQL, resulting in a new kind of tightly integrated, highly efficient platform for providing CSA service. We tested the throughput and latency of this service using a well-known stream processing benchmark and with WebOS based Palm phones. The test results show that the proposed approach is highly competitive. Providing CSA cloud service using HP Neoview parallel database engine is currently explored.
KeywordsCloud Service Mobile Application Stream Processing Query Execution Continuous Query
Unable to display preview. Download preview PDF.
- 1.Arasu, A., Babu, S., Widom, J.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 15(2) (June 2006) Google Scholar
- 2.Chandrasekaran, S., et al.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: CIDR 2003 (2003)Google Scholar
- 3.Chen, Q., Hsu, M.: Experience in Extending Query Engine for Continuous Analytics, Tech. Rep. HPL-2010-44 (2010)Google Scholar
- 4.Chen, Q., Hsu, M.: Cooperating SQL Dataflow Processes for In-DB Analytics. In: Proc. CoopIS 2009 (2009)Google Scholar
- 5.Chen, Q., Therber, A., Hsu, M., Zeller, H., Zhang, B., Wu, R.: Efficiently Support Map-Reduce alike Computation Models Inside Parallel DBMS. In: Proc. Thirteenth International Database Engineering & Applications Symposium (IDEAS 2009) (2009)Google Scholar
- 6.Chen, Q., Hsu, M., Liu, R.: Extend UDF Technology for Integrated Analytics. In: Proc. DaWaK 2009 (2009)Google Scholar
- 7.Dean, J.: Experiences with MapReduce, an abstraction for large-scale computation. In: Int. Conf. on Parallel Architecture and Compilation Techniques. ACM, New York (2006)Google Scholar
- 8.DeWitt, D.J., Paulson, E., Robinson, E., Naughton, J., Royalty, J., Shankar, S., Krioukov, A.: Clustera: An Integrated Computation And Data Management System. In: VLDB 2008 (2008)Google Scholar
- 9.Franklin, M.J., et al.: Continuous Analytics: Rethinking Query Processing in a Network-Effect World. In: CIDR 2009 (2009)Google Scholar
- 10.Gedik, B., Andrade, H., Wu, K.-L., Yu, P.S., Doo, M.C.: SPADE: The System S Declarative Stream Processing Engine. In: ACM SIGMOD 2008 (2008)Google Scholar
- 11.Jain, N., et al.: Design, Implementation, and Evaluation of the Linear Road Benchmark on the Stream Processing Core. In: SIGMOD (2006)Google Scholar
- 12.Liarou, E., et al.: Exploiting the Power of Relational Databases for Efficient Stream Processing. In: EDBT 2009 (2009)Google Scholar