Skip to main content
SpringerLink
Log in

Batch Composite Transactions in Stream Processing

  • Chapter
  • First Online:
Transactions on Large-Scale Data- and Knowledge-Centered Systems XXXIV

Part of the book series: Lecture Notes in Computer Science ((TLDKS,volume 10620))

Abstract

Stream processing is about processing continuous streams of data by programs in a workflow. Continuous execution is discretized by grouping input stream tuples into batches and using one batch at a time for the execution of programs. As source input batches arrive continuously, several batches may be processed in the workflow simultaneously. Ensuring correctness of these concurrent executions is important. As in databases and several advanced applications, the transaction concept can be applied to regulate concurrent executions and ensure their correctness in stream processing. The first step is defining transactions corresponding to the executions in a meaningful way. A general requirement in stream processing is that each batch be processed completely in the workflow. That is, all the programs triggered by the batch, directly and transitively, in the workflow must be executed successfully. Then, considering each program execution as a transaction, all the transactions involved in processing a batch can be grouped into a single batch composite transaction, abbreviated as BCT, and transactional properties applied to these BCTs. This works well when a batch is processed individually and completely in isolation. However, when the batches are split, merged or overlapped along the workflow computation, the resulting BCTs will have some transactions in common and applying transactional properties for them becomes complicated. We overcome the problems by defining nonblocking BCTs that have disjoint collections of transactions. They satisfy some properties analogous to those of the database transactions and facilitate (i) defining correctness of concurrent executions in terms of equivalent serial executions of composite transactions and (ii) processing each batch either completely or not at all, and rolling back partially processed batches without affecting the processing of other batches. We also suggest an appropriate roll back mechanism.

This research is supported in part by the Natural Sciences and Engineering Research Council of Canada Discovery Grant 3182.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A subgraph H of an acyclic graph G is a prefix of G if all the edges from H to the rest of the graph are outdirected.

References

  1. Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R.J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., Whittle, S.: The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proc. VLDB Endow. 8(12), 1792–1803 (2015). http://dx.doi.org/10.14778/2824032.2824076

  2. Botan, I., Fischer, P.M., Kossmann, D., Tatbul, N.: Transactional stream processing. In: Proceedings of the 15th International Conference on Extending Database Technology EDBT 2012, pp. 204–215. ACM, New York (2012). http://doi.acm.org/10.1145/2247596.2247622

  3. Conway, N.: Transactions and data stream processing. In: Online Publication, pp. 1–28 (2008). http://neilconway.org/docs/stream_txn.pdf

  4. De Matteis, T., Mencagli, G.: Parallel patterns for window-based stateful operators on data streams: an algorithmic skeleton approach. Int. J. Parallel Prog., pp. 1–20 (2016)

    Google Scholar 

  5. Golab, L., Özsu, M.: Issues in data stream management. ACM SIGMOD Rec. 32(2), 5–14 (2003)

    Article  Google Scholar 

  6. Gürgen, L., Roncancio, C., Labbé, S., Olive, V.: Transactional issues in sensor data management. In: Proceedings of the 3rd International Workshop on Data Management for Sensor Networks (DMSN 2006), Seoul, South Korea, pp. 27–32 (2006)

    Google Scholar 

  7. Hirzel, M., Soulé, R., Schneider, S., Gedik, B., Grimm, R.: A catalog of stream processing optimizations. ACM Comput. Surv. 46(4), 46: 1–46: 34 (2014). http://doi.acm.org/10.1145/2528412

    Article  Google Scholar 

  8. Hummer, W., Satzger, B., Dustdar, S.: Elastic stream processing in the cloud. Wiley Interdisc. Rev. Data Min. Knowl. Disc. 3, 333–345 (2013)

    Article  Google Scholar 

  9. Meehan, J., Tatbul, N., Zdonik, S., Aslantas, C., Cetintemel, U., Du, J., Kraska, T., Madden, S., Maier, D., Pavlo, A., Stonebraker, M., Tufte, K., Wang, H.: S-store: streaming meets transaction processing. Proc. VLDB Endow. 8(13), 2134–2145 (2015)

    Article  Google Scholar 

  10. Mehrotra, S., Rastogi, R., Silberschatz, A., Korth, H.F.: A transaction model formultidatabase systems. In: Proceedings of the 12th International Conference on Distributed Computing Systems 1992, pp. 56–63. IEEE (1992)

    Google Scholar 

  11. Oyamada, M., Kawashima, H., Kitagawa, H.: Continuous query processing with concurrency control: reading updatable resources consistently. In: Proceedings of the 28th Annual ACM Symposium on Applied Computing SAC 2013, pp. 788–794. ACM, New York (2013). http://doi.acm.org/10.1145/2480362.2480514

  12. Schuldt, H., Alonso, G., Beeri, C., Schek, H.: Atomicity and isolation for transactional processes. ACM Trans. Database Syst. 27, 63–116 (2002)

    Article  Google Scholar 

  13. Vidyasankar, K.: Transactional properties of compositions of internet of things services. In: 2015 IEEE First International Smart Cities Conference (ISC2), pp. 1–6, October 2015

    Google Scholar 

  14. Vidyasankar, K.: A transaction model for executions of compositions on internet of things services. In: Procedia Computer Science, pp. 195–202. Elsevier (2016)

    Google Scholar 

  15. Vidyasankar, K.: Atomicity of batches in stream processing. J. Ambient Intell. Humanized Comput. (2017)

    Google Scholar 

  16. Vidyasankar, K., Krishna, P.R., Karlapalem, K.: A multi-level model for activity commitments in e-contracts. In: Meersman, R., Tari, Z. (eds.) OTM 2007. LNCS, vol. 4803, pp. 300–317. Springer, Heidelberg (2007). doi:10.1007/978-3-540-76848-7_20

    Chapter  Google Scholar 

  17. Vidyasankar, K., Vossen, G.: Multi-level modeling of web service compositions with transactional properties. Database Manag. 22(2), 1–31 (2011)

    Article  Google Scholar 

  18. Wang, D., Rundensteiner, E.A., , Ellison III, R.T.: Active complex event processing over event streams. In: Proceedings of the VLDB Endowment, pp. 634–645. ACM Press (2011)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Memorial University, St. John’s, Newfoundland, A1B 3X5, Canada

    K. Vidyasankar

Authors
  1. K. Vidyasankar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. Vidyasankar .

Editor information

Editors and Affiliations

  1. IRIT, Paul Sabatier University, Toulouse, France

    Abdelkader Hameurlain

  2. FAW, University of Linz, Linz, Austria

    Josef Küng

  3. FAW, University of Linz, Linz, Austria

    Roland Wagner

  4. Polytechnic University of Valencia, Valencia, Spain

    Hendrik Decker

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer-Verlag GmbH Germany

About this chapter

Cite this chapter

Vidyasankar, K. (2017). Batch Composite Transactions in Stream Processing. In: Hameurlain, A., Küng, J., Wagner, R., Decker, H. (eds) Transactions on Large-Scale Data- and Knowledge-Centered Systems XXXIV. Lecture Notes in Computer Science(), vol 10620. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-55947-5_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-55947-5_2

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-55946-8

  • Online ISBN: 978-3-662-55947-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation