A Reliable Replica Mechanism for Stream Processing
In Internet of Things, data would be fast generated from massive sensors as real-time data stream, and the replica mechanism is essential to guarantee availability during stream processing. Traditional mechanisms always assume the redundant replicas were exactly correct, but in the practical conditions even slight errors of replica would lead to the calamity for recovery. In this paper, a reliable mechanism is proposed in which space-bounded signature of checkpoint is used for validation during the replica placement. The mechanism has been analyzed theoretically, and also demonstrated by extensive experiments in various conditions.
KeywordsStream processing Replica Availability Space-bounded Signature
This work was supported by the R&D General Program of Beijing Education Commission (No. KM2015_10009007), the Key Young Scholars Foundation for the Excellent Talents of Beijing (No. 2014000020124G011) and Foundation for the Excellent Youth Scholars of North China University of Technology (XN072-006).
- 2.Bockermann, C.: A Survey of the Stream Processing Landscape. Lehrstuhl furk unstliche Intelligenz Technische Universität Dortmund (2014)Google Scholar
- 3.Barlow, M.: Real-Time Big Data Analytics: Emerging Architecture. O’Reilly Media Inc., Sebastopol (2013)Google Scholar
- 4.Ding, W., Han, Y., Wang, J., Zhao, Z.: Feature-based high availability mechanism for extreme aggregation tasks in real-time data stream processing. J. Internet Technol. 14, 327–340 (2013)Google Scholar
- 5.Hwang, J.H., Balazinska, M., Rasin, A., Cetintemel, U., Michael, S., Stan, Z.: High-availability algorithms for distributed stream processing. In: The 21st International Conference on Data Engineering, pp. 779–790 (2005)Google Scholar
- 6.Balazinska, M., Balakrishnan, H., Madden, S.R., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 13–24. ACM (2005)Google Scholar
- 7.Liu, Q., Lui, J.C., He, C., Pan, L., Fan, W., Shi, Y.: SAND: a fault-tolerant streaming architecture for network traffic analytics. In: The 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2014), Atlanta, Georgia, USA, pp. 80–87 (2014)Google Scholar
- 10.Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., Bhagat, N., Mittal, S., Ryaboy, D.: Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM, Snowbird (2014)Google Scholar
- 11.Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In: Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware, pp. 1–9. Springer, New York (2009)Google Scholar
- 17.Ding, W., Zhao, Z., Han, Y.: A framework to improve the availability of stream computing. In: 23rd IEEE International Conference on Web Services (ICWS 2016), pp. 594–601. IEEE (2016)Google Scholar