In Internet of Things, data would be fast generated from massive sensors as real-time data stream, and the replica mechanism is essential to guarantee availability during stream processing. Traditional mechanisms always assume the redundant replicas were exactly correct, but in the practical conditions even slight errors of replica would lead to the calamity for recovery. In this paper, a reliable mechanism is proposed in which space-bounded signature of checkpoint is used for validation during the replica placement. The mechanism has been analyzed theoretically, and also demonstrated by extensive experiments in various conditions.


Stream processing Replica Availability Space-bounded Signature 



This work was supported by the R&D General Program of Beijing Education Commission (No. KM2015_10009007), the Key Young Scholars Foundation for the Excellent Talents of Beijing (No. 2014000020124G011) and Foundation for the Excellent Youth Scholars of North China University of Technology (XN072-006).


  1. 1.
    Ding, W., Han, Y., Wang, J., Zhao, Z.: Feature-based high-availability mechanism for quantile tasks in real-time data stream processing. Softw. Pract. Experience 44, 855–871 (2014)CrossRefGoogle Scholar
  2. 2.
    Bockermann, C.: A Survey of the Stream Processing Landscape. Lehrstuhl furk unstliche Intelligenz Technische Universität Dortmund (2014)Google Scholar
  3. 3.
    Barlow, M.: Real-Time Big Data Analytics: Emerging Architecture. O’Reilly Media Inc., Sebastopol (2013)Google Scholar
  4. 4.
    Ding, W., Han, Y., Wang, J., Zhao, Z.: Feature-based high availability mechanism for extreme aggregation tasks in real-time data stream processing. J. Internet Technol. 14, 327–340 (2013)Google Scholar
  5. 5.
    Hwang, J.H., Balazinska, M., Rasin, A., Cetintemel, U., Michael, S., Stan, Z.: High-availability algorithms for distributed stream processing. In: The 21st International Conference on Data Engineering, pp. 779–790 (2005)Google Scholar
  6. 6.
    Balazinska, M., Balakrishnan, H., Madden, S.R., Stonebraker, M.: Fault-tolerance in the borealis distributed stream processing system. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, pp. 13–24. ACM (2005)Google Scholar
  7. 7.
    Liu, Q., Lui, J.C., He, C., Pan, L., Fan, W., Shi, Y.: SAND: a fault-tolerant streaming architecture for network traffic analytics. In: The 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2014), Atlanta, Georgia, USA, pp. 80–87 (2014)Google Scholar
  8. 8.
    Akidau, T., Balikov, A., Bekiroglu, K., Chernyak, S., Haberman, J., Lax, R., McVeety, S., Mills, D., Nordstrom, P., Whittle, S.: MillWheel: fault-tolerant stream processing at internet scale. Proc. VLDB Endow. 6, 1033–1044 (2013)CrossRefGoogle Scholar
  9. 9.
  10. 10.
    Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., Bhagat, N., Mittal, S., Ryaboy, D.: Storm@twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM, Snowbird (2014)Google Scholar
  11. 11.
    Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In: Proceedings of the 10th ACM/IFIP/USENIX International Conference on Middleware, pp. 1–9. Springer, New York (2009)Google Scholar
  12. 12.
    Miller, G.L.: Riemann’s hypothesis and tests for primality. J. Comput. Syst. Sci. 13, 300–317 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Rabin, M.O.: Probabilistic algorithm for testing primality. J. Number Theory 12, 128–138 (1980)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Agrawal, M., Kayal, N., Saxena, N.: PRIMES is in P. Ann. Math. 160, 781–793 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
  16. 16.
  17. 17.
    Ding, W., Zhao, Z., Han, Y.: A framework to improve the availability of stream computing. In: 23rd IEEE International Conference on Web Services (ICWS 2016), pp. 594–601. IEEE (2016)Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2017

Authors and Affiliations

  1. 1.Data Engineering InstituteNorth China University of TechnologyBeijingChina
  2. 2.Beijing Key Laboratory on Integration and Analysis of Large-Scale Stream DataBeijingChina

Personalised recommendations