Service Oriented Computing and Applications

, Volume 13, Issue 2, pp 169–183 | Cite as

Auto-scaling for real-time stream analytics on HPC cloud

  • Yingchao Cheng
  • Zhifeng HaoEmail author
  • Ruichu Cai
Original Research Paper


There are very-high-volume streaming data in the cyber world today. With the popularization of 5G technology, the streaming Big Data grows larger. Moreover, it needs to be analyzed in real time. We propose a new strategy HPC2-ARS to enable streaming services on HPC platforms. This strategy includes a three-tier high-performance cloud computing (HPC2) platform and a novel autonomous resource-scheduling (ARS) framework. The HPC2 platform is our de facto base platform for research on streaming service. It has three components: Tianhe-2 high-performance computer, custom OpenStack cloud computing software, and Apache Storm stream data analytic system. Our ARS framework ensures real-time response on unpredictable and fluctuating stream, especially streaming Big Data in the 5G era. This strategy addresses an essential problem in the convergence of HPC Cloud, Big Data, and streaming service. Specifically, Our ARS framework provides theoretical and practical solutions for resource provisioning, placement, and scheduling optimization. Extensive experiments have validated the effectiveness of the proposed strategy.


Distributed computing High-performance computing Resource management Service computing Stream Utility theory 



This work was supported in part by the Natural Science Foundation of China (NSFC)-Guangdong Joint Fund under Grant U1501254, in part by the NSFC under Grant 61876043 and Grant 61472089, in part by the China Scholarship Council under Grant 201608440336, in part by the Natural Science Foundation of Guangdong under Grant 2014A030306004 and Grant 2014A030308008, in part by the Guangdong High-level Personnel of Special Support Program under Grant 2015TQ01X140, in part by the Guangdong Provincial Key Laboratory of Cyber-Physical System under Grant 2016B030301008, and in part by the Pearl River S&T Nova Program of Guangzhou under Grant 201610010101.


  1. 1.
    Padgavankar MH, Gupta SR (2014) Big data storage and challenges. Int J Comput Sci Inf Technol 5(2):2218–2223 Google Scholar
  2. 2.
    Chen CLP, Zhang CY (2014) Data-intensive applications, challenges, techniques and technologies: a survey on Big Data. Inf Sci 275(11):314–347Google Scholar
  3. 3.
    Fu TZJ, Ding J, Ma RTB, Winslett M, Yang Y, Zhang Z (2015) DRS: dynamic resource scheduling for real-time analytic over fast streams. In: IEEE, international conference on distributed computing systems, vol 690. IEEE, pp 411–420Google Scholar
  4. 4.
    Kambatla K, Kollias G, Kumar V, Grama A (2014) Trends in Big Data analytic. J Parallel Distrib Comput 74(7):2561–2573Google Scholar
  5. 5.
    Khan M, Li M, Ashton P, Taylor G, Liu J (2014). Big Data analytic on PMU measurements. In: International conference on fuzzy systems and knowledge discovery. IEEE. (IEEE Transactions)Google Scholar
  6. 6.
    Ramírez-Gallego S, Krawczyk B, García S, Woźniak M, Herrera F (2017) A survey on data preprocessing for data stream mining: current status and future directions. Neurocomputing 239:39–57Google Scholar
  7. 7.
    Jin CQ, Qian WN, Zhou AY (2004) Analysis and management of streaming data: a survey. J Softw 15(8):1172–1181zbMATHGoogle Scholar
  8. 8.
    Liao X, Xiao L, Yang C, Lu Y (2014) Milkyway-2 supercomputer: system and application. Front Comput Sci 8(3):345–356MathSciNetGoogle Scholar
  9. 9.
    Assunção MD, Calheiros RN, Bianchi S, Netto MA, Buyya R (2015) Big Data computing and clouds: trends and future directions. J Parallel Distrib Comput 79:3–15Google Scholar
  10. 10.
    Rehr JJ, Vila FD, Gardner JP, Svec L, Prange M (2010) Scientific computing in the cloud. Comput Sci Eng 12(3):34–43Google Scholar
  11. 11.
    Kingsbury BK (1986) The network queueing system Tech. Rep. NASA-CR-177433, NASAGoogle Scholar
  12. 12.
    Henderson RL (1995) Job scheduling under the portable batch system. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, Heidelberg, pp 279–294Google Scholar
  13. 13.
    Slapničar P, Seitz U, Bode A, Zoraja I (2001) Resource management in message passing environments. J Comput Inf Technol 9(1):43–54zbMATHGoogle Scholar
  14. 14.
    Litzkow MJ, Livny M, Mutka MW (1988) Condor-a hunter of idle workstations. In: 8th international conference on distributed computing systems, 1988. IEEE, pp 104–111Google Scholar
  15. 15.
    Capit N, Da Costa G, Georgiou Y, Huard G, Martin C, Mounié G et al (2005) A batch scheduler with high level components. In: IEEE international symposium on cluster computing and the grid, 2005. CCGrid 2005, vol 2. IEEE, pp 776–783 Google Scholar
  16. 16.
    Zhou S, Zheng X, Wang J, Delisle P (1993) Utopia: a load sharing facility for large, heterogeneous distributed computer systems. Softw Pract Exp 23(12):1305–1336Google Scholar
  17. 17.
    Newhouse T, Pasquale J (2006) ALPS: an application-level proportional-share scheduler. In: HPDC, pp 279–290Google Scholar
  18. 18.
    Yoo AB, Jette, MA, Grondona M (2003) Slurm: simple linux utility for resource management. In: Workshop on job scheduling strategies for parallel processing. Springer, Berlin, Heidelberg, pp. 44–60Google Scholar
  19. 19.
    Chen M, Mao S, Liu Y (2014) Big data: a survey. Mobile Netw Appl 19(2):171–209Google Scholar
  20. 20.
    Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51(1):107–113Google Scholar
  21. 21.
    Schwarzkopf M, Konwinski A, Abd-El-Malek M, Wilkes J (2013) Omega: flexible, scalable schedulers for large compute clusters. In: Proceedings of the 8th ACM European conference on computer systems. ACM, pp 351–364Google Scholar
  22. 22.
    Verma A, Pedrosa L, Korupolu M, Oppenheimer D, Tune E, Wilkes J (2015) Large-scale cluster management at Google with Borg. In: Proceedings of the tenth European conference on computer systems. ACM, p 18Google Scholar
  23. 23.
    Hindman B, Konwinski A, Zaharia M, Ghodsi A, Joseph AD, Katz RH et al (2011) Mesos: a platform for fine-grained resource sharing in the data center. In: Proceedings of the 8th USENIX conference on Networked systems design and implementation, vol 11, pp 295–308Google Scholar
  24. 24.
    Vavilapalli VK, Murthy AC, Douglas C, Agarwal S. Konar M, Evans R et al (2013) Apache Hadoop YARN: yet another resource negotiator. In: Symposium on cloud computing. ACM, pp 1–16Google Scholar
  25. 25.
    Lin Y, Agrawal D, Chen C, Ooi BC, Wu S (2011) Llama: leveraging columnar storage for scalable join processing in the MapReduce framework. In: Proceedings of the 2011 ACM SIGMOD international conference on management of data. ACM, pp 961–972Google Scholar
  26. 26.
    Saha B, Shah H, Seth S, Vijayaraghavan G, Murthy A, Curino C (2015) Apache tez: a unifying framework for modeling and building data processing applications. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data. ACM, pp 1357–1369Google Scholar
  27. 27.
    Bernstein D (2014) Containers and cloud: from lxc to docker to kubernetes. IEEE Cloud Comput 1(3):81–84Google Scholar
  28. 28.
    Dittrich J, Quiané-Ruiz JA (2012) Efficient big data processing in Hadoop MapReduce. Proc VLDB Endow 5(12):2014–2015Google Scholar
  29. 29.
    Bird SL, Smith BJ (2011) PACORA: performance aware convex optimization for resource allocation. In: Proceedings of the 3rd USENIX workshop on hot topics in parallelismGoogle Scholar
  30. 30.
    Ousterhout K, Wendell P, Zaharia M, Stoica I (2013) Sparrow: distributed, low latency scheduling. In: Proceedings of the twenty-fourth ACM symposium on operating systems principles. ACM, pp 69–84Google Scholar
  31. 31.
    Hirzel M, Soulé R, Schneider S, Gedik B, Grimm R (2014) A catalog of stream processing optimizations. ACM Comput Surv (CSUR) 46(4):46Google Scholar
  32. 32.
    Abadi DJ, Carney D, Çetintemel U, Cherniack M, Convey C, Lee S et al (2003) Aurora: a new model and architecture for data stream management. VLDB J 12(2):120–139Google Scholar
  33. 33.
    Abadi DJ, Ahmad Y, Balazinskaur M, Cetintemel U, Cherniack M, Hwang J-H, Lindner W, Maskey AS, Rasin A, Ryvkina E, Tatbul N, Xing Y, Zdonik S (2005) The design of the borealis stream processing engine. In: 2nd biennial conference on innovative data systems research (CIDR’05)Google Scholar
  34. 34.
    Hormati AH, Choi Y, Woh M, Kudlur M, Rabbah R, Mudge T, Mahlke S (2010) MacroSS: macro-SIMDization of streaming applications. In: ACM SIGARCH computer architecture news, vol 38, no. 1. ACM, pp 285–296Google Scholar
  35. 35.
    Thies W, Karczmarek M, Amarasinghe S (2002) StreamIt: a language for streaming applications. In: International conference on compiler construction. Springer, Berlin, Heidelberg, pp 179–196Google Scholar
  36. 36.
    Welsh M, Culler D, Brewer E (2001) SEDA: an architecture for well-conditioned, scalable internet services. In: ACM SIGOPS operating systems review, vol 35, no. 5. ACM, pp 230–243Google Scholar
  37. 37.
    Arpaci-Dusseau RH, Anderson E, Treuhaft N, Culler DE, Hellerstein JM, Patterson D, Yelick K (1999) Cluster I/O with river: making the fast case common. In: Proceedings of the sixth workshop on I/O in parallel and distributed systems. ACM, pp 10–22Google Scholar
  38. 38.
    Wolf J, Bansal N, Hildrum K, Parekh S, Rajan D, Wagle R et al (2008) SODA: an optimizing scheduler for large-scale stream-based distributed computer systems. In: Proceedings of the 9th ACM/IFIP/USENIX international conference on middleware. Springer, New York, pp 306–325Google Scholar
  39. 39.
    Zaharia M, Xin RS, Wendell P, Das T, Armbrust M, Dave A et al (2016) Apache spark: a unified engine for big data processing. Commun ACM 59(11):56–65Google Scholar
  40. 40.
    Carbone P, Katsifodimos A, Ewen S, Markl V, Haridi S, Tzoumas K (2015) Apache flink: stream and batch processing in a single engine. In: Bulletin of the IEEE computer society technical committee on data engineering, Vol 36Google Scholar
  41. 41.
    Toshniwal A, Taneja S, Shukla A, Ramasamy K, Patel JM, Kulkarni S et al (2014) Storm@ twitter. In: Proceedings of the 2014 ACM SIGMOD international conference on Management of data. ACM, pp 147–156Google Scholar
  42. 42.
    Kulkarni S, Bhagat N, Fu M, Kedigehalli V, Kellogg C, Mittal S et al (2015) Twitter heron: stream processing at scale. In: Proceedings of the 2015 ACM SIGMOD international conference on management of data. ACM, pp 239–250Google Scholar
  43. 43.
    Bitran GR, Morabito R (1996) State-of-the-art survey: open queueing networks: optimization and performance evaluation models for discrete manufacturing systems. Prod Oper Manag 5(2):163–193Google Scholar
  44. 44.
    Armbrust M, Fox A, Griffith R, Joseph AD, Katz R, Konwinski A et al (2010) A view of cloud computing. Commun ACM 53(4):50–58Google Scholar
  45. 45.
    Mathis M, Mahdavi J, Floyd S, Romanow A (1996) TCP selective acknowledgment options (No. RFC 2018)Google Scholar
  46. 46.
    Savitzky A, Golay MJ (1964) Smoothing and differentiation of data by simplified least squares procedures. Anal Chem 36(8):1627–1639Google Scholar
  47. 47.
    “Sahara”. Retrieved 24 September 2014Google Scholar
  48. 48.
    Vavilapalli VK, Murthy AC, Douglas C, Agarwal S, Konar M, Evans R et al (2013) Apache hadoop yarn: yet another resource negotiator. In: Proceedings of the 4th annual symposium on cloud computing. ACM, p 5Google Scholar
  49. 49.
    Pang Z, Xie M, Zhang J, Zheng Y, Wang G, Dong D, Suo G (2014) The TH express high-performance interconnect networks. Front Comput Sci 8(3):357–366MathSciNetGoogle Scholar
  50. 50.
    Sefraoui O, Aissaoui M, Eleuldj M (2012) OpenStack: toward an open-source solution for cloud computing. Int J Comput Appl 55(3):38–42Google Scholar
  51. 51.
    Nguyen DT, Jung JE (2017) Real-time event detection for online behavioral analytic of big social data. Future Gen Comput Syst 66:137–145Google Scholar
  52. 52.
    Aiello LM, Petkos G, Martin C, Corney D, Papadopoulos S, Skraba R et al (2013) Sensing trending topics in twitter. IEEE Trans Multimed 15(6):1268–1282Google Scholar
  53. 53.
    Liu Y, Wang J, Li Z, Li H (2017) Efficient logo recognition by local feature groups. Multimed Syst 23(3):1–9Google Scholar
  54. 54.
    Romberg S, Pueyo LG, Lienhart R, Zwol RV (2011) Scalable logo recognition in real-world images. In: ACM international conference on multimedia retrieval. ACM, pp 25Google Scholar
  55. 55.
    Yun U (2007) Mining lossless closed frequent patterns with weight constraints. Knowl-Based Syst 20(1):86–97Google Scholar
  56. 56.
    Chen Y, Tu L (2007) Density-based clustering for real-time stream data. In: ACM SIGKDD international conference on knowledge discovery and data mining, San Jose, California, USA, August. DBLP, pp 133–142Google Scholar
  57. 57.
    Aggarwal CC, Han J, Wang J, Yu PS (2003) A framework for clustering evolving data streams. In: Proceedings of VLDB, pp 81–92Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of ComputersGuangdong University of TechnologyGuangzhouChina
  2. 2.School of Mathematics and Big DataFoshan UniversityFoshanChina
  3. 3.Department of StatisticsTexas A&M UniversityCollege StationUSA

Personalised recommendations