Cluster Computing

, Volume 22, Issue 3, pp 911–928 | Cite as

Analysis model for server consolidation of virtualized heterogeneous data centers providing internet services

  • Bo WangEmail author
  • Ying Song
  • Yuzhong Sun
  • Jun Liu


Server consolidation based on virtualization technology simplifies system administration, reduces the cost of power and physical infrastructure, and improves resource utilizations in today’s service-oriented Internet data centers. How many servers for the underlying physical infrastructure are saved via server consolidation in virtualized data centers is of great interest to the administrators and designers of the data centers. Various workload consolidations differ in saving physical servers for the infrastructure. The impacts caused by virtualization to these concurrent services are fluctuating considerably which may have a great effect on server consolidation. This paper proposes an analytic model for server consolidation in virtualized Internet data centers based on the queuing theory. According to the features of these services’ workloads, this model can provide the supremum number of consolidated physical servers needed to guarantee QoS with same loss probabilities of requests as in dedicated servers. We verify the model via a case study. The experiments results confirm the superior accuracy of our model and show that the virtual machine-based server consolidation saves up to 50% physical infrastructure and improves 50% CPU resource utilization as well as 2.67 times in I/O bandwidth utilization, satisfying required QoS.


Analysis model Queuing theory Optimization Server consolidation Virtualization 



The research was supported in part by National Science Foundation of China under Grant Nos. 61202060, 912183001, 61173112 and 61221062; National High Technology Research and Development Program 863 of China under Grant No. 2013AA01A212; The Ministry of Education Innovation Research Team No. IRT13035; Key Projects in the National Science and Technology Pillar Program under Grant No. 2012BAH16F02.


  1. 1.
    Gupta, D., Lee, S., Vrable, M., Savage, S., Snoeren, Alex C., Varghese, George, Voelker, Geoffrey M., Vahdat, Amin: Difference engine: harnessing memory redundancy in virtual machines. Commun. ACM 53(10), 85–93 (2010)CrossRefGoogle Scholar
  2. 2.
    Willmann, P., Rixner, S., Cox, A.L.: Protection strategies for direct access to virtualized I/O devices. In: USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC’08, pp. 15–28. (2008)Google Scholar
  3. 3.
    Cui, L., Li, J., Li, B., Huai, J., Hu, C., Wo, T., Al-Aqrabi, H., Liu, L..: VMScatter: migrate virtual machines to many hosts. In: Proceedings of the 9th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments, VEE ’13, pp. 63–72, ACM, New York, NY, USA (2013)Google Scholar
  4. 4.
    Guo, Y., Stolyar, A.L., Walid, A.: Shadow-routing based dynamic algorithms for virtual machine placement in a network cloud. In: 2013 Proceedings of the IEEEINFOCOM, pp. 620–628 (2013)Google Scholar
  5. 5.
    Song, Y., Wang, H., Li, Y., Feng, B., Sun, Y.: Multi-tiered on-demand resource scheduling for vm-based data center. In: Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, CCGRID ’09, pp. 148–155. IEEE Computer Society, Washington, DC, USA (2009)Google Scholar
  6. 6.
    Song, Y., Sun, Y., Shi, W.: A two-tiered on-demand resource allocation mechanism for vm-based data centers. IEEE Trans. Serv. Comput. 6(1), 116–129 (2013)CrossRefGoogle Scholar
  7. 7.
    Padala, P., Shin, K.G., Zhu, X., Uysal, M., Wang, Z., Singhal, S., Merchant, A., Salem, K.: Adaptive control of virtualized resources in utility computing environments. In: Proceedings of the 2Nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, EuroSys ’07, pp. 289–302. ACM, New York, NY, USA (2007)Google Scholar
  8. 8.
    Vasić, N., Novaković, D., Miučin, S., Kostić, D., Bianchini, R.: DejaVu: Accelerating resource allocation in virtualized environments. In: Proceedings of the Seventeenth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XVII, pp. 423–436. ACM, New York, NY, USA (2012)Google Scholar
  9. 9.
    Wood, T., Cherkasova, L., Ozonat, K., Shenoy, P.: Profiling and modeling resource usage of virtualized applications. In: Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware, Middleware ’08, pp. 366–387. Springer, New York, NY, USA (2008)Google Scholar
  10. 10.
    Chiang, R.C., Hwang, J., Huang, H.H., Wood, T.: Matrix: achieving predictable virtual machine performance in the clouds. In: 11th International Conference on Autonomic Computing (ICAC 14), pp. 45–56, Philadelphia, PA (2014)Google Scholar
  11. 11.
    Kundu, S., Rangaswami, R., Dutta, K., Zhao, M.: Application performance modeling in a virtualized environment. In: 2010 IEEE 16th International Symposium on High Performance Computer Architecture (HPCA), pp. 1–10. (2010)Google Scholar
  12. 12.
    Kundu, S., Rangaswami, R., Gulati, A., Zhao, M., Dutta, K.: Modeling virtualized applications using machine learning techniques. In: Proceedings of the 8th ACM SIGPLAN/SIGOPS Conference on Virtual Execution Environments, VEE ’12, pp. 3–14. ACM, New York, NY, USA (2012)Google Scholar
  13. 13.
    Song, Y., Zhang, Y., Sun, Y., Shi, W.: Utility analysis for internet-oriented server consolidation in VM-based data centers. In: 2009 IEEE International Conference on Cluster Computing and Workshops, CLUSTER ’09, pp. 1–10. (2009)Google Scholar
  14. 14.
    Gross, D., Harris, C.M.: Fundamentals of Queueing Theory. Wiley, New York, NY (1988)zbMATHGoogle Scholar
  15. 15.
    Heidelberger, P., Lavenberg, Stephen S: Computer performance evaluation methodology. Comput. IEEE Trans. C–33(12), 1195–1220 (1984)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Smith, J.E.: Characterizing computer performance with a single number. Commun. ACM 31(10), 1202–1206 (1988)CrossRefGoogle Scholar
  17. 17.
    Lilja, David J.: Measuring Computer Performance: A Practitioner’s Guide. Cambridge University Press, Cambridge (2000)CrossRefGoogle Scholar
  18. 18.
    John, L.K.: More on finding a single number to indicate overall performance of a Benchmark suite. SIGARCH Comput. Archit. News 32(1), 3–8 (2004)CrossRefGoogle Scholar
  19. 19.
    Krishnaswamy, U., Scherson, I.D.: A Framework for computer performance evaluation using benchmark sets. Comput. IEEE Trans. 49(12), 1325–1338 (2000)CrossRefGoogle Scholar
  20. 20.
    Chen, T., Chen, Y., Guo, Q., Temam, O., Wu, Y., Hu, W.: Statistical performance comparisons of computers. In: 2012 IEEE 18th International Symposium on High Performance Computer Architecture (HPCA), pp. 1–12 (2012)Google Scholar
  21. 21.
    Chen, T., Guo, Q., Temam, O., Wu, Y., Bao, Y., Xu, Z., Chen, Y.: Statistical performance comparisons of computers. Comput. IEEE Trans. 64, 1442–1455 (2014)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Cao, J., Andersson, M., Nyberg, C., Kihl, M.: Web server performance modeling using an M/G/1/K*PS queue. In: 10th International Conference on Telecommunications, ICT 2003, vol. 2, pp. 1501–1506. (2003)Google Scholar
  23. 23.
    Liu, X., Heo, J., Sha, L.: Modeling 3-tiered web applications. In: Proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, 2005, pp. 307–310. (2005)Google Scholar
  24. 24.
    Ghaith, S., Wang, M., Perry, P., Murphy, L.: Software contention aware queueing network model of three-tier web systems. In: Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering, ICPE ’14, pp. 273–276. ACM, New York, NY, USA (2014)Google Scholar
  25. 25.
    Malkowski, S., Hedwig, M., Pu, C.: Experimental evaluation of N-tier systems: observation and analysis of multi-bottlenecks. In: IEEE International Symposium on Workload Characterization, IISWC 2009, pp. 118–127. IEEE (2009)Google Scholar
  26. 26.
    Lim, S.-H., Huh, J.-S., Kim, Y., Shipman, G.M., Das, C.R.: D-factor: a quantitative model of application slow-down in multi-resource shared systems. In: Proceedings of the 12th ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS ’12, pp. 271–282. CM, New York, NY, USA (2012)Google Scholar
  27. 27.
    Yao, J., Jung, G.: Bottleneck detection and solution recommendation for cloud-based multi-tier application. In: Xavier, F., Aditya, G., Grace, L., Sami, B. (eds.) Service-Oriented Computing. Lecture Notes in Computer Science, pp. 470–477. Springer, Berlin (2014)CrossRefGoogle Scholar
  28. 28.
    Padala, P., Zhu, X., Wang, Z., Singhal, S., Shin, K.G, et al.: Performance Evaluation of Virtualization Technologies for Server Consolidation. HP Labs Tec. Report (2007)Google Scholar
  29. 29.
    Deshane, T., Shepherd, Z., Matthews, J., Ben-Yehuda, M., Shah, A., Rao, B.: Quantitative Comparison of Xen and KVM, pp. 1–2. Xen Summit, Boston (2008)Google Scholar
  30. 30.
    Scheepers, M.J.: Virtualization and containerization of application infrastructure: A Comparison. In: 21st Twente Student Conference on IT, Enschede, The Netherlands (2014)Google Scholar
  31. 31.
    Iosup, A., Ostermann, S., Yigitbasi, M.N., Prodan, R., Fahringer, T., Epema, D.H.J.: Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans. Parallel Distrib. Syst. 22(6), 931–945 (2011)CrossRefGoogle Scholar
  32. 32.
    Gupta, D., Gardner, R., Cherkasova, L.: XenMon: QoS monitoring and performance profiling tool. Technical report, HP Labs, HPL-2005-187 (2005)Google Scholar
  33. 33.
    Benevenuto, F., Fernandes, C., Santos, M., Almeida, V., Almeida, J., Janakiraman, G., Santos, J.: Performance models for virtualized applications. In: Frontiers of High Performance Computing and Networking–ISPA 2006 Workshops, pp. 427–439. Springer (2006)Google Scholar
  34. 34.
    Menascé, D.A.: Virtualization: concepts, applications, and performance modeling. In: International Conference on CMG, pp. 407–414 (2005)Google Scholar
  35. 35.
    Huber, N., von Quast, M., Brosig, F., Kounev, S.: Analysis of the performance-influencing factors of virtualization platforms. In: OTM 2010 on the On the Move to Meaningful Internet Systems, pp. 811–828. Springer, Berlin (2010)Google Scholar
  36. 36.
    Huber, N., von Quast, M., Brosig, F., Hauck, M., Kounev, S.: A method for experimental analysis and modeling of virtualization performance overhead. In: Cloud Computing and Services Science, pp. 353–370. Springer, New York (2012)Google Scholar
  37. 37.
    Minhas, U.F., Yadav, J., Aboulnaga, A., Salem, K.: Database systems on virtual machines: how much do you lose? In: IEEE 24th International Conference on Data Engineering Workshop, 2008. ICDEW 2008, pp. 35–41 (2008)Google Scholar
  38. 38.
    Zhang, S., Qian, Z., Luo, Z., Wu, J., Lu, S.: Burstiness-aware resource reservation for server consolidation in computing clouds. IEEE Trans. Parallel Distrib. Syst. 27(4), 964–977 (2016)CrossRefGoogle Scholar
  39. 39.
    He, L., Zou, D., Zhang, Z., Chen, C., Jin, Hai, Jarvis, Stephen A.: Developing resource consolidation frameworks for moldable virtual machines in clouds. Future Gener. Comput. Syst. 32, 69–81 (2014)CrossRefGoogle Scholar
  40. 40.
    Speitkamp, B., Bichler, Martin: A mathematical programming approach for server consolidation problems in virtualized data centers. IEEE Trans. Serv. Comput. 3(4), 266–278 (2010)CrossRefGoogle Scholar
  41. 41.
    Chiang, R.C., Huang, H.H.: Profiling-based workload consolidation and migration in virtualized data centers. IEEE Trans. Parallel Distrib. Syst. 26(3), 878–890 (2015)CrossRefGoogle Scholar
  42. 42.
    Ferreto, T.C., Netto, M.A.S., Calheiros, R.N., Rose, C.A.F.: Server consolidation with migration control for virtualized data centers. Future Generat. Comput. Syst. 27(8), 1027–1034 (2011)CrossRefGoogle Scholar
  43. 43.
    Srikantaiah, S., Kansal, A., Zhao, F.: Energy aware consolidation for cloud computing. In: Proceedings of the 2008 Conference on Power Aware Computing and Systems, HotPower’08 (2008)Google Scholar
  44. 44.
    Nathuji, R., Schwan, K.: VirtualPower: coordinated Power Management in Virtualized Enterprise Systems. In: Proceedings of Twenty-first ACM SIGOPS Symposium on Operating Systems Principles, SOSP ’07, pp. 265–278. ACM, New York, NY, USA (2007)Google Scholar
  45. 45.
    Beloglazov, A., Buyya, R.: Managing overloaded hosts for dynamic consolidation of virtual machines in cloud data centers under quality of service constraints. IEEE Trans. Parallel Distrib. Syst. 24(7), 1366–1379 (2013)CrossRefGoogle Scholar
  46. 46.
    Medina, V., García, Juan Manuel: A survey of migration mechanisms of virtual machines. ACM Comput. Surv. 46(3), 30:1–30:33 (2014)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Verma, A., Ahuja, P., Neogi, A.: pMapper: Power and migration cost aware application placement in virtualized systems. In: Proceedings of the 9th ACM/IFIP/USENIX International Conference on Middleware, Middleware ’08, pp. 243–264. Springer, New York, USA (2008)Google Scholar
  48. 48.
    Strunk, A., Dargie, W.: Does live migration of virtual machines cost energy? In: 2013 IEEE 27th International Conference on Advanced Information Networking and Applications (AINA), pp. 514–521 (2013)Google Scholar
  49. 49.
    Liu, H., Xu, C.-Z., Jin, H., Gong, J., Liao, X.: Performance and energy modeling for live migration of virtual machines. In: Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC ’11, pp. 171–182. ACM, New York, NY, USA (2011)Google Scholar
  50. 50.
    Fei, X., Fangming, L., Hai, J., Vasilakos, A.V.: Managing performance overhead of virtual machines in cloud computing. Proc. IEEE Surv. State Art Future Dir. 102, 11–31 (2014)Google Scholar
  51. 51.
    Menasce, D.: Two-level iterative queuing modeling of software contention. In: Proceedings of the 10th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunications Systems, 2002, pp. 267–276. MASCOTS (2002)Google Scholar
  52. 52.
    Rao, L., Liu, X., Xie, L., Liu, W.: Coordinated energy cost management of distributed internet data centers in smart grid. IEEE Trans. Smart Grid 3(1), 50–58 (2012)CrossRefGoogle Scholar
  53. 53.
    Lin, M., Wierman, A., Andrew, L.L.H., Thereska, E.: Dynamic right-sizing for power-proportional data centers. IEEE/ACM Trans. Netw 21(5), 1378–1391 (2013)CrossRefGoogle Scholar
  54. 54.
  55. 55.
    Storn, R., Price, Kenneth: Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11(4), 341–359 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  56. 56.
    Deb, Kalyanmoy: Multi-objective Optimization Using Evolutionary Algorithms. Wiley, Hoboken (2001)zbMATHGoogle Scholar
  57. 57.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield, A.: Xen and the art of virtualization. In: Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP ’03, pp. 164–177. ACM, New York, NY, USA (2003)Google Scholar
  58. 58.
    Cain, H.W., Rajwar, R., Marden, M., Lipasti, M.H.: An Architectural Evaluation of Java TPC-W. In: Proceedings of the The Seventh International Symposium on High-Performance Computer Architecture, HPCA, pp. 229–240 (2001)Google Scholar
  59. 59.
    Cooper, B.F, Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, pp. 143–154. ACM, New York, NY, USA (2010)Google Scholar
  60. 60.
    SysBench: a System Performance Benchmark. (2015)
  61. 61.
    MySQL: The World’s Most Popular Open-Source Database. (2015)
  62. 62.
    Korf, R.E.: A new algorithm for optimal bin packing. In: Proceedings of the Eighteenth National Conference on Artificial Intelligence, pp. 731–736. American Association for Artificial Intelligence Menlo Park, CA, USA (2002)Google Scholar
  63. 63.
    Meisner, D., Sadler, C.M., Barroso, L.A., Weber, W.-D., Wenisch, T.F.: Power management of online data-intensive services. In: 38th Annual International Symposium on Computer Architecture (ISCA), 2011, pp. 319–330. IEEE (2011)Google Scholar
  64. 64.
    Rameshan, N., Navarro, L., Monte, E., Vlassov, V.: Stay-away, protecting sensitive applications from performance interference. In: Proceedings of the 15th International Middleware Conference, Middleware ’14, pp. 301–312. ACM, New York, NY, USA (2014)Google Scholar
  65. 65.
    Xu, F., Liu, F., Liu, L., Jin, H., Li, B., Li, B.: iAware: making live migration of virtual machines interference-aware in the cloud. IEEE Trans. Comput. 63(12), 3012–3025 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  66. 66.
    Chiang, R.C., Huang, H.H.: TRACON: interference-aware scheduling for data-intensive applicationsin virtualized environments. IEEE Trans. Parallel Distrib. Syst. 25(5), 1349–1358 (2014)CrossRefGoogle Scholar
  67. 67.
    Novaković, D., Vasić, N., Novaković, S., Kostić, D., Bianchini, R.: DeepDive: transparently identifying and managing performance interference in virtualized environments. In: Proceedings of the 2013 USENIX Conference on Annual Technical Conference., pp. 219–230. USENIX ATC’13 (2013)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Software Engineering CollegeZhengzhou University of Light IndustryZhengzhouPeople’s Republic of China
  2. 2.Shaanxi Province Key Lab. of Satellite and Terrestrial Network Tech. R&D, Department of Computer Science and TechnologyXi’an Jiaotong UniversityXi’anPeople’s Republic of China
  3. 3.Computer SchoolBeijing Information Science and Technology UniversityBeijingPeople’s Republic of China
  4. 4.Key Laboratory of Computer System and Architecture, Institute of Computing TechnologyChinese Academy of SciencesBeijingPeople’s Republic of China

Personalised recommendations