Is Distributed Database Evaluation Cloud-Ready?

  • Daniel SeyboldEmail author
  • Jörg Domaschka
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 767)


The database landscape has significantly evolved over the last decade as cloud computing enables to run distributed databases on virtually unlimited cloud resources. Hence, the already non-trivial task of selecting and deploying a distributed database system becomes more challenging. Database evaluation frameworks aim at easing this task by guiding the database selection and deployment decision. The evaluation of databases has evolved as well by moving the evaluation focus from performance to distribution aspects such as scalability and elasticity. This paper presents a cloud-centric analysis of distributed database evaluation frameworks based on evaluation tiers and framework requirements. It analysis eight well adopted evaluation frameworks. The results point out that the evaluation tiers performance, scalability, elasticity and consistency are well supported, in contrast to resource selection and availability. Further, the analysed frameworks do not support cloud-centric requirements but support classic evaluation requirements.


NoSQL Distributed database Database evaluation Cloud 



The research leading to these results has received funding from the EC’s Framework Programme HORIZON 2020 under grant agreement number 644690 (CloudSocket) and 731664 (MELODIC). We thank Moritz Keppler and the Daimler TSS for their valuable and constructive discussions.


  1. 1.
    Agrawal, D., Abbadi, A., Das, S., Elmore, A.J.: Database scalability, elasticity, and autonomy in the cloud. In: Yu, J.X., Kim, M.H., Unland, R. (eds.) DASFAA 2011. LNCS, vol. 6587, pp. 2–15. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-20149-3_2 CrossRefGoogle Scholar
  2. 2.
    Armstrong, T.G., Ponnekanti, V., Borthakur, D., Callaghan, M.: Linkbench: a database benchmark based on the facebook social graph. In: SIGMOD (2013)Google Scholar
  3. 3.
    Barahmand, S., Ghandeharizadeh, S.: Bg: A benchmark to evaluate interactive social networking actions. In: CIDR (2013)Google Scholar
  4. 4.
    Baur, D., Seybold, D., Griesinger, F., Tsitsipas, A., Hauser, C.B., Domaschka, J.: Cloud orchestration features: Are tools fit for purpose? In: UCC (2015)Google Scholar
  5. 5.
    Bermbach, D., Kuhlenkamp, J.: Consistency in distributed storage systems. In: Gramoli, V., Guerraoui, R. (eds.) NETYS 2013. LNCS, vol. 7853, pp. 175–189. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-40148-0_13 CrossRefGoogle Scholar
  6. 6.
    Bermbach, D., Kuhlenkamp, J., Dey, A., Sakr, S., Nambiar, R.: Towards an extensible middleware for database benchmarking. In: Nambiar, R., Poess, M. (eds.) Performance Characterization and Benchmarking: Traditional to Big Data. LNCS, pp. 82–96. Springer, Cham (2015). doi: 10.1007/978-3-319-15350-6_6 CrossRefGoogle Scholar
  7. 7.
    Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with ycsb. In: SoCC (2010)Google Scholar
  8. 8.
    Dey, A., Fekete, A., Nambiar, R., Rohm, U.: Ycsb+t: Benchmarking web-scale transactional databases. In: ICDEW (2014)Google Scholar
  9. 9.
    Difallah, D.E., Pavlo, A., Curino, C., Cudre-Mauroux, P.: Oltp-bench: An extensible testbed for benchmarking relational databases. VLDB 7, 277–288 (2013)Google Scholar
  10. 10.
    Dory, T., Mejias, B., Roy, P., Tran, N.L.: Measuring elasticity for cloud databases. In: Cloud Computing (2011)Google Scholar
  11. 11.
    Friedrich, S., Wingerath, W., Gessert, F., Ritter, N., Pldereder, E., Grunske, L., Schneider, E., Ull, D.: Nosql oltp benchmarking: A survey. In: GI-Jahrestagung (2014)Google Scholar
  12. 12.
    Ghazal, A., Rabl, T., Hu, M., Raab, F., Poess, M., Crolotte, A., Jacobsen, H.A.: Bigbench: towards an industry standard benchmark for big data analytics. In: SIGMOD (2013)Google Scholar
  13. 13.
    Gilbert, S., Lynch, N.: Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM Sigact News 33, 51–59 (2002)CrossRefGoogle Scholar
  14. 14.
    Gray, J.: Benchmark Handbook: For Database and Transaction Processing Systems. Morgan Kaufmann Publishers Inc, San Francisco (1993)zbMATHGoogle Scholar
  15. 15.
    Grolinger, K., Higashino, W.A., Tiwari, A., Capretz, M.A.: Data management in cloud environments: Nosql and newsql data stores. JoCCASA 2, 22 (2013)Google Scholar
  16. 16.
    Khazaei, H., Fokaefs, M., Zareian, S., Beigi-Mohammadi, N., Ramprasad, B., Shtern, M., Gaikwad, P., Litoiu, M.: How do i choose the right NoSQL solution? a comprehensive theoretical and experimental survey. BDIA 2, 1 (2016)Google Scholar
  17. 17.
    Mell, P., Grance, T.: The nist definition of cloud computing. Technical report, National Institute of Standards & Technology (2011)Google Scholar
  18. 18.
    Patil, S., Polte, M., Ren, K., Tantisiriroj, W., Xiao, L., López, J., Gibson, G., Fuchs, A., Rinaldi, B.: Ycsb++: benchmarking and performance debugging advanced features in scalable table stores. In: SoCC (2011)Google Scholar
  19. 19.
    Reniers, V., Van Landuyt, D., Rafique, A., Joosen, W.: On the state of nosql benchmarks. In: ICPE (2017)Google Scholar
  20. 20.
    Sadalage, P.J., Fowler, M.: NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence. Pearson Education, London (2012)Google Scholar
  21. 21.
    Seybold, D., Domaschka, J.: A cloud-centric survey on distributed database evaluation. Technical report. Ulm University (2017)Google Scholar
  22. 22.
    Seybold, D., Wagner, N., Erb, B., Domaschka, J.: Is elasticity of scalable databases a myth? In: IEEE Big Data (2016)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Institute of Information Resource ManagementUlm UniversityUlmGermany

Personalised recommendations