Consistency Benchmarking: Evaluating the Consistency Behavior of Middleware Services in the Cloud

  • Markus Klems
  • Michael Menzel
  • Robin Fischer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6470)


Cloud service providers such as Amazon Web Services offer a set of next-generation storage and messaging middleware services that can be utilized on-demand over the Internet. Outsourcing software into the cloud, however, confronts application developers with the challenge of understanding the behavior of distributed systems, which are out of their control. This work proposes an approach to benchmark the consistency behavior of services by example of Amazon Simple Queue Service (SQS), a hosted, Web-scale, distributed message queue that is exposed as a Web service. The data of our consistency benchmarking tests are evaluated with the metric harvest as described by Fox and Brewer (1999). Our tests with SQS indicate that the client-service interaction intensity has an influence on harvest.


cloud computing distributed systems service-oriented computing 


  1. 1.
    Bernstein, P.A.: Middleware: A model for distributed system services. ACM Commun 39(2) (1996)Google Scholar
  2. 2.
    Fox, A., Brewer, E.A.: Harvest, yield, and scalable tolerant systems. In: Proceedings of the 7th Workshop on Hot Topics in Operating Systems (1999)Google Scholar
  3. 3.
    Gilbert, S., Lynch, N.: Brewer’s conjecture and the feasibility of consistent available partition-tolerant web services. ACM SIGACT News 3(2), 51–59 (2002)CrossRefGoogle Scholar
  4. 4.
    Vogels, W.: Eventually consistent. ACM Comm. 52(1), 40–44 (2009)CrossRefGoogle Scholar
  5. 5.
    Pritchett, D.: BASE: An ACID alternative. Queue 6(3), 48–55 (2008)CrossRefGoogle Scholar
  6. 6.
    Amazon Web Services LLC: Amazon Simple Queue Service developer guide - API version 2009-02-01 (2010)Google Scholar
  7. 7.
    Garfinkel, S.: An evaluation of Amazon’s grid computing services: EC2, S3 and SQS. Tech. Rep. TR-08-07, Harvard University (2007)Google Scholar
  8. 8.
    Binnig, C., Kossmann, D., Kraska, T., Loesing, S.: How is the Weather tomorrow? Towards a Benchmark for the Cloud. In: DBTest Workshop (ACM SIGMOD), Providence, USA (2009)Google Scholar
  9. 9.
    Yu, H., Vahdat, A.: Design and evaluation of a conit-based continuous consistency model for replicated services. ACM Transactions on Computer Systems 20(3), 239–282 (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Markus Klems
    • 1
  • Michael Menzel
    • 2
  • Robin Fischer
    • 1
  1. 1.Karlsruhe Institute of Technology (KIT)KarlsruheGermany
  2. 2.FZI Forschungszentrum InformatikKarlsruheGermany

Personalised recommendations