Benchmarking Heterogeneous Cloud Functions
Cloud Functions, often called Function-as-a-Service (FaaS), pioneered by AWS Lambda, are an increasingly popular method of running distributed applications. As in other cloud offerings, cloud functions are heterogeneous, due to different underlying hardware, runtime systems, as well as resource management and billing models. In this paper, we focus on performance evaluation of cloud functions, taking into account heterogeneity aspects. We developed a cloud function benchmarking framework, consisting of one suite based on Serverless Framework, and one based on HyperFlow. We deployed the CPU-intensive benchmarks: Mersenne Twister and Linpack, and evaluated all the major cloud function providers: AWS Lambda, Azure Functions, Google Cloud Functions and IBM OpenWhisk. We make our results available online and continuously updated. We report on the initial results of the performance evaluation and we discuss the discovered insights on the resource allocation policies.
KeywordsCloud computing FaaS Cloud functions Performance evaluation
This work was supported by the National Science Centre, Poland, grant 2016/21/B/ST6/01497.
- 4.Leitner, P., Scheuner, J.: Bursting with possibilities - an empirical study of credit-based bursting cloud instance types. In: 8th IEEE/ACM International Conference on Utility and Cloud Computing, UCC 2015, Limassol, Cyprus, 7–10 December 2015, pp. 227–236 (2015). http://doi.ieeecomputersociety.org/10.1109/UCC.2015.39
- 5.Malawski, M.: Towards serverless execution of scientific workflows - HyperFlow case study. In: WORKS 2016 Workshop, Workflows in Support of Large-Scale Science, in Conjunction with SC 2016 Conference. CEUR-WS.org, Salt Lake City, November 2016Google Scholar
- 7.McGrath, M.G., Short, J., Ennis, S., Judson, B., Brenner, P.R.: Cloud event programming paradigms: applications and analysis. In: 9th IEEE International Conference on Cloud Computing, CLOUD 2016, San Francisco, CA, USA, 27 June – 2 July 2016, pp. 400–406. IEEE Computer Society (2016)Google Scholar
- 9.Spillner, J.: Snafu: Function-as-a-Service (FaaS) runtime design and implementation. CoRR abs/1703.07562 (2017). http://arxiv.org/abs/1703.07562
- 10.Villamizar, M., Garces, O., Ochoa, L., Castro, H., Salamanca, L., Verano, M., Casallas, R., Gil, S., Valencia, C., Zambrano, A., Lang, M.: Infrastructure cost comparison of running web applications in the cloud using AWS lambda and monolithic and microservice architectures. In: 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 179–182, May 2016Google Scholar
- 11.Wagner, B., Sood, A.: Economics of resilient cloud services. In: 1st IEEE International Workshop on Cyber Resilience Economics, August 2016. http://arxiv.org/abs/1607.08508