Evaluating Docker storage performance: from workloads to graph drivers

  • Vasily TarasovEmail author
  • Lukas Rupprecht
  • Dimitris Skourtis
  • Wenji Li
  • Raju Rangaswami
  • Ming Zhao


Containers are a widely successful technology today popularized by Docker. They improve system utilization by increasing workload density and enable seamless deployment of workloads across development, test, and production environments. Docker’s unique approach to data management, which involves frequent snapshot creation and removal, presents a new set of exciting challenges for storage systems. At the same time, storage management for Docker containers has remained largely unexplored with a dizzying array of solution choices and configuration options. In this paper we unravel the multi-faceted nature of Docker storage and demonstrate its impact on system and workload performance. As we uncover new properties of the popular Docker storage drivers, this is a sobering reminder that widespread use of new technologies can often precede their careful evaluation.


Containers Docker Storage Performance 



This manuscript is an extended version of the paper titled “In Search of the Ideal Storage Configuration for Docker Containers” published in Proceedings of the Workshop on Autonomic Management of Large Scale Container-based Systems (AMLCS) in 2017. We thank the co-authors of the workshop paper Amit Warke, Dean Hildebrand, Mohamed Mohamed, and Nagapramod Mandagere for the contributions to the project. We also thank the AMLCS reviewers for their valuable comments. This work was supported in part by the NSF via Grants CNS-1563883, CNS-1320426, CNS-1562837, and CNS-1619653.


  1. 1.
    451 Research: Application Containers Will Be a \$2.7Bn Market by 2020. Accessed March 2018
  2. 2.
    Control Group v2. Accessed March 2018
  3. 3.
    Biederman, E.: Multiple instances of the global Linux namespaces. In: Linux Symposium (2006)Google Scholar
  4. 4.
    Docker. Accessed March 2018
  5. 5.
    AUFS—Another Union Filesystem. Accessed March 2018
  6. 6.
  7. 7.
    Rodeh, O., Bacik, J., Mason, C.: BTRFS: The Linux B-Tree Filesystem. ACM Trans. Storage 9(3), 9 (2013)CrossRefGoogle Scholar
  8. 8.
    Bonwick, J., Ahrens, M., Henson, V., Maybee, M., Shellenbaum, M.: The Zettabyte File System. In: Proceedings of the 2nd Usenix Conference on File and Storage Technologies (FAST) (2003)Google Scholar
  9. 9.
    Device Mapper Thin Provisioning Targets. Accessed March 2018
  10. 10.
    Harter, T., Salmon, B., Liu, R., Arpaci-Dusseau, A., Arpaci-Dusseau, R.: Slacker: Fast Distribution with Lazy Docker Containers. In: Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST) (2016)Google Scholar
  11. 11.
    Lamourine, M.: Storage Options for Software Containers. ;login: The USENIX Magazine 40(1) (2015)Google Scholar
  12. 12.
    Docker Hub. Accessed March 2018
  13. 13.
    Docker Swarm Mode. Accessed March 2018
  14. 14.
    Kubernetes. Accessed March 2018
  15. 15.
    Juju. Accessed March 2018
  16. 16.
    fabric8. Accessed March 2018
  17. 17.
    ZFS on Linux. Accessed March 2018
  18. 18.
    Solaris Porting Layer. Accessed March 2018
  19. 19.
    Select a Storage Driver. Accessed March 2018
  20. 20.
    Spillane, R.P., Wang, W., Lu, L., Austruy, M., Rivera, R., Karamanolis, C.: Exo-clones: Better Container Runtime Image Management Across the Clouds. In: Proceedings of the 8th USENIX Workshop on Hot Topics in Storage and File Systems (HotStorage) (2016)Google Scholar
  21. 21.
    Amazon EC2 Instance Types. Accessed March 2018
  22. 22.
    Docker and OverlayFS in Practice. Accessed March 2018
  23. 23.
    Aufs Returns ENOSUP when Writing in the Middle of a File on XFS. Accessed March 2018
  24. 24.
    ZFS on Linux for RHEL and CentOS. Accessed March 2018
  25. 25.
    Tarasov, V., Zadok, E., Shepler, S.: Filebench: A Flexible Framework for File System Benchmarking. ;login: The USENIX Magazine 41(1) (2016)Google Scholar
  26. 26.
    Campello, D., Lopez, H., Useche, L., Koller, R., Rangaswami, R.: Non-blocking Writes to Files. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST) (2015)Google Scholar
  27. 27.
    Dockerfile Reference. Accessed March 2018
  28. 28.
    Santana, R., Rangaswami, R., Tarasov, V., Hildebrand, D.: A fast and slippery slope for file systems. ACM SIGOPS Oper. Syst. Rev. 49(2), 27–34 (2016)CrossRefGoogle Scholar
  29. 29.
  30. 30.
    Ubuntu Thinks Btrfs Disk Is Full but It Is Not. Accessed March 2018
  31. 31.
    Btrfs Out of Space, Even Though There Should Be 10% Left. Accessed March 2018
  32. 32.
    Btrfs: No Space Left on Device. Accessed March 2018
  33. 33.
    Eder, J., Murphy, C.: Performance Analysis of Docker on Red Hat Enterprise Linux 7 (2014). Accessed March 2018
  34. 34.
    Eder, J.: Comprehensive overview of storage scalability in Docker (2014). Accessed March 2018
  35. 35.
    Estes, P.: Storage Drivers in Docker: A Deep Dive (2016). Accessed March 2018
  36. 36.
    Simon, R., von Eicken, T.: Perspectives on Docker (2014). Accessed March 2018
  37. 37.
    Tyczynski, W.: 1000 Nodes and Beyond: Updates to Kubernetes Performance and Scalability in 1.2 (2016). Accessed March 2018
  38. 38.
    Sharma, P., Chaufournier, L., Shenoy, P.J., Tay, Y.: Containers and Virtual Machines at Scale: A Comparative Study. In: Proceedings of the 17th International Middleware Conference (Middleware) (2016)Google Scholar
  39. 39.
    Hegde, A., Ghosh, R., Mukherjee, T., Sharma, V.: SCoPe: A Decision System for Large Scale Container Provisioning Management. In: Proceedings of the 9th IEEE International Conference on Cloud Computing (CLOUD) (2016)Google Scholar
  40. 40.
    Jannen, W., Yuan, J., Zhan, Y., Akshintala, A., Esmet, J., Jiao, Y., Mittal, A., Pandey, P., Reddy, P., Walsh, L., Bender, M., Farach-Colton, M., Johnson, R., Kuszmaul, B.C., Porter, D.E.: BetrFS: A Right-Optimized Write-Optimized File System. In: Proceedings of the 13th USENIX Conference on File and Storage Technologies (FAST) (2015)Google Scholar
  41. 41.
    Zhao, F., Xu, K., Shain, R.: Improving Copy-on-Write Performance in Container Storage Drivers. In: Storage Developers Conference (2016)Google Scholar
  42. 42.
    Wu, X., Wang, W., Jiang, S.: TotalCOW: Unleash the Power of Copy-on-Write for Thin-Provisioned Containers. In: Proceedings of the 6th Asia-Pacific Workshop on Systems (APsys) (2015)Google Scholar
  43. 43.
    Canon, R.S., Jacobsen, D.: Shifter: Containers for HPC. In: Cray User Group (2016)Google Scholar
  44. 44.
    Nathan, S., Ghosh, R., Mukherjee, T., Narayanan, K.: CoMICon: A Co-Operative Management System for Docker Container Images. In: Proceedings of the 5th IEEE International Conference on Cloud Engineering (IC2E) (2017)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  • Vasily Tarasov
    • 1
    Email author
  • Lukas Rupprecht
    • 1
  • Dimitris Skourtis
    • 1
  • Wenji Li
    • 2
  • Raju Rangaswami
    • 3
  • Ming Zhao
    • 2
  1. 1.IBM ResearchSan JoseUSA
  2. 2.Arizona State UniversityTempeUSA
  3. 3.Florida International UniversityMiamiUSA

Personalised recommendations