Advertisement

Computing

, Volume 101, Issue 2, pp 97–118 | Cite as

Scalable Byzantine fault-tolerant state-machine replication on heterogeneous servers

  • Michael EischerEmail author
  • Tobias Distler
Article

Abstract

When provided with more powerful or extra hardware, state-of-the-art Byzantine fault-tolerant (BFT) replication protocols are unable to effectively exploit the additional computing resources: on the one hand, in settings with heterogeneous servers existing protocols cannot fully utilize servers with higher performance capabilities. On the other hand, using more servers than the minimum number of replicas required for Byzantine fault tolerance in general does not lead to improved throughput and latency, but instead actually degrades performance. In this paper, we address these problems with Omada, a BFT system architecture that is able to benefit from additional hardware resources. To achieve this property while still providing strong consistency, Omada first parallelizes agreement into multiple groups and then executes the requests handled by different groups in a deterministic order. By varying the number of requests to be ordered between groups as well as the number of groups that a replica participates in between servers, Omada offers the possibility to individually adjust the resource usage per server. Moreover, the fact that not all replicas need to take part in every group enables the architecture to exploit additional servers.

Keywords

Byzantine fault tolerance State-machine replication Scalability Heterogeneity Resource efficiency 

Mathematics Subject Classification

68M15 (Reliability, testing and fault tolerance) 68M14 (Distributed systems) 

Notes

Acknowledgements

This work was partially supported by the German Research Council (DFG) under Grant No. DI 2097/1-2 (“REFIT—Resource-Efficient Fault and Intrusion Tolerance”).

References

  1. 1.
    Aguilera MK, Strom RE (2000) Efficient atomic broadcast using deterministic merge. In: Proceedings of the 19th symposium on principles of distributed computing, pp 209–218Google Scholar
  2. 2.
    Amir Y, Coan B, Kirsch J, Lane J (2007) Customizable fault tolerance for wide-area replication. In: Proceedings of the 26th international symposium on reliable distributed systems, pp 65–82Google Scholar
  3. 3.
    Amir Y, Coan B, Kirsch J, Lane J (2011) Prime: Byzantine replication under attack. In: IEEE transactions on dependable and secure computing, vol 8, no 4, pp 564–577Google Scholar
  4. 4.
    Aublin PL, Mokhtar SB, Quéma V (2013) RBFT: Redundant Byzantine fault tolerance. In: Proceedings of the 33rd international conference on distributed computing systems, pp 297–306Google Scholar
  5. 5.
    Babay A, Amir Y (2016) Fast total ordering for modern data centers. In: Proceedings of the 36th international conference on distributed computing systems, pp 669–679Google Scholar
  6. 6.
    Behl J, Distler T, Kapitza R (2015) Consensus-oriented parallelization: how to earn your first million. In: Proceedings of the 16th middleware conference, pp 173–184Google Scholar
  7. 7.
    Bessani A, Sousa J, Alchieri EEP (2014) State machine replication for the masses with BFT-SMaRt. In: Proceedings of the 44th international conference on dependable systems networks, pp 355–362Google Scholar
  8. 8.
    Castro M, Liskov B (1999) Practical Byzantine fault tolerance. In: Proceedings of the 3rd symposium on operating systems design and implementation, pp 173–186Google Scholar
  9. 9.
    Castro M, Rodrigues R, Liskov B (2003) BASE: using abstraction to improve fault tolerance. In: ACM transactions on computer systems, vol 21, no 3, pp 236–269Google Scholar
  10. 10.
    Clement A, Kapritsos M, Lee S, Wang Y, Alvisi L, Dahlin M, Riche T (2009) UpRight cluster services. In: Proceedings of the 22nd symposium on operating systems principles, pp 277–290Google Scholar
  11. 11.
    Distler T, Kapitza R, Reiser HP (2010) State transfer for hypervisor-based proactive recovery of heterogeneous replicated services. In: Proceedings of the 5th “Sicherheit, Schutz und Zuverlässigkeit” conference, pp 61–72Google Scholar
  12. 12.
    Distler T, Cachin C, Kapitza R (2016) Resource-efficient Byzantine fault tolerance. In: IEEE transactions on computers, vol 65, no 9, pp 2807–2819Google Scholar
  13. 13.
    Garcia M, Bessani A, Gashi I, Neves N, Obelheiro R (2014) Analysis of operating system diversity for intrusion tolerance. In: Software—practice & experience, vol 44, no 6, pp 735–770Google Scholar
  14. 14.
    Hunt P, Konar M, Junqueira F, Reed B (2010) ZooKeeper: wait-free coordination for Internet-scale systems. In: Proceedings of the 2010 USENIX annual technical conference, pp 145–158Google Scholar
  15. 15.
    Junqueira F, Bhagwan R, Hevia A, Marzullo K, Voelker GM (2005) Surviving Internet catastrophes. In: Proceedings of the 2005 USENIX annual technical conference, pp 45–60Google Scholar
  16. 16.
    Kapitza R, Behl J, Cachin C, Distler T, Kuhnle S, Mohammadi SV, Schröder-Preikschat W, Stengel K (2012) CheapBFT: resource-efficient Byzantine fault tolerance. In: Proceedings of the 7th European conference on computer systems, pp 295–308Google Scholar
  17. 17.
    Kapritsos M, Junqueira FP (2010) Scalable agreement: toward ordering as a service. In: Proceedings of the 6th workshop on hot topics in system dependability, pp 7–12Google Scholar
  18. 18.
    Li B, Xu W, Abid MZ, Distler T, Kapitza R (2016) SAREK: optimistic parallel ordering in Byzantine fault tolerance. In: Proceedings of the 12th European dependable computing conference, pp 77–88Google Scholar
  19. 19.
    Mao Y, Junqueira FP, Marzullo K (2008) Mencius: building efficient replicated state machines for WANs. In: Proceedings of the 8th conference on operating systems design and implementation, pp 369–384Google Scholar
  20. 20.
    Ou Z, Zhuang H, Lukyanenko A, Nurminen JK, Hui P, Mazalov V, Ylä-Jääski A (2013) Is the same instance type created equal? Exploiting heterogeneity of public clouds. In: IEEE transactions on cloud computing, vol 1, no 2, pp 201–214Google Scholar
  21. 21.
    Papadimitriou CH, Steiglitz K (1998) Combinatorial optimization: algorithms and complexity. Dover Publications, New YorkzbMATHGoogle Scholar
  22. 22.
    Pease M, Shostak R, Lamport L (1980) Reaching agreement in the presence of faults. J ACM 27(2):228–234MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Veronese GS, Correia M, Bessani AN, Lung LC (2009) Spin one’s wheels? Byzantine fault tolerance with a spinning primary. In: Proceedings of the 28th international symposium on reliable distributed systems, pp 135–144Google Scholar
  24. 24.
    Veronese GS, Correia M, Bessani AN, Lung LC (2010) EBAWA: efficient Byzantine agreement for wide-area networks. In: Proceedings of the 12th symposium on high-assurance systems engineering, pp 10–19Google Scholar
  25. 25.
    Yin J, Martin JP, Venkataramani A, Alvisi L, Dahlin M (2003) Separating agreement from execution for Byzantine fault tolerant services. In: Proceedings of the 19th symposium on operating systems principles, pp 253–267Google Scholar

Copyright information

© Springer-Verlag GmbH Austria, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Friedrich-Alexander University Erlangen-Nürnberg (FAU)ErlangenGermany

Personalised recommendations