Abstract
Recent years, writing-intensive workloads on big data make log-structured style storage popular in distributed data storage systems, which provides both large-volume storage capacity and high-performance data updates. Rapidly generating valid keys for append records can significantly improve the data write performance of log-structured storage systems. In distributed and high concurrency environment, however, both the huge disk IO and the interaction overhead of a traditional lock manager limit the transactional throughput for generating auto-increment keys. In this paper, we design an efficient auto-increment keys generation manager (AKGM), a memory management structure that cannot only avoid disk IO but also eliminate the interaction overhead of traditional lock manager for transactions of generating auto-increment keys. We also propose a protocol called adaptive batch processing (ABP), which enables systems implementing AKGM to achieve high transactional throughput even under high contention workloads. We implement these protocols in an open-source database based on log-structured storage, and our experimental results show the superior performance of AKGM and ABP.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hbase website. http://hbase.apache.org/
OceanBase website. https://github.com/alibaba/oceanbase/
SQLite website. https://sqlite.org/
Sysbench website. http://dev.mysql.com/downloads/benchmarks.html/
Agrawal, R., Carey, M.J., Livny, M.: Concurrency control performance modeling: Alternatives and implications. In: Performance of Concurrency Control Mechanisms in Centralized Database Systems, pp. 58–105 (1996)
Bernstein, P.A., Goodman, N.: Concurrency control in distributed database systems. ACM Comput. Surv. 13(2), 185–221 (1981)
Chang, F., Dean, J., et al.: Bigtable: a distributed storage system for structured data. In: Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation, vol. 7, p. 15 (2006)
Comer, D.: The ubiquitous b-tree. ACM Comput. Surv. 11(2), 121–137 (1979)
Diaconu, C., et al.: Hekaton: SQL server’s memory-optimized OLTP engine. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2013, New York, NY, USA, 22–27 June 2013, pp. 1243–1254 (2013)
Gray, J., Reuter, A.: Transaction Processing: Concepts and Techniques. Morgan Kaufmann, Burlington (1993)
Harizopoulos, S., Abadi, D.J., Madden, S., Stonebraker, M.: OLTP through the looking glass, and what we found there. In: SIGMOD Conference, pp. 981–992. ACM (2008)
Johnson, R., Pandis, I., Ailamaki, A.: Improving OLTP scalability using speculative lock inheritance. PVLDB 2(1), 479–489 (2009)
Lakshman, A., Malik, P.: Cassandra: a decentralized structured storage system. Operating Syst. Rev. 44(2), 35–40 (2010)
O’Neil, P.E., Cheng, E., Gawlick, D., O’Neil, E.J.: The log-structured merge-tree (LSM-tree). Acta Inf. 33(4), 351–385 (1996)
Pandis, I., Johnson, R., Hardavellas, N., Ailamaki, A.: Data-oriented transaction execution. PVLDB 3(1), 928–939 (2010)
Ren, K., Thomson, A., Abadi, D.J.: Lightweight locking for main memory database systems. PVLDB 6(2), 145–156 (2012)
Stonebraker, M., Madden, S., Abadi, D.J., Harizopoulos, S., Hachem, N., Helland, P.: The end of an architectural era (it’s time for a complete rewrite). In: Proceedings of the 33rd International Conference on Very Large Data Bases, University of Vienna, Austria, 23–27 September 2007, pp. 1150–1160 (2007)
Thomasian, A.: Two-phase locking performance and its thrashing behavior. In: Performance of Concurrency Control Mechanisms in Centralized Database Systems, pp. 166–214 (1996)
Acknowledgements
The project is partially supported by National Key R&D Plan Project under grant numbers 2016YFB1000905 and 2018YFB1003400, National Science Foundation of China under grant numbers U1401256, 61432006 and 61332006, and Shanghai Agriculture Applied Technology Development Program, China (T20170303).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, J., Guo, J., Zhang, Z., Qian, W., Zhou, A. (2018). Efficient Auto-Increment Keys Generation for Distributed Log-Structured Storage Systems. In: Hacid, H., Cellary, W., Wang, H., Paik, HY., Zhou, R. (eds) Web Information Systems Engineering – WISE 2018. WISE 2018. Lecture Notes in Computer Science(), vol 11234. Springer, Cham. https://doi.org/10.1007/978-3-030-02925-8_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-02925-8_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-02924-1
Online ISBN: 978-3-030-02925-8
eBook Packages: Computer ScienceComputer Science (R0)