Abstract
Large numeric matrices and multidimensional data arrays appear in many science domains, as well as in applications of financial and business warehousing. Common applications include eigenvalue determination of large matrices, which decompose into a set of linear algebra operations. With the rise of in-memory databases it is now feasible to execute these complex analytical queries directly in a relational database system without the need of transfering data out of the system and being restricted by hard disc latencies for random accesses. In this paper, we present a way to integrate linear algebra operations and large matrices as first class citizens into an in-memory database following a two-layered architectural model. The architecture consists of a logical component receiving manipulation statements and linear algebra expressions, and of a physical layer, which autonomously administrates multiple matrix storage representations. A cost-based hybrid storage representation is presented and an experimental implementation is evaluated for matrix-vector multiplications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Array Query Language, http://www.xldb.org/arrayql/.
- 2.
Basic Linear Algebra Subprograms, http://www.netlib.org/blas/.
- 3.
Automatically Tuned Linear Algebra Software, http://math-atlas.sourceforge.net/.
- 4.
Intel Math Kernel Library 11.0, http://software.intel.com/en-us/intel-mkl.
- 5.
For illustration purposes we regard a relatively small matrix. Depending on the scenario, the matrices can reach dimensions of up to \(10^{10}\times 10^{10}\).
- 6.
\(T\) is proportional to the number of single element operations, according to the RAM model.
References
Ghoting, A., Krishnamurthy, R., Pednault, E., Reinwald, B., et al.: SystemML: Declarative machine learning on MapReduce. In: ICDE, pp. 231–242. IEEE (2011)
Garcia-Molina, H., Salem, K.: Main memory database systems: an overview. IEEE Trans. Knowl. Data Eng. 4(6), 509–516 (1992)
Brown, P.G.: Overview of SciDB: large scale array storage, processing and analysis. In: SIGMOD, pp. 963–968. ACM (2010)
Kersten, M., Zhang, Y., Ivanova, M., Nes, N.: SciQL, a query language for science applications. In: EDBT/ICDT Workshop on Array Databases, pp. 1–12. ACM (2011)
Cohen, J., Dolan, B., Dunlap, M., Hellerstein, J.M., et al.: MAD skills: new analysis practices for big data. VLDB 2(2), 1481–1492 (2009)
Stonebraker, M., Madden, S., Dubey, P.: Intel “Big Data” science and technology center vision and execution plan. SIGMOD Rec. 42(1), 44–49 (2013)
Coppersmith, D., Winograd, S.: Matrix multiplication via arithmetic progressions. J. Symb. Comput. 9(3), 251–280 (1990)
Vassilevska Williams, V.: Breaking the coppersmith-winograd barrier (2011)
Valsalam, V., Skjellum, A.: A framework for high-performance matrix multiplication based on hierarchical abstractions, algorithms and optimized low-level kernels. CCPE 14(10), 805–839 (2002)
Amossen, R.R., Pagh, R.: Faster join-projects and sparse matrix multiplications. In: ICDT, pp. 121–126. ACM (2009)
Yuster, R., Zwick, U.: Fast sparse matrix multiplication. ACM Trans. Algorithms 1(1), 2–13 (2005)
Buluç, A., Gilbert, J.R.: Parallel sparse matrix-matrix multiplication and indexing: implementation and experiments. SIAM J. Sci. Comput. 34(4), 170–191 (2012)
Yzelman, A.N., Bisseling, R.H.: Cache-oblivious sparse matrix-vector multiplication by using sparse matrix partitioning methods. SIAM J. Sci. Comput. 31(4), 3128–3154 (2009)
Saad, Y.: SPARSKIT: A Basic Tool Kit for Sparse Matrix Computations, Version 2 (1994)
Roth, R.: Importance truncation for large-scale configuration interaction approaches. Phys. Rev. C79, 064324 (2009)
Amossen, R.R., Campagna, A., Pagh, R.: Better size estimation for sparse matrix products. In: Serna, M., Shaltiel, R., Jansen, K., Rolim, J. (eds.) APPROX 2010. LNCS, vol. 6302, pp. 406–419. Springer, Heidelberg (2010)
Morton, G.: A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing. International Business Machines Company (1966)
Hinneburg, A., Keim, D.A.: A general approach to clustering in large databases with noise. Knowl. Inf. Syst. 5(4), 387–415 (2003)
Vary, J.P., Maris, P., Ng, E., Yang, C., Sosonkina, M.: Ab initio nuclear structure the large sparse matrix eigenvalue problem. J. Phys. Conf. Ser. 180(1), 012083 (2009)
Acknowledgements
We want to thank the group of Prof. Roth from the Institute of Theoretical Nuclear Physics at TU Darmstadt (http://theorie.ikp.physik.tu-darmstadt.de/tnp/index.php) for providing us sparse matrix data and algorithmic insights into their domain-specific problem setting.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Kernert, D., Köhler, F., Lehner, W. (2015). Bringing Linear Algebra Objects to Life in a Column-Oriented In-Memory Database. In: Jagatheesan, A., Levandoski, J., Neumann, T., Pavlo, A. (eds) In Memory Data Management and Analysis. IMDM IMDM 2013 2014. Lecture Notes in Computer Science(), vol 8921. Springer, Cham. https://doi.org/10.1007/978-3-319-13960-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-13960-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-13959-3
Online ISBN: 978-3-319-13960-9
eBook Packages: Computer ScienceComputer Science (R0)