The ATree: A Data Structure to Support Very Large Scientific Databases
The datasets generated by satellite observations and super-computer simulations are overwhelming conventional methods of storage and access, leading to unreasonably long delays in data analysis. The major problem that we address is the slow access, from large datasets in archival storage, to small subsets needed for scientific visualization and analysis. The goal is to minimize the amount of storage that has to be read when a subset of the data is needed. A second goal is to enhance the accessibility of data subsets by applying data reduction and indexing methods to the subsets. The reduced format allows larger datasets to be stored on local disk for analysis. Data indexing permits efficient manipulation of the data, and thus improves the productivity of the researcher. A data structure called the ATree is described that meets the demands of interactive scientific applications. The ATree data structure is suitable for storing data abstracts as well as original data. It allows quick access to a subset of interest and is suitable for feature-based queries. It intrinsically partitions the data and organizes the chunks in a linear sequence on secondary/tertiary storage. It can store data at various resolutions and incorporates hierarchical compression methods.
KeywordsTransformation Function Subdivision Scheme Location Code Raster Data Data Chunk
Unable to display preview. Download preview PDF.
- 1.Gaining insight through data visualization - the competitive advantage for data intensive applications — technology white paper. Technical report, Advanced Visual Systems, Inc., 1997.Google Scholar
- 6.Committe on Earth and Environmental Sciences. Our Changing Planet: The FY 1992 U.S. Global Change Research Program. Office of Science and Technology Policy, Washington, D.C., 1991.Google Scholar
- 7.J. A. Fine. Abstracts: A latency-hiding technique for high-capacity mass-storage systems. Technical Report Sequoia 2000 #92/11, University of California, Berkeley, 1992.Google Scholar
- 8.G. M. Morton. A computer oriented geodetic data base and a new technique in file sequencing. IBM Ltd., Ottawa, Canada, 1966.Google Scholar
- 9.J. A. Orenstein and T. H. Merrett. A class of data structures for associative searching. In Proceedings of the Third ACM SIGACT―SIGMOD Symposium on Principles of Database Systems, pages 181–190, Waterloo, Canada, April 1984.Google Scholar
- 10.W. H. Press. Wavelet transforms. Harward-Smitsonian Center for Astrophysics, (Preprint No. 3184), 1991.Google Scholar
- 12.H. Samet. Applications of Spatial Data Structures: Computer Graphics, Image Processing, and GIS. Addison-Wesley, Reading, MA, 1990.Google Scholar
- 13.H. Samet. The Design and Analysis of Spatial Data Structures. Addison-Wesley, Reading, MA, 1990.Google Scholar
- 14.H. Samet and W. G. Aref. Spatial data models and query processing. In Modern Database Systems, The Object Model, Interoperability and Beyond, W. Kim, ed., pages 338–360. ACM Press and Addison-Wesley, New York, NY, 1995.Google Scholar
- 15.H. Shoshani. Properties of statistical and scientific databases. In Statistical and Scientific Databases, Z. Michalewicz, ed. Ellis Horwood, February 1991.Google Scholar
- 16.M. Stonebraker, J. Frew, and J. Dozier. The sequoia 2000 project. In Proceedings of the 3rd International Symposium on Large Spatial Databases, SSD’93, Singapore, June 1993.Google Scholar