Using XQuery for Flat-File Based Scientific Datasets

Abstract

XQuery is a recently developed query language for XML datasets. In this paper, we focus on the use of XQuery and other XML technologies for flat-file based scientific datasets. Traditionally, complex and domain-specific data layouts have complicated the processing of large datasets arising from scientific applications. The use of XML schemas and XQuery’s high-level structure can simplify the analysis on these datasets.

Though scientific data processing applications can be conveniently represented in XQuery, compiling them to achieve efficient execution involves a number of challenges. These are, 1) analysis of recursive functions to identify reduction computations involving only associative and commutative operations, 2) replacement of recursive functions with iterative constructs, 3) application of data-centric transformations on the structure of XQuery, and 4) translation of XQuery processing to an imperative language like C/C++, which is required for using a middleware that offers low-level data access functionality. This paper describes our solutions towards these problems and demonstrates significant benefits from the transformations we have developed.

This work was supported by NSF grant ACR-9982087, NSF CAREER award ACR-9733520, NSF grant ACR-0130437 and NSF grant ACI- 0203846. The equipment used for the experiments reported here was purchased under the grant EIA-9986052.