SQL TVF Controlling Forms - Express Structured Parallel Data Intensive Computing

Abstract

A key issue in supporting the synthesis of data intensive computation and data management is to liberate users from low-level parallel programming, by specifying applications functionally independent of the underlying server infrastructure, and further, by providing high-level primitives to express the control flow of applying functions to data partitions. Currently only few such primitives, e.g. Map-Reduce and Cross-Apply, are available, and their expressive power is limited to “flat parallel computing”. To deal with “structured parallel computing” where a function is applied to multiple objects with execution order dependencies, a general framework for creating and combining such primitives is required.

We propose the SQL-FCF framework as the database centric solution to the above problem. We embed into SQL queries the Function Controlling Forms (FCFs) to specify the flow control of applying Table Valued Functions (TVFs) to multiple data partitions. We further support the extensibility of this framework by allowing new FCFs to be defined from existing ones with SQL phrases. Based on this approach, we provided a SQL based high-level interface for “structured parallel computing” in architecting a hydrologic scientific computation platform. Envisioning that the simple parallel computing primitives will evolve and form a general framework, our effort is a step towards that goal.