Data Preparation

Abstract

In the practical examples so far we have read data from CSV files, placed it into SQL queries, and inserted it into a database. In general, this three step process is known as ETL for Extract, Transform, Load. Extract means getting data out of some non-database file. Transform means converting it to match our ontology and type system. Load means loading it into the database. ETL is often performed on massive scales, with many computers working on the various steps on multiple data sources simultaneously. For example, this happens when a transport client sends you a hard disc with a terabyte of traffic sensor data on it. In the “big data” movement, the transformation step might not be so important, as the philosophy here is to worry about ontology only at runtime, and store the data in whatever form you can manage when it arrives.