Spark SQL (Foundations)

Abstract

As Spark evolves as a unified data processing engine with more features in each new release, its programming abstraction also evolves. The RDD was the initial core programming abstraction when Spark was introduced to the world in 2012. In Spark 1.6, a new programming abstraction, called Structured APIs, was introduced. This is the preferred way of performing data processing for the majority of use cases. The Structured APIs were designed to enhance developers’ productivity with easy-to-use, intuitive, and expressive APIs. In this new way of doing data processing, the data needs to be organized into a structured format, and the data computation logic needs to follow a certain structure. Armed with these two pieces of information, Spark can perform optimizations to speed up data processing applications.