Building a YARN Application

Abstract

Hadoop 2.0 allows a developer to plug in to other frameworks. A large number of frameworks have developed around data stored in the HDFS, and some have been covered in this book (HAMA and Spark, for example). Some of these frameworks were developed to overcome the limitations of using MapReduce for all types of problems. For example, the key limitation of MapReduce is that each MapReduce phase reads and writes data to the HDFS, so iterative algorithms run several times slower in MapReduce. Each iteration is a separate MapReduce job that reads the output of the earlier iteration’s MapReduce job from the HDFS. Frameworks such as HAMA and Spark were developed to address these limitations. They leveraged the HDFS, but existed as separate frameworks that had their own resource management capabilities. YARN or Hadoop 2.0 allows these frameworks to be integrated into the larger Hadoop framework, which centralizes resource management across all types of jobs in the cluster.