Future Thoughts: The Big Data Discussion and the Key Roles in Analytics

Abstract

Big data. Hype and more hype. But what is the reality here? Is big data the real deal as many experts would have us believe? For seasoned data mining practitioners, big data has always been the norm rather than the exception. Building predictive models in the late eighties at American Express exposed practitioners to a million cardholder records with hundreds of millions of transaction records. The challenge, then, was to translate this raw source information into a meaningful analytical file that could be used as inputs for developing predictive models. At that time, marketing databases and data warehouses were new to practitioners. The use of crude tools, such as legacy mainframe systems, required a certain level of technical knowledge concerning the use of JCL (job control language) and the storage of data. Accessing data and data output on tape or disk required technical knowledge to optimize the use of data mining in this big data environment.