Applications of Big Data

Abstract

Over the last few decades, big business houses in various disparate areas have been accumulating data from different departments in various formats and have been struggling to correlate the datasets and make any valuable business decisions. The key stumbling block has been the inability of the available systems to process large data when the data are part structured and part unstructured. As witnessed in the previous chapters, the technology strides made over the last few years have broken the stigma of processing large datasets and have enabled mining and analysis of large data. Corporations in the data warehousing space have seen this trend as the next big opportunity to help their clients mine their historical data and help further their businesses in terms of adding strategic and tactical value based on the insights gained from their accumulated data over decades. In this chapter, we will see typical examples of how different businesses analyze their data and enhance their business objectives. We will present some examples in the fields of financial services, retail, manufacturing, telecommunications, social media, and health care.

Exercises

• 1. Write an application to recommend new music labels for users based on the historical listening profiles of the user base. The basket analysis can use the dataset that is available at http://ocelma.net/MusicRecommendationDataset/lastfm-360K.html.

• 2. Yahoo! Messenger is a popular instant messaging application used by many users to communicate to their friends. A sample dataset of so-called friends graph or the social network is available at http://webscope.sandbox.yahoo.com/catalog.php?datatype=g titled “Yahoo! Instant Messenger Friends Connectivity Graph.” Write an application to identify the top 5 users who have most influence in the given social network.

• 3. Visualize the social network of users for the dataset indicated in Exercise 2 above. Use the open-source graph analysis tool called Gephi for this visualization (available at http://gephi.github.io/users/download/. Use the quick start tutorial at http://gephi.github.io/tutorials/ to render and identify the communities for the above dataset.

• 4. Microsoft Corp. has published a dataset which captures the areas of www.microsoft.com that users have visited over a one-week time frame. This dataset is freely available to users at http://kdd.ics.uci.edu/databases/msweb/msweb.html.

  Write an application to predict the areas of www.microsoft.com that a user can visit based on data on what other areas he or she visited.

• 5. Using sentiment analysis concepts/algorithms gained in the earlier chapters, analyze the movie reviews/feedback data available at http://www.kaggle.com/c/sentiment-analysis-on-movie-reviews/data to build a model to predict the positive, negative, and neutral sentiment of the reviewers. Use 75 % of the data for the model and the remaining 25 % of the data to validate the model.

• 6. Using R open-source statistical and data analysis tools, write an application to predict the movement of a stock belonging to DOW Jones. The sample dataset is provided at https://archive.ics.uci.edu/ml/machine-learning-databases/00312/

• 7. Demonstrate with an example how to build a prediction model based on Naive Bayes for text. And then demonstrate with an example using the built model to do text prediction.