Abstract
The analysis of big data represents an important capability for companies and in research and teaching. Data scientists, confronted with complex system configuration and implementation tasks, require affordable and state-of-the-art solutions, which are flexibly configurable to enable diverse analytical research scenarios. In this research, we describe an architecture for the collection, preprocessing, and analysis of social media data based on Hadoop, which we used in a master-level course. We demonstrate how to configure and integrate different components of the Hadoop/Spark ecosystem in order to manage the collection of large data volumes as social media data streams over Web APIs, distributed data storage, the definition of schemas, data preprocessing, and feature extraction, as well as the calculation of descriptive statistics and predictive models. Three exemplary student projects, shortly described in this paper, demonstrate the versatility of the presented solution. Our results can serve as a blueprint for similar endeavors at other educational institutions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An example is the virtual machine provided by Cloudera (http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cloudera_quickstart_vm.html, accessed on January 15, 2016).
- 2.
When connecting the Twitter Streaming API, for example, one terabyte is roughly 5 months of tweet data.
- 3.
https://dev.twitter.com/streaming/reference/get/statuses/sample (accessed on January 16, 2016).
- 4.
https://dev.twitter.com/overview/api/tweets (accessed on January 18, 2016).
- 5.
http://www.meetup.com/meetup_api/ (accessed on January 18, 2016).
- 6.
https://github.com/cloudera/cdh-twitter-example (accessed on January 18, 2016).
- 7.
https://aws.amazon.com/de/elasticmapreduce/ (accessed on January 19, 2016).
References
Apache. (2015). Spark SQL, dataframes and datasets guide. Retrieved from http://spark.apache.org/docs/latest/sql-programming-guide.html.
Bello-Orgaz, G., Jung, J. J., & Camacho, D. (2016). Social big data: Recent achievements and new challenges. Information Fusion, 28, 45–59.
Besenbruch, C. (2015). Analyzing differences between IOS and Android device users. In Data science seminar 2016. University of St. Gallen.
Chang, R. M., Kauffman, R. J., & Kwon, Y. (2014). Understanding the paradigm shift to computational social science in the presence of big data. Decision Support Systems, 63, 67–80.
Chen, C. P., & Zhang, C.-Y. (2014). Data-intensive applications, challenges, techniques and technologies: A survey on big data. Information Sciences, 275, 314–347.
Davenport, T. H. (2014). Big data at work: Dispelling the myths, uncovering the opportunities. Harvard Business Review Press.
Davenport, T. H., & Patil, D. (2012). Data scientist. Harvard Business Review, 90, 70–76.
Dhar, V. (2013). Data science and prediction. Communications of the ACM, 56(12), 64–73.
Gandomi, A., & Haider, M. (2015). Beyond the hype: Big data concepts, methods, and analytics. International Journal of Information Management, 35(2), 137–144. https://doi.org/10.1016/j.ijinfomgt.2014.10.007.
Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google file system. Paper presented at the ACM SIGOPS operating systems review.
Herschel, G., Linden, A., & Kart, L. (2014). Magic quadrant for advanced analytics platforms, Gartner, Inc. Retrieved from.
Jung-Loddenkemper, A. (2015). Analysis of meetup RSVPs: How about fake RSVPs. In Data science seminar. University of St. Gallen.
Landset, S., Khoshgoftaar, T. M., Richter, A. N., & Hasanin, T. (2015). A survey of open source tools for machine learning with big data in the Hadoop ecosystem. Journal of Big Data, 2(1), 1–36.
Lazer, D., & Radford, J. (2017). Data ex Machina: Introduction to big data. Annual Review of Sociology, 43, 19–39. https://doi.org/10.1146/annurev-soc-060116-053457.
Mueller, M. (2015). Analysis of twitter sentiment data of a U.S. presidential candidate. In Data science seminar. University of St. Gallen.
Pääkkönen, P., & Pakkala, D. (2015). Reference architecture and classification of technologies, products and services for big data systems. Big Data Research.
Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The hadoop distributed file system. Paper presented at the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST).
Tambe, P. (2014). Big data investment, skills, and firm value. Management Science, 60(6), 1452–1469.
Thusoo, A., Sarma, J. S., Jain, N., Shao, Z., Chakka, P., Zhang, N., … Murthy, R. (2010). Hive-a petabyte scale data warehouse using hadoop. Paper presented at the 2010 IEEE 26th International Conference on Data Engineering (ICDE).
Weiguo, F. A. N., & Gordon, M. D. (2014). The power of social media analytics. Communications of the ACM, 57(6), 74–81. https://doi.org/10.1145/2602574.
White, T. (2015). Hadoop: The definitive guide (Vol. 4). O’Reilly Media, Inc.
Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M. … Stoica, I. (2012). Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Paper presented at the Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this chapter
Cite this chapter
Wulf, J. (2018). A Big Data Reference Architecture for Teaching Social Media Mining. In: Spector, J., et al. Frontiers of Cyberlearning. Lecture Notes in Educational Technology. Springer, Singapore. https://doi.org/10.1007/978-981-13-0650-1_5
Download citation
DOI: https://doi.org/10.1007/978-981-13-0650-1_5
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-0649-5
Online ISBN: 978-981-13-0650-1
eBook Packages: EducationEducation (R0)