Data Warehousing and Exploratory Analysis for Market Monitoring
With the growing trend of digitalization, many companies plan to use machine learning to improve their business processes or to provide new data-driven services. These companies often collect data from different locations with sometimes conflicting context. However, before machine learning can be applied, heterogeneous datasets often need to be integrated, harmonized, and cleaned. In other words, a data warehouse is often the foundation for subsequent analytics tasks.
In this chapter, we first provide an overview on best practices of building a data warehouse. In particular, we describe the advantages and disadvantage of the major types of data warehouse architectures based on Inmon and Kimball. Afterward, we describe a use case on building an e-commerce application where the users of this platform are provided with information about healthy products as well as products with sustainable production. Unlike traditional e-commerce applications, where users need to log into the system and thus leave personalized traces when they search for specific products or even buy them afterward, our application allows full anonymity of the users in case they do not want to log into the system. However, analyzing anonymous user interactions is a much harder problem than analyzing named users. The idea is to apply modern data warehousing, big data technologies, as well as machine learning algorithms to discover patterns in the user behavior and to make recommendations for designing new products.
Unable to display preview. Download preview PDF.
The work was funded by the Swiss Commission for Technology and Innovation (CTI) under grant 16053.2.
- Apache Mahout. Retrieved August 24, 2018., from http://mahout.apache.org/
- Casper, J., & Olukotun, K. (2014). Hardware acceleration of database operations. In Proceedings of the 2014 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (pp. 151–160). ACM.Google Scholar
- Clifton, B. (2012). Advanced web metrics with Google analytics. Hoboken, NJ: Wiley.Google Scholar
- Ehrenmann, M., Pieringer, R., & Stockinger, K. (2012). Is there a cure-all for business analytics case studies of exemplary businesses in banking, telecommunications, and retail. Business Intelligence Journal, 17(3). TDWI.Google Scholar
- Ester, M., & Kriegel, H.P., & Sander, J. & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. In International Conference on Knowledge Discovery and Data Mining, 1996.Google Scholar
- Feuerstein, S., & Pribyl, B. (2005). Oracle Pl/SQL programming. O’Reilly Media, Newton, MA.Google Scholar
- Hultgren, H. (2012). Modeling the agile data warehouse with data vault. Denver, CO: New Hamilton.Google Scholar
- Inmon, B. (1992). Building the data warehouse. Hoboken, NJ: Wiley.Google Scholar
- JasperSoft. Retrieved July 21, 2017, from https://www.jaspersoft.com/
- Kimball, R. (2002). The data warehouse toolkit. Hoboken, NJ: Wiley.Google Scholar
- MacQueen, J. B. (1967). Some methods for classification and analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Probability, University of California Press.Google Scholar
- McCallum, A., Nigam, K., & Ungar, L. H. (2000). Efficient clustering of high dimensional data sets with application to reference matching. In SIGKDD International Conference on Knowledge Discovery and Data Mining.Google Scholar
- Pentaho. Retrieved July 21, 2017, from http://www.pentaho.com/
- Postgres. Retrieved July 21, 2017, from https://www.postgresql.org/
- Talend. Retrieved July 21, 2017, from https://www.talend.com/