Abstract
Data silo can grow to be a large-scale data for years, overlapping and has an indefinite quality. It allows an organization to develop their own analytical capabilities. Data lake has the ability to solve this problem efficiently with the data analysis by using statistical and predictive modeling techniques which can be applied to enhance and support an organization’s business strategy. This study provides an overview of the process of decision-making, operational efficiency, and creating the solution for an organization. Machine Learning can distribute the architecture of data model and integrate the data silo with other organizations data to optimize the operational business processes within an organization in order to improve data quality and efficiency. Testing is done by utilizing the data from the Malaysia’s and Singapore’s Government Open Data on the Air Pollutant Index to determine the condition of air pollution levels for the health and safety of the population.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Das, T.K., Mishra, M.R.: A study on challenges and opportunities in master data management. Int. J. Database Manag. Syst. 3, 129–139 (2011)
Bilal, M., Oyedele, L.O., Akinade, O.O., Ajayi, S.O., Alaka, H.A., Owolabi, H.A., Bello, S.A.: Big data architecture for construction waste analytics (CWA): a conceptual framework. J. Build. Eng. 6, 144–156 (2016)
Bobek, S., Nalepa, G.J.: Uncertain context data management in dynamic mobile environments. Future Gener. Comput. Syst. 66, 110–124 (2016)
Philip Chen, C., Zhang, C.Y.: Data-intensive applications, challenges, techniques and technologies: a survey on Big Data. Inf. Sci. 275, 314–347 (2014)
Khanna, A., Bhatt, H., Hardik, M., Sakhuja, R.: Data lake for socio-knowledge platforms. Int. J. Eng. Appl. Sci. Technol. 1, 110–113 (2016)
Milanov, G., Njeguš, A.: Using scrum in master data management development projects. J. Emerg. Trends Comput. Inf. Sci. 3, 1446–1452 (2012)
Indrajani, I.: Master data management model in company: challenges and opportunity. ComTech. 6, 514–524 (2015)
Konverse: Big Data Lake (2015)
Margaret, R.: Data Silo Definition (2015). http://searchcloudapplications.techtarget.com/definition/data-silo
Fenwick, T., Brunsdon, D., Seville, E.: Reducing the Impact of Organisational Silos on Resilience (2009). http://www.resorgs.org.nz/images/stories/pdfs/silos.pdf
Yaqoob, I., Hashem, I.A.T., Gani, A., Mokhtar, S., Ahmed, E., Anuar, N.B., Vasilakos, A.V.: Big data: from beginning to future. Int. J. Inf. Manage. 36(6), 1231–1247 (2016)
Gandomi, A., Haider, M.: Beyond the hype: big data concepts, methods, and analytics. Int. J. Inf. Manage. 35(2), 137–144 (2015)
Eygm, K.W., Peng, T.: big data changing the way businesses. Int. J. Simul. Syst. Sci. Technol. 16, 28 (2014)
Lomotey, R.K., Deters, R.: Terms extraction from unstructured data silos. In: Proceedings of 2013 8th International Conference on System of Systems Engineering: SoSE in Cloud Computing and Emerging Information Technology Applications, SoSE 2013, pp. 19–24 (2013)
Krensky, P.: Winter is not coming: eliminating data silos and ending information hoarding. 10 (2015)
Lowans, B., Perkins, E.: Market Guide for Data-Centric Audit and Protection (2014)
Shahrokni, A., Soderberg, J.: Beyond information silos challenges in integrating industrial model-based data. CEUR Workshop Proc. 1406, 63–72 (2015)
Laskowski, N.: Data lake governance: a big data do or die (2016). http://searchcio.techtarget.com/feature/Data-lake-governance-A-big-data-do-or-die
Kottursamy, K., Raja, G., Padmanabhan, J., Srinivasan, V.: An improved database synchronization mechanism for mobile data using software-defined networking control. Comput. Electr. Eng. 1–11 (2016)
Hegde, S.D., Ravinarayana, B.: Survey paper on data lake. Int. J. Sci. Res. 5(7) (2016)
Miloslavskaya, N., Tolstoy, A.: Big data, fast data and data lake concepts. Procedia Comput. Sci. 88, 300–305 (2016)
Terrizzano, I., Schwarz, P., Roth, M., Colino, J.E.: Data wrangling: the challenging journey from the wild to the lake. 7th Biennial Conference on Innovative Data Systems Research, CIDR 2015 (2015)
Means, A.: The Role of Data Lakes in Healthcare. North America, Perficient (2014)
Tan, P.: Introduction to Data Mining. Pearson Education, Boston (2006)
Budiharto, W.: Machine learning & computational intelligence. Penerbit Andi (2016)
Zhou, L., Pan, S., Wang, J., Vasilakos, A.V.: Machine learning on big data: opportunities and challenges. Neurocomputing (2017)
Nugroho, A.S.: Pengantar Support Vector Machine (2007)
Sun, S., Luo, C., Chen, J.: A review of natural language processing techniques for opinion mining system. Elsevier B.V. (2016)
Tsangaratos, P., Ilia, I.: Comparison of a logistic regression and Naïve Bayes classifier in landslide susceptibility assessments: the influence of models complexity and training dataset size. CATENA 145, 164–179 (2016)
Mayilvaganan, M., Kalpanadevi, D.: Comparison of classification techniques for predicting the performance of students’ academic environment. In: International Conference on IEEE Communication and Network Technologies (ICCN 2014), pp. 113–118 (2014)
Gray, G., McGuinness, C., Owende, P.: An application of classification models to predict learner progression in tertiary education. In: IEEE International Advance Computing Conference (IACC), pp. 549–554 (2014)
El-Alfy, E.-S.M., Alshammari, M.A.: Towards scalable rough set based attribute subset selection for intrusion detection using parallel genetic algorithm in MapReduce. Simul. Model. Pract. Theory., 1–12 (2016)
Fang, B.W., Hu, B.Q.: Probabilistic graded rough set and double relative quantitative decision-theoretic rough set. Int. J. Approx. Reason. 74, 1–12 (2016)
Jiang, L., Li, C., Wang, S., Zhang, L.: Deep feature weighting for naive Bayes and its application to text classification. Eng. Appl. Artif. Intell. 52, 26–39 (2016)
Pawlak, Z.: Rough sets. Int. J. Comput. Inf. Sci. 11, 341–356 (1982)
Rissino, S., Lambert-Torres, G.: Rough Set Theory–Fundamental Concepts, Principals, Data Extraction, and Applications. In: Ponce, J., Karahoca, A. (eds.) Data Mining and Knowledge Discovery in Real Life Applications. InTech Publisher (2009)
Velez, F., Agrawal, S.: Data lakes: Discovering, Governing and Transforming Raw Data into Strategic Data Assets. Persistent System (2016)
Murray, C.J., Vos, T., Lozano, R., Naghavi, M., Flaxman, A.D., Michaud, C., Bridgett, L.: Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990–2010: a systematic analysis for the global burden of disease study 2010. Lancet 380(9859), 2197–2223 (2013)
Bank, World: Indonesia’s Fire and Haze Crisis (2015). http://www.worldbank.org/en/news/feature/2015/12/01/indonesias-fire-and-haze-crisis
Kim, H., Park, Y., Park, K., Yoo, B.: Association between pollen risk indexes, air pollutants, and allergic diseases in korea. Osong Public Heal. Res. Perspect. 7, 172–179 (2016)
Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of International Joint Conference on AI, pp. 1137–1145 (1995)
Verbert, K., Govaerts, S., Duval, E., Santos, J.L., Van Assche, F., Parra, G., et al.: Learning dashboards: an overview and future research opportunities. Personal and Ubiquitous Computing 18(6), 1499e1514 (2014)
Acknowledgements
This work is supported by Ministry of Higher Education Malaysia (MOHE), Ministry of Science, Technology and Innovation Malaysia (MOSTI), and Universiti Teknologi Malaysia (UTM). This paper is financially supported by E-Science Fund, R.J130000.7928.4S117, PRGS Grant, R.J130000.7828.4L680, GUP Tier 1 UTM, Q.J130000.2528.13H48, FRGS Grant, R.J130000.7828.4F634 and IDG Grant, R.J130000.7728.4J170. The Authors would like to express their deepest gratitude to the Research Management Centre (RMC), UTM for the support in research and development and Soft Computing Research Group (SCRG) for the inspiration and make this research success. Authors would also like to thank the anonymous reviewers who have contributed enormously to this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Wibowo, M., Sulaiman, S., Shamsuddin, S.M. (2017). Machine Learning in Data Lake for Combining Data Silos. In: Tan, Y., Takagi, H., Shi, Y. (eds) Data Mining and Big Data. DMBD 2017. Lecture Notes in Computer Science(), vol 10387. Springer, Cham. https://doi.org/10.1007/978-3-319-61845-6_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-61845-6_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-61844-9
Online ISBN: 978-3-319-61845-6
eBook Packages: Computer ScienceComputer Science (R0)