Abstract
Big data is a term that describes the large volume of data, both structured and unstructured. Due to its huge quantity, big data are stored by partitioning and distributing into smaller chunks of data in multiple machines for quick and efficient analysis, because it is not possible for a single machine to hold all of the big data by itself. However, these datasets are generally incomplete because it contains many instances of missing values. Missing values are a serious impediment to data analysis, and Multiple Imputation is a preferred method for handling missing values. All existing multiple imputation implementations in statistical software packages are all based on the in-memory processing of data and are unsuitable if the data is distributed. So there is a need for handling missing values using multiple imputation if the data is distributed. The goal of this work is to implement a multiple imputation algorithm for missing values using fuzzy clustering on a distributed computing system built with Apache Spark. The results show that the multiple imputation algorithm outperforms traditional imputation techniques for missing values in a distributed computing system in terms of imputation accuracy.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kang, H.: The prevention and handling of the missing data. Korean J. Anesthesiol. 64(5), 402–406 (2013)
Zaharia, M., et al.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: USENIX Symposium on Networked Systems Design and Implementation (2012)
Azur, M.J., Stuart, E.A., Frangakis, C., Leaf, P.J.: Multiple imputation by chained equations: what is it and how does it work? Int. J. Methods Psychiatr. Res. 20(1), 40–49 (2011)
Houari, R., Bounceur, A., Tari, A., Kechadi, M.T.: Handling missing data problems with sampling methods. In: International Conference on Advanced Distributed Systems and Applications (2014)
Ye, H.: Missing data analysis using multiple imputation: getting to the heart of the matter. Circ. Cardiovasc. Qual. Outcomes 3(1), 98–105 (2010)
Harel, O., Zhou, X.H.: Multiple imputation - review of theory, implementation and software. Stat. Med. 26(16), 3057–3077 (2007)
Rubin, D.B.: Basic ideas of multiple imputation for nonresponse. Stat. Can. 12(1), 37–47 (1986)
Nikfalazar, S., Khorshidi, H.A., Bedingfield, S., Yeh, C.-H.: A new iterative fuzzy clustering algorithm for multiple imputation of missing data. In: IEEE International Conference on Fuzzy Systems, Fuzzy Systems, FUZZ-IEEE, Naples (2017)
Bharill, N., Tiwari, A., Malviya, A.: Fuzzy based clustering algorithms to handle big data with implementation on Apache Spark. In: IEEE Second International Conference on Big Data Computing Service and Applications, Exeter College, Oxford, UK, pp. 95–104 (2016)
Armina, R., Zain, A.M., Ali, N.A., Sallehuddin, R.: A review on missing value estimation using imputation algorithm. J. Phys. Conf. Ser. (JPCS) 892(1), 4 (2017)
Saravanan, P., Sailakshmi, P.: Missing value imputation using fuzzy possibilistic C means optimized with support vector regression and genetic algorithm. J. Theoret. Appl. Inf. Technol. 72(1), 34–39 (2015)
Software for Multiple Imputation. http://multiple-imputation.com/software.html
Apache Spark. https://spark.apache.org
Open Government Data Platform (OGD) India. https://data.gov.in
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Kaliamoorthy, S., Bhanu, S.M.S. (2018). Multiple Imputation Inference for Missing Values in Distributed Datasets Using Apache Spark. In: Singh, M., Gupta, P., Tyagi, V., Flusser, J., Ören, T. (eds) Advances in Computing and Data Sciences. ICACDS 2018. Communications in Computer and Information Science, vol 906. Springer, Singapore. https://doi.org/10.1007/978-981-13-1813-9_3
Download citation
DOI: https://doi.org/10.1007/978-981-13-1813-9_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-13-1812-2
Online ISBN: 978-981-13-1813-9
eBook Packages: Computer ScienceComputer Science (R0)