Skip to main content
SpringerLink
Log in

Empowering Domain Experts to Preprocess Massive Distributed Datasets

  • Conference paper
  • First Online:
Business Information Systems (BIS 2020)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 389))

Included in the following conference series:

Abstract

In recent years, the amount of data is growing extensively. In companies, spreadsheets are one common approach to conduct data processing and statistical analysis. However, especially when working with massive amounts of data, spreadsheet applications have their limitations. To cope with this issue, we introduce a human-in-the-loop approach for scalable data preprocessing using sampling. In contrast to state-of-the-art approaches, we also consider conflict resolution and recommendations based on data not contained in the sample itself. We implemented a fully functional prototype and conducted a user study with 12 participants. We show that our approach delivers a significantly higher error correction than comparable approaches which only consider the sample dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    OpenRefine: https://openrefine.org.

  2. 2.

    KNIME: https://www.knime.com.

  3. 3.

    RapidMiner: https://rapidminer.com.

  4. 4.

    Trifacta Wrangler: https://www.trifacta.com.

References

  1. Abedjan, Z., et al.: Detecting data errors: where are we and what needs to be done? Proc. VLDB Endow. 9(12), 993–1004 (2016)

    Article  Google Scholar 

  2. Bendre, M., et al.: Anti-freeze for large and complex spreadsheets: asynchronous formula computation. In: Proceedings of the International Conference on Management of Data (SIGMOD) (2019)

    Google Scholar 

  3. Cypher, A. (ed.): Watch What I Do - Programming by Demonstration. MIT Press, Cambridge (1993)

    Google Scholar 

  4. Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml

  5. Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P.: From data mining to knowledge discovery in databases. AI Mag. 17(3), 37 (1996)

    Google Scholar 

  6. Gandel, S.: Damn Excel! How the ‘most important software application of all time’ is ruining the world (2013). http://fortune.com/2013/04/17/damn-excel-how-the-most-important-software-application-of-all-time-is-ruining-the-world/

  7. Gulwani, S., Marron, M.: NLyze: interactive programming by natural language for spreadsheet data analysis and manipulation. In: Proceedings of the International Conference on Management of Data (SIGMOD) (2014)

    Google Scholar 

  8. Gulwani, S., et al.: Spreadsheet data manipulation using examples. Commun. ACM 55(8), 97–105 (2012)

    Article  Google Scholar 

  9. International Business Machines Corporation: Transforming the Common Spreadsheet: A Smarter Approach to Budgeting, Planning and Forecasting, Technical report (2009)

    Google Scholar 

  10. Kandel, S., et al.: Wrangler: interactive visual specification of data transformation scripts. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI) (2011)

    Google Scholar 

  11. Kemper, H.G., et al.: Datenbereitstellung und -modellierung. In: Business Intelligence - Grundlagen und praktische Anwendungen: Eine Einführung in die IT-basierte Managementunterstützung (2010)

    Google Scholar 

  12. Lohr, S.L.: Sampling: Design and Analysis. Brooks/Cole (2009)

    Google Scholar 

  13. Mack, K., et al.: Characterizing scalability issues in spreadsheet software using online forums. In: Extended Abstracts of the Conference on Human Factors in Computing Systems (CHI EA) (2018)

    Google Scholar 

  14. Moore, S.: Gartner says more than 40 percent of data science tasks will be automated by 2020 (2017). https://www.gartner.com/en/newsroom/press-releases/2017-01-16-gartner-says-more-than-40-percent-of-data-science-tasks-will-be-automated-by-2020

  15. Reimann, P., Schwarz, H., Mitschang, B.: A pattern approach to conquer the data complexity in simulation workflow design. In: Meersman, R., et al. (eds.) OTM 2014. LNCS, vol. 8841, pp. 21–38. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-662-45563-0_2

    Chapter  Google Scholar 

  16. Rekatsinas, T., et al.: HoloClean - holistic data repairs with probabilistic inference. Proc. VLDB Endow. 10(11) (2017)

    Google Scholar 

  17. Shearer, C.: The CRISP-DM model: the new blueprint for data mining. J. Data Warehouse. 5(4) (2000)

    Google Scholar 

  18. Stodder, D.: Visual Analytics for Making Smarter Decisions Faster. Technical report, TDWI, Renton, WA, USA (2015)

    Google Scholar 

  19. Wache, H., et al.: Ontology-based integration of information - a survey of existing approaches. In: Proceedings of the Workshop on Ontologies and Information Sharing, International Joint Conference on Artificial Intelligence (IJCAI) (2001)

    Google Scholar 

Download references

Acknowlegements

This research was performed in the project ‘IMPORT’ as part of the Software Campus program, which is funded by the German Federal Ministry of Education and Research (BMBF) under Grant No.: 01IS17051.

Author information

Authors and Affiliations

  1. Institute for Parallel and Distributed Systems, University of Stuttgart, 70569, Stuttgart, Germany

    Michael Behringer, Pascal Hirmer, Manuel Fritz & Bernhard Mitschang

Authors
  1. Michael Behringer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pascal Hirmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Fritz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bernhard Mitschang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Behringer .

Editor information

Editors and Affiliations

  1. Poznań University of Economics and Business, Poznan, Poland

    Witold Abramowicz

  2. University of Colorado, Colorado Springs, CO, USA

    Gary Klein

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Behringer, M., Hirmer, P., Fritz, M., Mitschang, B. (2020). Empowering Domain Experts to Preprocess Massive Distributed Datasets. In: Abramowicz, W., Klein, G. (eds) Business Information Systems. BIS 2020. Lecture Notes in Business Information Processing, vol 389. Springer, Cham. https://doi.org/10.1007/978-3-030-53337-3_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-53337-3_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-53336-6

  • Online ISBN: 978-3-030-53337-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation