Abstract
Data is ubiquitous. Many modern software and data analytics applications rely on robust and quality datasets. Data engineering becomes a common pipeline in systems running in start-up and enterprise businesses. Data engineering projects in the past were perceived as a set of programming scripts which were typically in a “build-then-scrap” cycle. As the data analytics applications became parts of the main trends, such projects require a serious planning and development to minimize the overhead of integration and maintenance due to scaling up. In this article, we discuss how to use type systems and formal methods to reduce these overheads.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
In Scala, String type is not an alias of List[Char].
References
Burmako, E. (2017). Unification of compile-time and runtime metaprogramming in Scala. Infoscience, p. 240.
Cheney, J., & Hinze, R. (2003). First-class phantom types. Technical report, Cornell University.
Dijkstra, E. W. (2017). The manuscripts of Edsger W. Dijkstra. Retrieved December 1, 2019, from https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/EWD303.html
Finne, S., Leijen, D., Meijer, E., & Peyton Jones, S. (1999). Calling hell from heaven and heaven from hell. In Proceedings of the Fourth ACM SIGPLAN International Conference on Functional Programming, ICFP ’99 (pp. 114–125). New York: ACM.
Gkatziouras, E. (2017). Java code geeks: Wordcount with Storm and Scala. https://www.javacodegeeks.com/2017/02/wordcount-storm-scala.html
Hadoop (2019). Apache hadoop. Retrieved December 1, 2019, from http://hadoop.apache.org/
Iry, J. (2010). Phantom types in Haskell and Scala. Retrieved December 1, 2019, form http://james-iry.blogspot.com/2010/10/phantom-types-in-haskell-and-scala.html
Lu, K. Z. M. (2019). Code examples in the article - lean launch data engineering projects with super type power. Retrieved November 1, 2019, from https://github.com/luzhuomi/lean_launch_data_engineering_ex
MyPY (2019). Optional static typing for python. Retrieved December 1, 2019, form http://mypy-lang.org/
Peyton Jones, S., Jones, M., & Meijer, E. (1997). Type classes: An exploration of the design space. In Haskell Workshop.
Spark (2019). Apache spark. Retrieved December 1, 2019, form https://spark.apache.org/
Storm (2019). Apache storm. Retrieved December 1, 2019, form https://storm.apache.org/
Sulzmann, M., & Lu, K. Z. M. (2008). Implementation and application of functional languages. In O. Chitil, Z. Horváth, & V. Zsók (Eds.), Implementation and application of functional languages, chapter XHaskell — Adding Regular Expression Types to Haskell (pp. 75–92). Berlin: Springer.
Wallace, M., & Runciman, C. (1999). Haskell and XML: Generic combinators or type-based translation? In International Conference on Functional Programming ICFP ’99 (pp. 148–159). New York: ACM Press.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Lu, K.Z.M. (2020). Lean Launch Data Engineering Projects with Super Type Power. In: Glauner, P., Plugmann, P. (eds) Innovative Technologies for Market Leadership. Future of Business and Finance. Springer, Cham. https://doi.org/10.1007/978-3-030-41309-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-41309-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-41308-8
Online ISBN: 978-3-030-41309-5
eBook Packages: Business and ManagementBusiness and Management (R0)