Managing Lifecycle of Big Data Applications

  • Ivan ErmilovEmail author
  • Axel-Cyrille Ngonga Ngomo
  • Aad Versteden
  • Hajira Jabeen
  • Gezim Sejdiu
  • Giorgos Argyriou
  • Luigi Selmi
  • Jürgen Jakobitsch
  • Jens Lehmann
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 786)


The growing digitization and networking process within our society has a large influence on all aspects of everyday life. Large amounts of data are being produced continuously, and when these are analyzed and interlinked they have the potential to create new knowledge and intelligent solutions for economy and society. To process this data, we developed the Big Data Integrator (BDI) Platform with various Big Data components available out-of-the-box. The integration of the components inside the BDI Platform requires components homogenization, which leads to the standardization of the development process. To support these activities we created the BDI Stack Lifecycle (SL), which consists of development, packaging, composition, enhancement, deployment and monitoring steps. In this paper, we show how we support the BDI SL with the enhancement applications developed in the BDE project. As an evaluation, we demonstrate the applicability of the BDI SL on three pilots in the domains of transport, social sciences and security.


Big Data Software methodologies Microservice architecture 



This work was supported by grant from the European Union’s Horizon 2020 research Europe flag and innovation program for the project Big Data Europe (GA no. 644564).


  1. 1.
    Auer, S., et al.: The BigDataEurope platform – supporting the variety dimension of big data. In: Cabot, J., Virgilio, R., Torlone, R. (eds.) ICWE 2017. LNCS, vol. 10360, pp. 41–59. Springer, Cham (2017). doi: 10.1007/978-3-319-60131-1_3. CrossRefGoogle Scholar
  2. 2.
    Ermilov, I.: Scalable spark/hdfs workbench using docker (2016), Retrieved 21 May 2017
  3. 3.
    Ermilov, I.: Developing spark applications with docker and BDE (2017), Retrieved 21 May 2017
  4. 4.
    Ermilov, I.: User interface integration in BDI platform (integrator UI application) (2017), Retrieved 21 May 2017
  5. 5.
    Grady, N.W.: KDD meets big data. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 1603–1608. IEEE (2016)Google Scholar
  6. 6.
    Harney, J., Lim, S.H., Sukumar, S., Stansberry, D., Xenopoulos, P.: On-demand data analytics in HPC environments at leadership computing facilities: Challenges and experiences. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 2087–2096. IEEE (2016)Google Scholar
  7. 7.
    Heit, J., Liu, J., Shah, M.: An architecture for the deployment of statistical models for the big data era. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 1377–1384. IEEE (2016)Google Scholar
  8. 8.
    Jabeen, H.: Bde vs. other hadoop distributions (2016), Retrieved 21 May 2017
  9. 9.
    Konstantopoulos, S., Charalambidis, A., Mouchakis, G., Troumpoukis, A., Jakobitch, J., Karkaletsis, V.: Semantic web technologies and big data infrastructures: SPARQL federated querying of heterogeneous big data stores. In: ISWC Demos and Posters Track (2016)Google Scholar
  10. 10.
    Kyzirakos, K., Karpathiotakis, M., Koubarakis, M.: Strabon: a semantic geospatial DBMS. In: Cudré-Mauroux, P., Heflin, J., Sirin, E., Tudorache, T., Euzenat, J., Hauswirth, M., Parreira, J.X., Hendler, J., Schreiber, G., Bernstein, A., Blomqvist, E. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 295–311. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-35176-1_19 CrossRefGoogle Scholar
  11. 11.
    Kyzirakos, K., Vlachopoulos, I., Savva, D., Manegold, S., Koubarakis, M.: Geotriples: a tool for publishing geospatial data as RDF graphs using R2RML mappings. In: Proceedings of the 2014 International Conference on Posters & Demonstrations Track, vol. 1272, pp. 393–396. CEUR-WS. org (2014)Google Scholar
  12. 12.
    Nikolaou, C., Dogani, K., Bereta, K., Garbis, G., Karpathiotakis, M., Kyzirakos, K., Koubarakis, M.: Sextant: Visualizing time-evolving linked geospatial data. Web Semant. Sci. Serv. Agents World Wide Web 35, 35–52 (2015)CrossRefGoogle Scholar
  13. 13.
    Rahman, F., Slepian, M., Mitra, A.: A novel big-data processing framework for healthcare applications: big-data-healthcare-in-a-box. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 3548–3555. IEEE (2016)Google Scholar
  14. 14.
    Rodriguez, P., Haghighatkhah, A., Lwakatare, L.E., Teppola, S., Suomalainen, T., Eskeli, J., Karvonen, T., Kuvaja, P., Verner, J.M., Oivo, M.: Continuous deployment of software intensive products and services: a systematic mapping study. J. Syst. Softw. 123, 263–291 (2017)CrossRefGoogle Scholar
  15. 15.
    Sebrechts, M., Borny, S., Vanhove, T., Van Seghbroeck, G., Wauters, T., Volckaert, B., De Turck, F.: Model-driven deployment and management of workflows on analytics frameworks. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 2819–2826. IEEE (2016)Google Scholar
  16. 16.
    Sezer, O.B., Dogdu, E., Ozbayoglu, M., Onal, A.: An extended iot framework with semantics, big data, and analytics. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 1849–1856. IEEE (2016)Google Scholar
  17. 17.
    Shearer, C.: The CRISP-DM model: the new blueprint for data mining. J. Data Warehouse. 5(4), 13–22 (2000)Google Scholar
  18. 18.
    Tsakalozos, K., Johns, C., Monroe, K., VanderGiessen, P., Mcleod, A., Rosales, A.: Open big data infrastructures to everyone. In: 2016 IEEE International Conference on Big Data (Big Data), pp. 2127–2129. IEEE (2016)Google Scholar
  19. 19.
    Versteden, A., Pauwels, E.: State-of-the-dart web applications using microservices and linked data. In: Maleshkova, M., Verborgh, R., Keppmann, F.L. (eds.) 4th Workshop on Services and Applications over Linked APIs and Data (SALAD), vol. 1629, pp. 25–36. CEUR Workshop Proceedings, Aachen (2016).

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Ivan Ermilov
    • 1
    Email author
  • Axel-Cyrille Ngonga Ngomo
    • 2
  • Aad Versteden
    • 3
  • Hajira Jabeen
    • 4
  • Gezim Sejdiu
    • 4
  • Giorgos Argyriou
    • 5
  • Luigi Selmi
    • 6
  • Jürgen Jakobitsch
    • 7
  • Jens Lehmann
    • 4
  1. 1.Instituts für Angewandte InformatikLeipzigGermany
  2. 2.Paderborn UniversityPaderbornGermany
  3. 3.TenforceLeuvenBelgium
  4. 4.University of BonnBonnGermany
  5. 5.University of AthensAthensGreece
  6. 6.Fraunhofer IAIS, Schloss BirlinghovenSankt AugustinGermany
  7. 7.Semantic Web Company GmbHViennaAustria

Personalised recommendations