The VLDB Journal

, Volume 27, Issue 4, pp 547–571 | Cite as

Multi-schema-version data management: data independence in the twenty-first century

  • Kai HerrmannEmail author
  • Hannes Voigt
  • Torben Bach Pedersen
  • Wolfgang Lehner
Regular Paper


Agile software development allows us to continuously evolve and run a software system. However, this is not possible in databases, as established methods are very expensive, error-prone, and far from agile. We present InVerDa, a multi-schema-version database management system (MSVDB) for agile database development. MSVDBs realize co-existing schema versions within one database, where each schema version behaves like a regular single-schema database and write operations are propagated between schema versions. Developers use a relationally complete and bidirectional database evolution language (BiDEL) to easily evolve existing schema versions to new ones. BiDEL scripts are more robust, orders of magnitude shorter, and cause only a small performance overhead compared to handwritten SQL scripts. We formally guarantee data independence: no matter how the data of the co-existing schema versions is physically materialized, each schema version is guaranteed to behave like a regular database. Since, the chosen physical materialization significantly determines the overall performance, we equip database administrators with an advisor that proposes an optimized materialization for the current workload, which can improve the performance by orders of magnitude compared to naïve solutions. To our best knowledge, we are the first to facilitate agile evolution of production databases with full support of co-existing schema versions and formally guaranteed data independence.


Database Evolution Data independence 



This work is partly funded by the German Research Foundation (DFG) within the RoSI RTG.


  1. 1.
    Agrawal, S., Chaudhuri, S., Narasayya, V.R.: Automated selection of materialized views and indexes in SQL databases. In: Proceedings of the 26th International Conference on Very Large Data Bases, 10–14 September 2000, pp. 496–505. Morgan Kaufmann Publishers Inc, San Francisco (2000)Google Scholar
  2. 2.
    Agrawal, S., Chaudhuri, S., Narasayya, V.: Materialized view and index selection tool for Microsoft SQL server 2000. ACM SIGMOD Rec. 30(2), 608 (2001)CrossRefGoogle Scholar
  3. 3.
    Arora, M., Gosain, A.: Schema evolution for data warehouse: a survey. IJCA 22(5), 6–14 (2011)CrossRefGoogle Scholar
  4. 4.
    Bentley, P., Corne, D.: Creative Evolutionary Systems. Morgan Kaufmann, San Francisco (2002)Google Scholar
  5. 5.
    Bernstein, P.A., Melnik, S.: Model management 2.0: manipulating richer mappings. In: International Conference on Management of Data (SIGMOD), Beijing, 12–14 June 2007, pp. 1–12 (2007).
  6. 6.
    Brodie, M.L., Liu, J.T.: The power and limits of relational technology in the age of information ecosystems. In: On the Move to Meaningful Internet Systems (OTM), Lecture Notes in Computer Science. Springer, Berlin (2010)Google Scholar
  7. 7.
    Curino, C., Moon, H.J., Deutsch, A., Zaniolo, C.: Automating the database schema evolution process. VLDB J. 22(1), 73–98 (2013)CrossRefGoogle Scholar
  8. 8.
    Curino, C., Moon, H.J., Zaniolo, C.: Graceful database schema evolution: the PRISM workbench. PVLDB 1(1), 761–772 (2008)Google Scholar
  9. 9.
    Curino, C., Moon, H.J., Tanca, L., Zaniolo, C.: Schema evolution in wikipedia-Toward a web information system benchmark. In: Proceedings of the Tenth International Conference on Enterprise Information Systems (ICEIS), Volume DISI, Barcelona, Spain, 12–16 June 2008, pp. 323–332 (2008)Google Scholar
  10. 10.
    Domínguez, E., Lloret, J., Rubio, Á.L., Zapata, M.A.: MeDEA: a database evolution architecture with traceability. DKE 65(3), 419–441 (2008)CrossRefGoogle Scholar
  11. 11.
    Floudas, C.A., Pardalos, P.M., Adjiman, C., Esposito, W.R., Gümüs, Z.H., Harding, S.T., Klepeis, J.L., Meyer, C.A., Schweiger, C.A.: Handbook of Test Problems in Local and Global Optimization. Nonconvex Optimization and Its Applications. Springer, New York (2013)zbMATHGoogle Scholar
  12. 12.
    Gupta, A., Mumick, I.S.: Materialized Views: Techniques, Implementations, and Applications. MIT Press, Cambridge (1999)Google Scholar
  13. 13.
    Herrmann, K.: Multi-Schema-Version Data Management. Ph.D. thesis,, Technische Universität Dresden, 2017. Accessed 6 Jun 2018
  14. 14.
    Herrmann, K., Voigt, Hv , Behrend, A., Lehner, W.: CoDEL—a relationally complete language for database evolution. In: ADBIS (2015)Google Scholar
  15. 15.
    Herrmann, K., Voigt, H., Behrend, A., Rausch, J., Lehner,W.: Living in parallel realities: Co-existing schema versions with a bidirectional database evolution language. In: Proceedings of the 2017 ACM International Conference on Management of Data (SIGMOD), Chicago, 14–19 May 2017, pp. 1101–1116 (2017).
  16. 16.
    Hofmann, M., Pierce, B.C., Wagner, D.: Symmetric lenses. In: Proceedings of the 38th Symposium on Principles of Programming Languages, POPL 2011, Austin, 26–28 January 2011, pp. 371–384 (2011).
  17. 17.
    Howard, P.: Data Migration Report, Bloor. (2011). Accessed 6 June 2018
  18. 18.
    McKinna, J.: Complements witness consistency. In: Bx (2016)Google Scholar
  19. 19.
    Mistry, H., Roy, P., Sudarshan, S., Ramamritham, K.: Materialized view selection and maintenance using multi-query optimization. In: Proceedings of the 2001 international conference on Management of data (SIGMOD), Santa Barbara, CA, USA, 21–24 May 2001, pp. 307–318 (2001).
  20. 20.
    Moon, H.J., Curino, C., Ham, M., Zaniolo, C.: PRIMA—archiving and querying historical data with evolving schemas. In: SIGMOD (2009)Google Scholar
  21. 21.
    Rahm, E., Bernstein, P.A.: An online bibliography on schema evolution. ACM SIGMOD Rec. 35(4), 30–31 (2006)CrossRefGoogle Scholar
  22. 22.
    Roddick, J.F.: SQL/SE—a query language extension for databases supporting schema evolution. ACM SIGMOD Rec. 21(2), 10–16 (1992)CrossRefGoogle Scholar
  23. 23.
    Roddick, J.F.: A survey of schema versioning issues for database systems. Inf. Softw. Technol. 37(7), 383–393 (1995)CrossRefGoogle Scholar
  24. 24.
    Skoulis, I., Vassiliadis, P., Zarras, A.V.: Open-Source databases: within, outside, or beyond Lehman’s laws of software evolution?. In: Advanced Information Systems Engineering - 26th International Conference, CAiSE 2014, Thessaloniki, Greece, 16–20 June 2014, pp. 379–393 (2014).
  25. 25.
    Terwilliger, J.F., Cleve, A., Curino, C.: How clean is your sandbox? - Towards a unified theoretical framework for incremental bidirectional transformations In: Theory and Practice of Model Transformations -5th International Conference, ICMT 2012, Prague, Czech Republic, 28–29 May 2012, pp. 1–23 (2012).
  26. 26.
    Wall, B., Angryk, R.: Minimal data sets vs. synchronized data copies in a schema and data versioning system. In: Proceedings of the 4th workshop on Workshop for Ph.D. students in information and knowledge management (IPKM), Glasgow, United Kingdom, 24–28 October 2011, pp. 67–74 (2011).
  27. 27.
    Wentao, W., Chi, Y., Hacígümüş, H., Naughton, J.F.: Towards predicting query execution time for concurrent and dynamic database workloads. PVLDB 6(10), 925–936 (2013)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Technische Universität DresdenDresdenGermany
  2. 2.Aalborg UniversitetAalborgDenmark

Personalised recommendations