Benchmarking bidirectional transformations: theory, implementation, application, and assessment


Bidirectional transformations (bx) are relevant for a wide range of application domains. While bx problems may be solved with unidirectional languages and tools, maintaining separate implementations of forward and backward synchronizers with mutually consistent behavior can be difficult, laborious, and error-prone. To address the challenges involved in handling bx problems, dedicated languages and tools for bx have been developed. Due to their heterogeneity, however, the numerous and diverse approaches to bx are difficult to compare, with the consequence that fundamental differences and similarities are not yet well understood. This motivates the need for suitable benchmarks that facilitate the comparison of bx approaches. This paper provides a comprehensive treatment of benchmarking bx, covering theory, implementation, application, and assessment. At the level of theory, we introduce a conceptual framework that defines and classifies architectures of bx tools. At the level of implementation, we describe Benchmarx, an infrastructure for benchmarking bx tools which is based on the conceptual framework. At the level of application, we report on a wide variety of solutions to the well-known Families-to-Persons benchmark, which were developed and compared with the help of Benchmarx. At the level of assessment, we reflect on the usefulness of the Benchmarx approach to benchmarking bx, based on the experiences gained from the Families-to-Persons benchmark.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29


  1. 1.

    In this paper, we consider only the case of two artifacts.

  2. 2.

  3. 3.

  4. 4.

    Merriam-Webster 2013.

  5. 5.

    The results of some test cases depend on the order of elementary change operations.

  6. 6.

    In practice, this could either represent runtime user interaction or compile-time design preferences.

  7. 7.

    EMF supports this via a notification framework.

  8. 8.

    Recall from Sect. 1 that the forward direction is from the families model to the persons model, while backward is from the persons model to the families model.

  9. 9.

  10. 10.

  11. 11.

  12. 12.

    The actual solution is a bit more complex as SyncL is further decomposed into two arrows.

  13. 13.

  14. 14.

  15. 15.

  16. 16.

  17. 17.

  18. 18.


  1. 1.

    Anjorin, A.: An introduction to triple graph grammars as an implementation of the delta-lens framework. In: Gibbons, J., Stevens, P. (eds.) Bidirectional Transformations—International Summer School, Oxford, UK, 25–29 July 2016, Tutorial Lectures, volume of 9715 Lecture Notes in Computer Science, pp. 29–72. Springer (2016)

  2. 2.

    Anjorin, A., Buchmann, T., Westfechtel, B.: The Families to Persons case. In: Garcia-Dominguez et al. [23], pp. 27–34

  3. 3.

    Anjorin, A., Cunha, A., Giese, H., Hermann, F., Rensink, A., Schürr, A.: Benchmarx. In: Candan et al. [8], pp. 82–86

  4. 4.

    Anjorin, A., Diskin, Z., Jouault, F., Ko, H.-S., Leblebici, E., Westfechtel, B.: Benchmarx reloaded: a practical benchmark framework for bidirectional transformations. In: Eramo, R., Johnson, M. (eds.) Proceedings of the 6th International Workshop on Bidirectional Transformations Co-located with The European Joint Conferences on Theory and Practice of Software, BX@ETAPS 2017, Uppsala, Sweden, April 29, 2017, volume of 1827 CEUR Workshop Proceedings, pp. 15–30. (2017)

  5. 5.

    Bergmann, G., Horváth, Á., Ráth, I., Daniel Varró. A benchmark evaluation of incremental pattern matching in graph transformation. In: Ehrig, H., Heckel, R., Rozenberg, G., Taentzer, G. (eds.) Graph Transformations, 4th International Conference, ICGT 2008, Leicester, UK, 7–13 Sept 2008. Proceedings, volume of 5214 Lecture Notes in Computer Science, pp. 396–410. Springer (2008)

  6. 6.

    Buchmann, T.: BXtend—a framework for (bidirectional) model transformations. In: Hamoudi, S., Ferreira Pires, L., Selic, B. (eds.) Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development: MODELSWARD (MODELSWARD 2018), vol. 1, pp. 336–345, Funchal, Madeira, Jan 2018. SciTePress

  7. 7.

    Buchmann, T., Greiner, S.: Handcrafting a triple graph transformation system to realize round-trip engineering between UML class models and java source code. In: Maciaszek, L.A., Cardoso, J.S., Ludwig, A., van Sinderen, M., Cabello, E. (eds.) Proceedings of the 11th International Joint Conference on Software Technologies (ICSOFT 2016)—Volume 2: ICSOFT-PT, Lisbon, Portugal, 24–26 July 2016, pp. 27–38. SciTePress (2016)

  8. 8.

    Candan, K.S., Amer-Yahia, S., Schweikardt, N., Christophides, V., Leroy, V. (eds.): Proceedings of the Workshops of the EDBT/ICDT 2014 Joint Conference (EDBT/ICDT 2014), Athens, Greece, March 28, 2014, volume 1133 of CEUR Workshop Proceedings. (2014)

  9. 9.

    Cheney, J., Gibbons, J., McKinna, J., Stevens, P.: Towards a principle of least surprise for bidirectional transformations. In: Cunha, A., Kindler, E. (eds.) Proceedings of the 4th International Workshop on Bidirectional Transformations Co-located with Software Technologies: Applications and Foundations, STAF 2015, L’Aquila, Italy, 24 July 2015, volume 1396 of CEUR Workshop Proceedings, pp. 66–80. (2015)

  10. 10.

    Cheney, J., McKinna, J., Stevens, P., Gibbons, J.: Towards a repository of bx examples. In: Candan et al. [8], pp. 87–91

  11. 11.

    Cicchetti, A., Di Ruscio, D., Eramo, R., Pierantonio, A.: JTL: a bidirectional and change propagating transformation language. In: Malloy, B., Staab, S., van den Brand, M. (eds.) Proceedings of the 3rd International Conference on Software Language Engineering (SLE 2010), volume 6563 of Lecture Notes of Computer Science, pp. 183–202, Eindhoven, The Netherlands, Oct 2010. Springer

  12. 12.

    Czarnecki, K., Nathan Foster, J., Hu, Z., Lämmel, R., Schürr, A., Terwilliger, J.F.: Bidirectional transformations: a cross-discipline perspective. In: Paige, R.F. (ed.) Proceedings of the Second International Conference on Theory and Practice of Model Transformations (ICMT 2009), volume 5563 of Lecture Notes of Computer Science, pp. 260–283, Zurich, Switzerland, June 2009. Springer

  13. 13.

    da Silva, A.R.: Model-driven engineering: a survey supported by the unified conceptual model. Comput. Lang. Syst. Struct. 43, 139–155 (2015)

    Google Scholar 

  14. 14.

    Diskin, Z.: Model synchronization: mappings, tiles, and categories. In: Fernandes, J.M., Lämmel, R., Visser, J., Saraiva, J. (eds.) Generative and Transformational Techniques in Software Engineering III—International Summer School, GTTSE 2009, Braga, Portugal, 6–11 July 2009. Revised Papers, volume 6491 of Lecture Notes in Computer Science, pp. 92–165. Springer (2009)

  15. 15.

    Diskin, Z., Gholizadeh, H., Wider, A., Czarnecki, K.: A three-dimensional taxonomy for bidirectional model synchronization. J. Syst. Softw. 111, 298–322 (2016)

    Article  Google Scholar 

  16. 16.

    Diskin, Z., Xiong, Y., Czarnecki, K.: From state- to delta-based bidirectional model transformations: the asymmetric case. J. Object Technol. 10(6), 1–25 (2011)

    Google Scholar 

  17. 17.

    Ehrig, K., Guerra, E., De Lara, J., Lengyel, L., Levendovszky, T., Prange, U., Taentzer, G., Varró, D., Varro-Gyapay, S.: Model transformation by graph transformation: a comparative study. In: Proceedings of the International Workshop on Model Transformations in Practice (MTiP 2005), Satellite Event of MoDELS 2005, volume 3844 of Lecture Notes of Computer Science, pp. 71–80, Montego Bay, Jamaica, 2005. Springer

  18. 18.

    Eramo, R., Bucaioni, A.: Understanding bidirectional transformations with TGGs and JTL. ECEASST 57, 1 (2013)

    Google Scholar 

  19. 19.

    Eramo, R., Marinelli, R., Pierantonio, A.: Towards a taxonomy for bidirectional transformation. In: Di Ruscio, D., Zaytsev, V. (eds.) Post-proceedings of the Seventh Seminar on Advanced Techniques and Tools for Software Evolution, SATToSE 2014, L’Aquila, Italy, 9–11 July 2014, volume 1354 of CEUR Workshop Proceedings, pp. 122–131. (2014)

  20. 20.

    Eramo, R., Pierantonio, A., Tucci, M.: Enhancing the JTL tool for bidirectional transformations. In: Conference Companion of the 2nd International Conference on Art, Science, and Engineering of Programming, Nice, France, 09–12 April 2018, pp. 36–41 (2018)

  21. 21.

    Nathan Foster, J., Greenwald, M.B., Moore, J.T., Pierce, B.C., Schmitt, A.: Combinators for bidirectional tree transformations: a linguistic approach to the view-update problem. ACM Trans. Program. Lang. Syst. 29(3), 17:1–17:65 (2007)

    MATH  Google Scholar 

  22. 22.

    Foster, N., Matsuda, K., Voigtländer, J.: Three complementary approaches to bidirectional programming. In: Gibbons, J. (ed.) Generic and Indexed Programming—International Spring School, SSGIP 2010, Oxford, UK, 22–26 March 2010, Revised Lectures, volume 7470 of Lecture Notes in Computer Science, pp. 1–46. Springer (2010)

  23. 23.

    Garcia-Dominguez, A., Hinkel, G., Krikava, F. (eds.): Proceedings of the 10th Transformation Tool Contest (TTC 2017). CEUR Workshop Proceedings, vol. 2026. Marburg, Germany (2017)

  24. 24.

    Gelfond, M., Lifschitz, V.: The stable model semantics for logic programming. In: Kowalski, R.A., Bowen, K.A. (eds.) Logic Programming, Proceedings of the 5th International Conference and Symposium, Seattle, Washington, USA, 15–19 Aug 1988 (2 volumes), pp. 1070–1080. MIT Press (1988)

  25. 25.

    Hidaka, S., Tisi, M., Cabot, J., Zhenjiang, H.: Feature-based classification of bidirectional transformation approaches. Softw. Syst. Model. 15(3), 907–928 (2016)

    Article  Google Scholar 

  26. 26.

    Hildebrandt, S., Lambers, L., Giese, H., Rieke, J., Greenyer, J., Schäfer, W., Lauder, M., Anjorin, A., Schürr, A.: A survey of triple graph grammar tools. ECEASST 57, 1 (2013)

    Google Scholar 

  27. 27.

    Hinkel, G.: An NMF solution to the Families to Persons case at the TTC 2017. In: Garcia-Dominguez et al. [23], pp. 35–39

  28. 28.

    Hinkel, G., Burger, E.: Change propagation and bidirectionality in internal transformation DSLs. Softw. Syst. Model. 18(1), 249–278 (2019)

    Article  Google Scholar 

  29. 29.

    Hinkel, G., Goldschmidt, T., Burger, E., Reussner, R.H.: Using internal domain-specific languages to inherit tool support and modularity for model transformations. Softw. Syst. Model. 18(1), 129–155 (2019)

    Article  Google Scholar 

  30. 30.

    Hinkel, G., Heinrich, R., Reussner, R.: An extensible approach to implicit incremental model analyses. Softw. Syst. Model. 18, 3151–3187 (2019)

    Article  Google Scholar 

  31. 31.

    Izsó, B., Szárnyas, G, Ráth, I., Varró, D.: MONDO-SAM: a framework to systematically assess MDE scalability. In: Kolovos, D.S., Di Ruscio, D., Drivalos Matragkas, N., de Lara, J., Ráth, I., Tisi, M. (eds.) Proceedings of the 2nd Workshop on Scalability in Model Driven Engineering co-located with the Software Technologies: Applications and Foundations Conference, BigMDE@STAF2014, York, UK, July 24, 2014, volume 1206 of CEUR Workshop Proceedings, pp. 40–43. (2014)

  32. 32.

    Jouault, F., Allilaire, F., Bézivin, J., Kurtev, I.: ATL: a model transformation tool. Sci. Comput. Program. 72(1–2), 31–39 (2008)

    MathSciNet  Article  Google Scholar 

  33. 33.

    Ko, H.-S., Hu, Z.: An axiomatic basis for bidirectional programming. Proc. ACM Program. Lang. 2(POPL), 41:1–41:29 (2018)

    Article  Google Scholar 

  34. 34.

    Ko, H.-S., Zan, T., Hu, Z.: BiGUL: a formally verified core language for putback-based bidirectional programming. In: Erwig, M., Rompf, T. (eds.) Proceedings of the 2016 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation, PEPM 2016, St. Petersburg, FL, USA, 20–22 Jan 2016, pp. 61–72. ACM (2016)

  35. 35.

    Kolovos, D., Rose, L., Paige, R., Garcia-Dominguez, A.: The Epsilon Book. (2018)

  36. 36.

    Lämmel, R.: Coupled software transformations revisited. In: van der Storm, T., Balland, E., Varró, D. (eds.) Proceedings of the 2016 ACM SIGPLAN International Conference on Software Language Engineering, Amsterdam, The Netherlands, 31 Oct–1 Nov 2016, pp. 239–252. ACM (2016)

  37. 37.

    Leblebici, E., Anjorin, A., Schürr, A.: Developing eMoflon with eMoflon. In: Di Ruscio, D., Varró, D. (eds.) Theory and Practice of Model Transformations—7th International Conference, ICMT 2014, Held as Part of STAF 2014, York, UK, 21–22 July 2014. Proceedings, volume 8568 of Lecture Notes in Computer Science, pp. 138–145. Springer (2014)

  38. 38.

    Leblebici, E., Anjorin, A., Schürr, A., Hildebrandt, S., Rieke, J., Greenyer, J.: A comparison of incremental triple graph grammar tools. ECEASST 67, 1 (2014)

    Google Scholar 

  39. 39.

    Macedo, N., Cunha, A.: Least-change bidirectional model transformation with QVT-R and ATL. Softw. Syst. Model. 15(3), 783–810 (2016)

    Article  Google Scholar 

  40. 40.

    Norbisrath, U., Jubeh, R., Zündorf, A.: Story Driven Modeling. CreateSpace Independent Publishing Platform, Scotts Valley (2013)

    Google Scholar 

  41. 41.

    Norell, U.: Dependently typed programming in Agda. In: Koopman, P.W.M., Plasmeijer, R., Doaitse Swierstra, S. (eds.) Advanced Functional Programming, 6th International School, AFP 2008, Heijen, The Netherlands, May 2008, Revised Lectures, volume 5832 of Lecture Notes in Computer Science, pp. 230–266. Springer (2008)

  42. 42.

    Object Management Group. Meta Object Facility (MOF) 2.0 Query/View/Transformation Specification Version 1.3. Needham, MA, formal/2016-06-03 edition, Feb 2016

  43. 43.

    Object Management Group. OMG Meta Object Facility (MOF) Core Specification Version 2.5.1. Needham, MA, formal/2016-11-01 edition, Nov 2016

  44. 44.

    Rosenberg, J.: Some misconceptions about lines of code. In: 4th IEEE International Software Metrics Symposium (METRICS 1997), 5–7 Nov 1997, Albuquerque, NM, USA, p. 137. IEEE Computer Society (1997)

  45. 45.

    Samimi-Dehkordi, L., Zamani, B., Kolahdouz-Rahimi, S.: EVL+Strace: a novel bidirectional transformation approach. Inf. Softw. Technol. 100, 47–72 (2018)

    Article  Google Scholar 

  46. 46.

    Samimi-Dehkordi, L., Zamani, B., Kolahdouz Rahimi, S.: Solving the Families to Persons case using EVL+Strace. In: Garcia-Dominguez et al. [23], pp. 54–62

  47. 47.

    Schürr, A.: Specification of graph translators with triple graph grammars. In: Mayr, E.W., Schmidt, G., Tinhofer, G. (eds.) Graph-Theoretic Concepts in Computer Science, 20th International Workshop, WG’94, Herrsching, Germany, 16–18 June 1994, Proceedings, volume 903 of Lecture Notes in Computer Science, pp. 151–163. Springer (1994)

  48. 48.

    Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF Eclipse Modeling Framework. The Eclipse Series, 2nd edn. Addison-Wesley, Upper Saddle River, NJ (2009)

    Google Scholar 

  49. 49.

    Stevens, P.: Bidirectional model transformations in QVT: semantic issues and open questions. Softw. Syst. Model. 9(1), 7–20 (2010)

    Article  Google Scholar 

  50. 50.

    Strüber, D., Kehrer, T., Arendt, T., Pietsch, C., Reuling, D.: Scalability of model transformations: position paper and benchmark set. In: Kolovos, D.S., Di Ruscio, D., Drivalos Matragkas, N., Sánchez Cuadrado, J., Ráth, I., Tisi, M. (eds.) Proceedings of the 4th Workshop on Scalable Model Driven Engineering, Vienna, Austria, 8 July 2016, volume 1652 of CEUR Workshop Proceedings, pp. 21–30. (2016)

  51. 51.

    Van Gorp, P., Mazanek, S.: SHARE: a web portal for creating and sharing executable research papers. In: Sato, M., Matsuoka, S., Sloot, P.M.A., Dick van Albada, G., Dongarra, J.J. (eds.) Proceedings of the International Conference on Computational Science, ICCS 2011, Nanyang Technological University, Singapore, 1–3 June 2011, volume 4 of Procedia Computer Science, pp. 589–597. Elsevier (2011)

  52. 52.

    Varró, G., Schürr, A., Varró, D.: Benchmarking for graph transformation. In: 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2005), 21–24 Sept 2005, Dallas, TX, USA, pp. 79–88. IEEE Computer Society (2005)

  53. 53.

    Westfechtel, B.: Case-based exploration of bidirectional transformations in QVT Relations. Softw. Syst. Model. 17(3), 989–1029 (2018)

    Article  Google Scholar 

  54. 54.

    Westfechtel, B.: Incremental bidirectional transformations: applying QVT relations to the families to persons benchmark. In: Damiani, E., Spanoudakis, G, Maciaszek, L. (eds.) Proceedings of the 13th International Conference on the Evaluation of Novel Approaches to Software Engineering (ENASE 2018), pp. 39–53, Funchal, Madeira, March 2018. SciTePress

  55. 55.

    Wohlin, C., Runeson, P., Hst, M., Ohlsson, M.C., Regnell, B., Wessln, A.: Experimentation in Software Engineering. Springer, Berlin (2012)

    Book  Google Scholar 

  56. 56.

    Zündorf, A., Weidt, A.: The SDMLib solution to the TTC 2017 Families 2 Persons case. In: Garcia-Dominguez et al. [23], pp. 41–45

Download references

Author information



Corresponding author

Correspondence to Anthony Anjorin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This paper is an extended version of work by Anjorin et al. [4]. We would like to thank Erhan Leblebici and our anonymous reviewers for their useful input and suggestions.

Communicated by Prof. Zhenjiang Hu.



  • Architecture (of a bx tool) External interface, defined by required inputs and provided outputs, and internal processing, defined by processing steps and their organization.

  • Automatic synchronization Synchronization that is not interactive.

  • Backward synchronization Directed synchronization in the direction of the source model.

  • Batch synchronization Directed synchronization that creates a new dependent model.

  • Benchmark A standardized test that serves as a basis for comparison or evaluation.

  • Bidirectional transformation (bx) A transformation that synchronizes a source model with a target model.

  • bx language A domain-specific language for defining bidirectional transformations.

  • bx law A condition that is satisfied by bidirectional transformations.

  • bx tool A tool for executing bidirectional transformations.

  • Change Any modification to the contents of a model.

  • Completeness The ability of a bidirectional transformation to process all models in the domain or range of the consistency relation.

  • Concurrent synchronization Synchronization in which both source and target models may be changed.

  • Conformance Satisfaction of requirements.

  • Consistency A condition on pairs of source and target models that ensures that both models agree on shared information.

  • Consistency relation A binary relation that includes all pairs of source and target models that are mutually consistent.

  • Correctness A bx law that demands consistency between the source model and the target model.

  • Delta Difference between two versions of the same model.

  • Dependent model The model that is created or changed in a directed synchronization.

  • Directed synchronization Synchronization from a master model to a dependent model.

  • Forward synchronization Directed synchronization in the direction of the target model.

  • Hippocraticness A bx law that excludes changes on source and target models that are already consistent before the execution of a bidirectional transformation.

  • Incremental synchronization Synchronization that modifies an already existing model.

  • Interactive synchronization Synchronization that is partially controlled by user interactions being performed during the synchronization.

  • Least change synchronization Synchronization that performs a minimal change with respect to a suitable metric to reestablish consistency.

  • Least surprise synchronization Synchronization which minimizes the surprise of the user of a bx tool and thus maximizes conformance to the user’s expectations.

  • Live synchronization Synchronization that is performed immediately after each elementary change.

  • Master model The model that is read but not changed in a directed synchronization.

  • Metamodel Model that defines the structure of a set of models.

  • Model Abstraction of a system under study.

  • Operational delta A delta which is defined by a sequence of change operations from an old to a new version of a model.

  • Round-trip law A bx law that refers to a round trip of directed synchronizations being performed in sequence.

  • Source model A model that may act as first component of a pair in the consistency relation maintained by a bidirectional transformation.

  • Structural delta A delta which is defined in terms of structural elements being contained in both or only one of the two model versions.

  • Symmetric synchronization Synchronization between two models, where neither model is a view of its opposite.

  • Synchronization Execution of a bidirectional transformation with the intent to establish or restore consistency between a source model and a target model.

  • Synchronization on-demand Synchronization that is performed only on explicit or implicit user request.

  • System Generic concept for designating a software application, software platform, or any other software artifact.

  • Target model A model that may act as second component of a pair in the consistency relation maintained by a bidirectional transformation.

  • Transformation A procedure that reads, creates, or changes a set of models.

  • Transformation definition The program that controls the transformation.

  • Termination An execution of a transformation which halts after a finite number of steps.

  • Version A state of an evolving model at a specific point in time, defined in terms of the model’s contents at that time.

  • View An abstraction of a model that may be computed automatically from the model’s content.

  • View-based synchronization Synchronization between a model and a view on this model.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Anjorin, A., Buchmann, T., Westfechtel, B. et al. Benchmarking bidirectional transformations: theory, implementation, application, and assessment. Softw Syst Model 19, 647–691 (2020).

Download citation


  • Bidirectional transformation
  • Benchmark
  • Model synchronization
  • Framework