Benchmarking bidirectional transformations: theory, implementation, application, and assessment

  • Anthony AnjorinEmail author
  • Thomas Buchmann
  • Bernhard Westfechtel
  • Zinovy Diskin
  • Hsiang-Shang Ko
  • Romina Eramo
  • Georg Hinkel
  • Leila Samimi-Dehkordi
  • Albert Zündorf
Regular Paper


Bidirectional transformations (bx) are relevant for a wide range of application domains. While bx problems may be solved with unidirectional languages and tools, maintaining separate implementations of forward and backward synchronizers with mutually consistent behavior can be difficult, laborious, and error-prone. To address the challenges involved in handling bx problems, dedicated languages and tools for bx have been developed. Due to their heterogeneity, however, the numerous and diverse approaches to bx are difficult to compare, with the consequence that fundamental differences and similarities are not yet well understood. This motivates the need for suitable benchmarks that facilitate the comparison of bx approaches. This paper provides a comprehensive treatment of benchmarking bx, covering theory, implementation, application, and assessment. At the level of theory, we introduce a conceptual framework that defines and classifies architectures of bx tools. At the level of implementation, we describe Benchmarx, an infrastructure for benchmarking bx tools which is based on the conceptual framework. At the level of application, we report on a wide variety of solutions to the well-known Families-to-Persons benchmark, which were developed and compared with the help of Benchmarx. At the level of assessment, we reflect on the usefulness of the Benchmarx approach to benchmarking bx, based on the experiences gained from the Families-to-Persons benchmark.


Bidirectional transformation Benchmark Model synchronization Framework 



  1. 1.
    Anjorin, A.: An introduction to triple graph grammars as an implementation of the delta-lens framework. In: Gibbons, J., Stevens, P. (eds.) Bidirectional Transformations—International Summer School, Oxford, UK, 25–29 July 2016, Tutorial Lectures, volume of 9715 Lecture Notes in Computer Science, pp. 29–72. Springer (2016)Google Scholar
  2. 2.
    Anjorin, A., Buchmann, T., Westfechtel, B.: The Families to Persons case. In: Garcia-Dominguez et al. [23], pp. 27–34Google Scholar
  3. 3.
    Anjorin, A., Cunha, A., Giese, H., Hermann, F., Rensink, A., Schürr, A.: Benchmarx. In: Candan et al. [8], pp. 82–86Google Scholar
  4. 4.
    Anjorin, A., Diskin, Z., Jouault, F., Ko, H.-S., Leblebici, E., Westfechtel, B.: Benchmarx reloaded: a practical benchmark framework for bidirectional transformations. In: Eramo, R., Johnson, M. (eds.) Proceedings of the 6th International Workshop on Bidirectional Transformations Co-located with The European Joint Conferences on Theory and Practice of Software, BX@ETAPS 2017, Uppsala, Sweden, April 29, 2017, volume of 1827 CEUR Workshop Proceedings, pp. 15–30. (2017)Google Scholar
  5. 5.
    Bergmann, G., Horváth, Á., Ráth, I., Daniel Varró. A benchmark evaluation of incremental pattern matching in graph transformation. In: Ehrig, H., Heckel, R., Rozenberg, G., Taentzer, G. (eds.) Graph Transformations, 4th International Conference, ICGT 2008, Leicester, UK, 7–13 Sept 2008. Proceedings, volume of 5214 Lecture Notes in Computer Science, pp. 396–410. Springer (2008)Google Scholar
  6. 6.
    Buchmann, T.: BXtend—a framework for (bidirectional) model transformations. In: Hamoudi, S., Ferreira Pires, L., Selic, B. (eds.) Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development: MODELSWARD (MODELSWARD 2018), vol. 1, pp. 336–345, Funchal, Madeira, Jan 2018. SciTePressGoogle Scholar
  7. 7.
    Buchmann, T., Greiner, S.: Handcrafting a triple graph transformation system to realize round-trip engineering between UML class models and java source code. In: Maciaszek, L.A., Cardoso, J.S., Ludwig, A., van Sinderen, M., Cabello, E. (eds.) Proceedings of the 11th International Joint Conference on Software Technologies (ICSOFT 2016)—Volume 2: ICSOFT-PT, Lisbon, Portugal, 24–26 July 2016, pp. 27–38. SciTePress (2016)Google Scholar
  8. 8.
    Candan, K.S., Amer-Yahia, S., Schweikardt, N., Christophides, V., Leroy, V. (eds.): Proceedings of the Workshops of the EDBT/ICDT 2014 Joint Conference (EDBT/ICDT 2014), Athens, Greece, March 28, 2014, volume 1133 of CEUR Workshop Proceedings. (2014)Google Scholar
  9. 9.
    Cheney, J., Gibbons, J., McKinna, J., Stevens, P.: Towards a principle of least surprise for bidirectional transformations. In: Cunha, A., Kindler, E. (eds.) Proceedings of the 4th International Workshop on Bidirectional Transformations Co-located with Software Technologies: Applications and Foundations, STAF 2015, L’Aquila, Italy, 24 July 2015, volume 1396 of CEUR Workshop Proceedings, pp. 66–80. (2015)Google Scholar
  10. 10.
    Cheney, J., McKinna, J., Stevens, P., Gibbons, J.: Towards a repository of bx examples. In: Candan et al. [8], pp. 87–91Google Scholar
  11. 11.
    Cicchetti, A., Di Ruscio, D., Eramo, R., Pierantonio, A.: JTL: a bidirectional and change propagating transformation language. In: Malloy, B., Staab, S., van den Brand, M. (eds.) Proceedings of the 3rd International Conference on Software Language Engineering (SLE 2010), volume 6563 of Lecture Notes of Computer Science, pp. 183–202, Eindhoven, The Netherlands, Oct 2010. SpringerGoogle Scholar
  12. 12.
    Czarnecki, K., Nathan Foster, J., Hu, Z., Lämmel, R., Schürr, A., Terwilliger, J.F.: Bidirectional transformations: a cross-discipline perspective. In: Paige, R.F. (ed.) Proceedings of the Second International Conference on Theory and Practice of Model Transformations (ICMT 2009), volume 5563 of Lecture Notes of Computer Science, pp. 260–283, Zurich, Switzerland, June 2009. SpringerGoogle Scholar
  13. 13.
    da Silva, A.R.: Model-driven engineering: a survey supported by the unified conceptual model. Comput. Lang. Syst. Struct. 43, 139–155 (2015)Google Scholar
  14. 14.
    Diskin, Z.: Model synchronization: mappings, tiles, and categories. In: Fernandes, J.M., Lämmel, R., Visser, J., Saraiva, J. (eds.) Generative and Transformational Techniques in Software Engineering III—International Summer School, GTTSE 2009, Braga, Portugal, 6–11 July 2009. Revised Papers, volume 6491 of Lecture Notes in Computer Science, pp. 92–165. Springer (2009)Google Scholar
  15. 15.
    Diskin, Z., Gholizadeh, H., Wider, A., Czarnecki, K.: A three-dimensional taxonomy for bidirectional model synchronization. J. Syst. Softw. 111, 298–322 (2016)CrossRefGoogle Scholar
  16. 16.
    Diskin, Z., Xiong, Y., Czarnecki, K.: From state- to delta-based bidirectional model transformations: the asymmetric case. J. Object Technol. 10(6), 1–25 (2011)Google Scholar
  17. 17.
    Ehrig, K., Guerra, E., De Lara, J., Lengyel, L., Levendovszky, T., Prange, U., Taentzer, G., Varró, D., Varro-Gyapay, S.: Model transformation by graph transformation: a comparative study. In: Proceedings of the International Workshop on Model Transformations in Practice (MTiP 2005), Satellite Event of MoDELS 2005, volume 3844 of Lecture Notes of Computer Science, pp. 71–80, Montego Bay, Jamaica, 2005. SpringerGoogle Scholar
  18. 18.
    Eramo, R., Bucaioni, A.: Understanding bidirectional transformations with TGGs and JTL. ECEASST 57, 1 (2013)Google Scholar
  19. 19.
    Eramo, R., Marinelli, R., Pierantonio, A.: Towards a taxonomy for bidirectional transformation. In: Di Ruscio, D., Zaytsev, V. (eds.) Post-proceedings of the Seventh Seminar on Advanced Techniques and Tools for Software Evolution, SATToSE 2014, L’Aquila, Italy, 9–11 July 2014, volume 1354 of CEUR Workshop Proceedings, pp. 122–131. (2014)Google Scholar
  20. 20.
    Eramo, R., Pierantonio, A., Tucci, M.: Enhancing the JTL tool for bidirectional transformations. In: Conference Companion of the 2nd International Conference on Art, Science, and Engineering of Programming, Nice, France, 09–12 April 2018, pp. 36–41 (2018)Google Scholar
  21. 21.
    Nathan Foster, J., Greenwald, M.B., Moore, J.T., Pierce, B.C., Schmitt, A.: Combinators for bidirectional tree transformations: a linguistic approach to the view-update problem. ACM Trans. Program. Lang. Syst. 29(3), 17:1–17:65 (2007)zbMATHGoogle Scholar
  22. 22.
    Foster, N., Matsuda, K., Voigtländer, J.: Three complementary approaches to bidirectional programming. In: Gibbons, J. (ed.) Generic and Indexed Programming—International Spring School, SSGIP 2010, Oxford, UK, 22–26 March 2010, Revised Lectures, volume 7470 of Lecture Notes in Computer Science, pp. 1–46. Springer (2010)Google Scholar
  23. 23.
    Garcia-Dominguez, A., Hinkel, G., Krikava, F. (eds.): Proceedings of the 10th Transformation Tool Contest (TTC 2017). CEUR Workshop Proceedings, vol. 2026. Marburg, Germany (2017)Google Scholar
  24. 24.
    Gelfond, M., Lifschitz, V.: The stable model semantics for logic programming. In: Kowalski, R.A., Bowen, K.A. (eds.) Logic Programming, Proceedings of the 5th International Conference and Symposium, Seattle, Washington, USA, 15–19 Aug 1988 (2 volumes), pp. 1070–1080. MIT Press (1988)Google Scholar
  25. 25.
    Hidaka, S., Tisi, M., Cabot, J., Zhenjiang, H.: Feature-based classification of bidirectional transformation approaches. Softw. Syst. Model. 15(3), 907–928 (2016)CrossRefGoogle Scholar
  26. 26.
    Hildebrandt, S., Lambers, L., Giese, H., Rieke, J., Greenyer, J., Schäfer, W., Lauder, M., Anjorin, A., Schürr, A.: A survey of triple graph grammar tools. ECEASST 57, 1 (2013)Google Scholar
  27. 27.
    Hinkel, G.: An NMF solution to the Families to Persons case at the TTC 2017. In: Garcia-Dominguez et al. [23], pp. 35–39Google Scholar
  28. 28.
    Hinkel, G., Burger, E.: Change propagation and bidirectionality in internal transformation DSLs. Softw. Syst. Model. 18(1), 249–278 (2019)CrossRefGoogle Scholar
  29. 29.
    Hinkel, G., Goldschmidt, T., Burger, E., Reussner, R.H.: Using internal domain-specific languages to inherit tool support and modularity for model transformations. Softw. Syst. Model. 18(1), 129–155 (2019)CrossRefGoogle Scholar
  30. 30.
    Hinkel, G., Heinrich, R., Reussner, R.: An extensible approach to implicit incremental model analyses. Softw. Syst. Model. 18, 3151–3187 (2019)CrossRefGoogle Scholar
  31. 31.
    Izsó, B., Szárnyas, G, Ráth, I., Varró, D.: MONDO-SAM: a framework to systematically assess MDE scalability. In: Kolovos, D.S., Di Ruscio, D., Drivalos Matragkas, N., de Lara, J., Ráth, I., Tisi, M. (eds.) Proceedings of the 2nd Workshop on Scalability in Model Driven Engineering co-located with the Software Technologies: Applications and Foundations Conference, BigMDE@STAF2014, York, UK, July 24, 2014, volume 1206 of CEUR Workshop Proceedings, pp. 40–43. (2014)Google Scholar
  32. 32.
    Jouault, F., Allilaire, F., Bézivin, J., Kurtev, I.: ATL: a model transformation tool. Sci. Comput. Program. 72(1–2), 31–39 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Ko, H.-S., Hu, Z.: An axiomatic basis for bidirectional programming. Proc. ACM Program. Lang. 2(POPL), 41:1–41:29 (2018)Google Scholar
  34. 34.
    Ko, H.-S., Zan, T., Hu, Z.: BiGUL: a formally verified core language for putback-based bidirectional programming. In: Erwig, M., Rompf, T. (eds.) Proceedings of the 2016 ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation, PEPM 2016, St. Petersburg, FL, USA, 20–22 Jan 2016, pp. 61–72. ACM (2016)Google Scholar
  35. 35.
    Kolovos, D., Rose, L., Paige, R., Garcia-Dominguez, A.: The Epsilon Book. (2018)
  36. 36.
    Lämmel, R.: Coupled software transformations revisited. In: van der Storm, T., Balland, E., Varró, D. (eds.) Proceedings of the 2016 ACM SIGPLAN International Conference on Software Language Engineering, Amsterdam, The Netherlands, 31 Oct–1 Nov 2016, pp. 239–252. ACM (2016)Google Scholar
  37. 37.
    Leblebici, E., Anjorin, A., Schürr, A.: Developing eMoflon with eMoflon. In: Di Ruscio, D., Varró, D. (eds.) Theory and Practice of Model Transformations—7th International Conference, ICMT 2014, Held as Part of STAF 2014, York, UK, 21–22 July 2014. Proceedings, volume 8568 of Lecture Notes in Computer Science, pp. 138–145. Springer (2014)Google Scholar
  38. 38.
    Leblebici, E., Anjorin, A., Schürr, A., Hildebrandt, S., Rieke, J., Greenyer, J.: A comparison of incremental triple graph grammar tools. ECEASST 67, 1 (2014)Google Scholar
  39. 39.
    Macedo, N., Cunha, A.: Least-change bidirectional model transformation with QVT-R and ATL. Softw. Syst. Model. 15(3), 783–810 (2016)CrossRefGoogle Scholar
  40. 40.
    Norbisrath, U., Jubeh, R., Zündorf, A.: Story Driven Modeling. CreateSpace Independent Publishing Platform, Scotts Valley (2013)Google Scholar
  41. 41.
    Norell, U.: Dependently typed programming in Agda. In: Koopman, P.W.M., Plasmeijer, R., Doaitse Swierstra, S. (eds.) Advanced Functional Programming, 6th International School, AFP 2008, Heijen, The Netherlands, May 2008, Revised Lectures, volume 5832 of Lecture Notes in Computer Science, pp. 230–266. Springer (2008)Google Scholar
  42. 42.
    Object Management Group. Meta Object Facility (MOF) 2.0 Query/View/Transformation Specification Version 1.3. Needham, MA, formal/2016-06-03 edition, Feb 2016Google Scholar
  43. 43.
    Object Management Group. OMG Meta Object Facility (MOF) Core Specification Version 2.5.1. Needham, MA, formal/2016-11-01 edition, Nov 2016Google Scholar
  44. 44.
    Rosenberg, J.: Some misconceptions about lines of code. In: 4th IEEE International Software Metrics Symposium (METRICS 1997), 5–7 Nov 1997, Albuquerque, NM, USA, p. 137. IEEE Computer Society (1997)Google Scholar
  45. 45.
    Samimi-Dehkordi, L., Zamani, B., Kolahdouz-Rahimi, S.: EVL+Strace: a novel bidirectional transformation approach. Inf. Softw. Technol. 100, 47–72 (2018)CrossRefGoogle Scholar
  46. 46.
    Samimi-Dehkordi, L., Zamani, B., Kolahdouz Rahimi, S.: Solving the Families to Persons case using EVL+Strace. In: Garcia-Dominguez et al. [23], pp. 54–62Google Scholar
  47. 47.
    Schürr, A.: Specification of graph translators with triple graph grammars. In: Mayr, E.W., Schmidt, G., Tinhofer, G. (eds.) Graph-Theoretic Concepts in Computer Science, 20th International Workshop, WG’94, Herrsching, Germany, 16–18 June 1994, Proceedings, volume 903 of Lecture Notes in Computer Science, pp. 151–163. Springer (1994)Google Scholar
  48. 48.
    Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF Eclipse Modeling Framework. The Eclipse Series, 2nd edn. Addison-Wesley, Upper Saddle River, NJ (2009)Google Scholar
  49. 49.
    Stevens, P.: Bidirectional model transformations in QVT: semantic issues and open questions. Softw. Syst. Model. 9(1), 7–20 (2010)MathSciNetCrossRefGoogle Scholar
  50. 50.
    Strüber, D., Kehrer, T., Arendt, T., Pietsch, C., Reuling, D.: Scalability of model transformations: position paper and benchmark set. In: Kolovos, D.S., Di Ruscio, D., Drivalos Matragkas, N., Sánchez Cuadrado, J., Ráth, I., Tisi, M. (eds.) Proceedings of the 4th Workshop on Scalable Model Driven Engineering, Vienna, Austria, 8 July 2016, volume 1652 of CEUR Workshop Proceedings, pp. 21–30. (2016)Google Scholar
  51. 51.
    Van Gorp, P., Mazanek, S.: SHARE: a web portal for creating and sharing executable research papers. In: Sato, M., Matsuoka, S., Sloot, P.M.A., Dick van Albada, G., Dongarra, J.J. (eds.) Proceedings of the International Conference on Computational Science, ICCS 2011, Nanyang Technological University, Singapore, 1–3 June 2011, volume 4 of Procedia Computer Science, pp. 589–597. Elsevier (2011)Google Scholar
  52. 52.
    Varró, G., Schürr, A., Varró, D.: Benchmarking for graph transformation. In: 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC 2005), 21–24 Sept 2005, Dallas, TX, USA, pp. 79–88. IEEE Computer Society (2005)Google Scholar
  53. 53.
    Westfechtel, B.: Case-based exploration of bidirectional transformations in QVT Relations. Softw. Syst. Model. 17(3), 989–1029 (2018)CrossRefGoogle Scholar
  54. 54.
    Westfechtel, B.: Incremental bidirectional transformations: applying QVT relations to the families to persons benchmark. In: Damiani, E., Spanoudakis, G, Maciaszek, L. (eds.) Proceedings of the 13th International Conference on the Evaluation of Novel Approaches to Software Engineering (ENASE 2018), pp. 39–53, Funchal, Madeira, March 2018. SciTePressGoogle Scholar
  55. 55.
    Wohlin, C., Runeson, P., Hst, M., Ohlsson, M.C., Regnell, B., Wessln, A.: Experimentation in Software Engineering. Springer, Berlin (2012)CrossRefGoogle Scholar
  56. 56.
    Zündorf, A., Weidt, A.: The SDMLib solution to the TTC 2017 Families 2 Persons case. In: Garcia-Dominguez et al. [23], pp. 41–45Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Anthony Anjorin
    • 1
    Email author
  • Thomas Buchmann
    • 2
  • Bernhard Westfechtel
    • 2
  • Zinovy Diskin
    • 3
  • Hsiang-Shang Ko
    • 4
  • Romina Eramo
    • 5
  • Georg Hinkel
    • 6
  • Leila Samimi-Dehkordi
    • 7
  • Albert Zündorf
    • 8
  1. 1.Paderborn UniversityPaderbornGermany
  2. 2.University of BayreuthBayreuthGermany
  3. 3.McMaster UniversityHamiltonCanada
  4. 4.National Institute of InformaticsTokyoJapan
  5. 5.University of L’AquilaL’AquilaItaly
  6. 6.WiesbadenGermany
  7. 7.MDSE Research Group, University of IsfahanIsfahanIran
  8. 8.University of KasselKasselGermany

Personalised recommendations