Software & Systems Modeling

, Volume 18, Issue 5, pp 3083–3096 | Cite as

Empirical study on the effectiveness and efficiency of model-driven architecture techniques

  • Shin-Shing ShinEmail author
Regular Paper


Previous studies have reported conflicting opinions on the feasibility of model-driven architecture (MDA). Studies have investigated the mechanics of MDA, but few have examined its effectiveness and efficiency from a developer’s perception. This study conducted empirical research in which a system was implemented by subjects using MDA; afterward, evaluated its perceived efficiency and effectiveness. In the model construction phase, Unified Modeling Language and Object Constraint Language were perceived as effective and efficient. In the model transformation phase, the query/view/transformation standard was perceived as marginally efficient rather than effective, and the round-trip engineering phase was not perceived as effective or efficient. These findings are explained using 12 themes identified in subjects’ opinions. This study may help scholars understand the importance of efficiency and effectiveness on MDA techniques and facilitate the development of more acceptable and practical MDA.


Model-driven architecture Effectiveness Efficiency Model transformation Unified Modeling Language Query/view/transformation 



  1. 1.
    Abrahao, S., Bourdeleau, F., Cheng, B., Kokaly, S., Paige, R., Stöerrle, H., Whittle, J.: User experience for model-driven engineering: challenges and future directions. In: Proceedings of the MODELS, pp. 229–236. IEEE, Austin, TX (2017)Google Scholar
  2. 2.
    Abran, A., Khelifi, A., Suryn, W., Seffah, A.: Usability meanings and interpretations in ISO standards. Softw. Qual. J. 11(4), 325–338 (2003)Google Scholar
  3. 3.
    Agarwal, R., Sinha, A.P.: Object-oriented modeling with UML: a study of developers’ perceptions. Commun. ACM 46(9), 248–256 (2003)Google Scholar
  4. 4.
    Albuquerque, D., Cafeo, B., Garcia, A., Barbosa, S., Abrahao, S., Ribeiro, A.: Quantifying usability of domain-specific languages: an empirical study on software maintenance. J Syst. Softw. 101(C), 245–259 (2015)Google Scholar
  5. 5.
    Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E., Moret-Bonillo, V.: Usability: a critical analysis and a taxonomy. Int. J. Hum.Comput. Int. 26(1), 53–74 (2010)Google Scholar
  6. 6.
    Al-Qutaish, R.E.: An investigation of the weaknesses of the ISO 9126 international standard. In: Proceedings of the ICCEE, pp. 275–279. IEEE, Dubai, United Arab Emirates (2009)Google Scholar
  7. 7.
    Armonas, A., Nemuraite, L.: Using Attributes and merging algorithms for transforming OCL expressions to code. Inf. Technol. Control 38(4), 283–293 (2009)Google Scholar
  8. 8.
    Asadi, M., Ramsin, R.: MDA-based methodologies: an analytical survey. In: Schieferdecker I., Hartman A. (eds.) Model Driven Architecture—Foundations and Applications. ECMDA-FA 2008. Lecture Notes in Computer Science, vol. 5095, pp. 419–431. Springer, Berlin (2008)Google Scholar
  9. 9.
    Bajwa, I.S., Lee, M.G.: Transformation rules for translating business rules to OCL constraints. In: France R.B., Kuester J.M., Bordbar B., Paige R.F. (eds.) Modelling Foundations and Applications. ECMFA 2011. Lecture Notes in Computer Science, vol. 6698, pp. 132–143. Springer, Berlin (2011)Google Scholar
  10. 10.
    Barišić, A., Amaral, V., Goulão, M., Barroca, B.: Quality in use of domain-specific languages: a case study. In: Proceedings of the 3rd ACM SIGPLAN Workshop on Evaluation and Usability of Programming Languages and Tools, pp. 65–72. ACM, Portland, OR (2011)Google Scholar
  11. 11.
    Bendraou, R., Jézéquel, J.-M., Gervais, M.-P., Blanc, X.: A comparison of six UML-based languages for software process modeling. IEEE Trans. Softw. Eng. 36(5), 662–675 (2010)Google Scholar
  12. 12.
    Bennett, J., Cooper, K., Dai, L.: Aspect-oriented model-driven skeleton code generation: a graph-based transformation approach. Sci. Comput. Program. 75(8), 689–725 (2010)zbMATHGoogle Scholar
  13. 13.
    Berardinelli, L., Bernardo, M., Cortellessa, V., Di Marco, A.: Multidimensional context modeling applied to non-functional analysis of software. Softw. Syst. Model (2017). Google Scholar
  14. 14.
    Berg, K.E., Latin, R.W.: Essentials of Research Methods in Health, Physical Education, Exercise Science, and Recreation. Lippincott Williams & Wilkins, Philadelphia (2008)Google Scholar
  15. 15.
    Bevan, N.: Measuring usability as quality of use. Softw. Qual. J. 4(2), 115–130 (1995)Google Scholar
  16. 16.
    Bevan, N., Carter, J., Earthy, J., Geis, T., Harker, S.: New ISO standards for usability, usability reports and usability measures. In: Kurosu, M. (ed.) Human-Computer Interaction. Theory, Design, Development and Practice. HCI 2016. Lecture Notes in Computer Science, vol. 9731, pp. 268–278. Springer, Cham (2016)Google Scholar
  17. 17.
    Bhiri, M.T., Mourad, K., Graiet, M., Aniorté, P.: UML/OCL and refinement. In: Proceedings of the International Conference on Engineering of Computer Based Systems, pp. 149–158. IEEE, Las Vegas (2011)Google Scholar
  18. 18.
    Boehm, B.W., Brown, J.R., Kaspar, H.: Characteristics of Software Quality. North-Holland Publishing Co., Amsterdam (1978)zbMATHGoogle Scholar
  19. 19.
    Braga, C., Menezes, R., Comicio, T., Santos, C., Landim, E.: Transformation contracts in practice. IET Softw. 6(1), 16–32 (2012)Google Scholar
  20. 20.
    Cabot, J., Teniente, E.: Transformation techniques for OCL constraints. J. Sci. Comput. Program. 68(3), 152–168 (2007)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Chen, X., Zhou, D., Wu, L.: Theory of automatic transformation from Computation Independent Model to Platform Independent Model for MDA. In: Proceedings of the ICETA, pp. 235–244. Tsingtao, China (2014)Google Scholar
  22. 22.
    Compeau, D., Higgins, C.A., Huff, S.: Social cognitive theory and individual reactions to computing technology: a longitudinal study. MIS Quart. 23(2), 145–158 (1999)Google Scholar
  23. 23.
    Curtis, B., Krasner, H., Iscoe, N.: A field study of the software design process for large systems. Commun. ACM 31(11), 1268–1287 (1988)Google Scholar
  24. 24.
    Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart. 13(3), 319–340 (1989)Google Scholar
  25. 25.
    De Castro, V., Marcos, E., Vara, J.M.: Applying CIM-to-PIM model transformations for the service-oriented development of information systems. Inf. Softw. Technol. 53(1), 87–105 (2011)Google Scholar
  26. 26.
    De Sousa Saraiva, J., Da Silva, A.R.: Chapter 5: Evaluation of MDE tools from a metamodeling perspective. In: Siau, K., Erickson, J. (eds.) Principal Advancements in Database Management Technologies, pp. 105–131. IGI Global, Hershey (2010).
  27. 27.
    Diskin, Z., Wider, A., Gholizadeh, H., Czarnecki, K.: Towards a rational taxonomy for increasingly symmetric model synchronization. In: Di Ruscio D., Varró D. (eds.) Theory and Practice of Model Transformations. ICMT 2014. Lecture Notes in Computer Science, vol. 8568, pp. 57–73. Springer, Cham (2014)Google Scholar
  28. 28.
    Domínguez-Mayo, F.J., Escalona, M.J., Mejías, M., Ross, M., Staples, G.: A quality management based on the quality model life cycle. Comput. Stand. Int. 34(4), 396–412 (2012)Google Scholar
  29. 29.
    Fabra, J., De Castro, V., Álvarez, P., Marcos, E.: Automatic execution of business process models: exploiting the benefits of model-driven engineering approaches. J. Syst. Softw. 85(3), 607–625 (2012)Google Scholar
  30. 30.
    Falessi, D., Juristo, N., Wohlin, C., Turhan, B., Münch, J., Jedlitschka, A., Oivo, M.: Empirical software engineering experts on the use of students and professionals in experiments. Empir. Softw. Eng. 23(1), 452–489 (2018)Google Scholar
  31. 31.
    Fouad, A., Phalp, K., Kanyaru, J.M., Jeary, S.: Embedding requirements within Model-Driven Architecture. Softw. Qual. J. 19(2), 411–430 (2011)Google Scholar
  32. 32.
    France, R.B., Rumpe, B.: Model-driven development of complex software: a research roadmap. In: Proceedings of the International Conference on Software Engineering, pp. 37–54. IEEE, Minneapolis (2007)Google Scholar
  33. 33.
    Gammaitoni, L., Kelsen, P.: F-Alloy: a relational model transformation language based on Alloy. Softw. Syst. Model. 1–35 (2017)Google Scholar
  34. 34.
    Gopher, D., Braune, R.: On the psychophysics of workload: Why bother with subjective measures? Hum. Factors 26(5), 519–532 (1984)Google Scholar
  35. 35.
    Guo, Y., Guan, W., Sierszecki, K., Angelov, C.: Building a software development environment for embedded systems. In: Proceedings of the International Conference on Green Computing and Communications and International Conference on Cyber, Physical and Social Computing, pp. 916–921. IEEE/ACM, Washington, DC (2010)Google Scholar
  36. 36.
    Hailpern, B., Tarr, P.: Model-driven development: the good, the bad, and the ugly. IBM Syst. J. 45(3), 451–461 (2006)Google Scholar
  37. 37.
    Hawkins, J.L., Ivkovic, I., Kontogiannis, K., McKegney, R., Ng, W., Tong, T.: U.S. Patent No. 8,418,125. Washington, DC: U.S. Patent and Trademark Office (2013)Google Scholar
  38. 38.
    Höfer, A., Tichy, W.F.: Status of empirical research in software engineering. In Basili V.R., Rombach D., Schneider K., Kitchenham B., Pfahl D., Selby R.W. (eds.) Empirical Software Engineering Issues. Critical Assessment and Future Directions. Lecture Notes in Computer Science, vol. 4336, pp. 10–19. Springer, Berlin (2007)Google Scholar
  39. 39.
    ISO 25010: System and software quality models. (2011)
  40. 40.
    ISO 9126-4, Quality in use metrics (2004) Accessed 07 Oct 2018
  41. 41.
    ISO 9241-11: Usability: Definitions and concepts (2018). Accessed 13 Feb 2018
  42. 42.
    Kuhlmann, M., Sohr, K., Gogolla, M.: Employing UML and OCL for designing and analysing role-based access control. Math. Struct. Comput. Sci. 23(04), 796–833 (2013)MathSciNetGoogle Scholar
  43. 43.
    Lehman, M.M.: Process models, process programs, programming support. In: Proceedings of the 9th International Conference on Software Engineering, pp. 14–16. IEEE, Los Alamitos (1987)Google Scholar
  44. 44.
    Lewis, J.R.: Usability: lessons learned … and yet to be learned. Int. J. Hum. Comput. Int. 30(9), 663–684 (2014)Google Scholar
  45. 45.
    Masood, Z., Xuequn, S., Yousaf, J.: Usability evaluation framework for software engineering methodologies. Lect. Notes Softw. Eng. 2(3), 225–232 (2014)Google Scholar
  46. 46.
    Mattsson, A., Fitzgerald, B., Lundell, B., Lings, B.: An approach for modeling architectural design rules in UML and its application to embedded software. ACM Trans. Softw. Eng. Methods 21(2), 10 (2012)Google Scholar
  47. 47.
    McCall, J.A., Richards, P.K., Walters, G.F.: Factors in software quality. Vol. 1. Concepts and definitions of software quality. Technical Report ADA049014. General Electric Co., Sunnyvale, California (1977)Google Scholar
  48. 48.
    McMeekin, D., von Konsky, B., Robey, M., Cooper, D.: The significance of participant experience when evaluating software inspection techniques. In: Proceedings of the ASWEC, pp. 200–209. IEEE, Gold Coast (2009)Google Scholar
  49. 49.
    Mepham, W.: Discrete event calculus using Semantic Web technologies. Ph.D. Thesis, University of Glamorgan, Trefforest, Wales (2012)Google Scholar
  50. 50.
    Meservy, T.O., Fenstermacher, K.D.: Transforming software development: an MDA road map. IEEE Comput. 38(9), 52–58 (2005)Google Scholar
  51. 51.
    Mohagheghi, P., Aagedal, J.: Evaluating quality in model-driven engineering. In: Proceedings of the International Workshop on Modeling in Software, IEEE, Minneapolis (2007)Google Scholar
  52. 52.
    Mohagheghi, P., Dehlen, V.: Developing a quality framework for model-driven engineering. In: Proceedings of the International Conference on Model Driven Engineering Languages and Systems, pp. 275–286. Springer, Berlin (2007)Google Scholar
  53. 53.
    Mohagheghi, P., Dehlen, V., Neple, T.: Definitions and approaches to model quality in model-based software development—a review of literature. Inf. Softw. Technol. 51(12), 1646–1669 (2009)Google Scholar
  54. 54.
    Moody, D.: The “physics” of notations: toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 35(6), 756–779 (2009)Google Scholar
  55. 55.
    Moore, G.C., Benbasat, I.: Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf. Syst. Res. 2(3), 192–222 (1991)Google Scholar
  56. 56.
    Mussbacher, G., Amyot, D., Breu, R., Bruel, J.M., Cheng, B.H., Collet, P., et al.: The relevance of model-driven engineering thirty years from now. In: Dingel J., Schulte W., Ramos I., Abrahão S., Insfran E. (eds.) Model-Driven Engineering Languages and Systems. MODELS 2014. Lecture Notes in Computer Science, vol. 8767, pp. 183–200. Springer, Cham (2014)Google Scholar
  57. 57.
    Nielsen, J., Mack, R.L.: Usability Inspection Methods. Wiley, New York (1994)Google Scholar
  58. 58.
    Object Management Group. MDA Guide rev. 2.0 (2014). Accessed 13 Feb 2018
  59. 59.
    O’Donnell, R.D., Eggemeier, F.T.: Workload assessment methodology. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.) Handbook of Perception and Human Performance, pp. 1–49. Wiley, New York (1986)Google Scholar
  60. 60.
    Paas, F., Van Merriënboer, J.J.G., Adam, J.J.: Measurement of cognitive-load in instructional research. Percept. Mot. Skills 79(1), 419–430 (1994)Google Scholar
  61. 61.
    Panach, J.I., Condori-Fernandez, N., Vos, T., Aquino, N., Valverde, F.: Early usability measurement in model-driven development: definition and empirical evaluation. Int. J. Softw. Eng. Knowl. 21(03), 339–365 (2011)Google Scholar
  62. 62.
    Perez-Medina, J.L., Dupuy-Chessa, S., Front, A.: A survey of model driven engineering tools for user interface design. In: Winckler M., Johnson H., Palanque P. (eds.) Task Models and Diagrams for User Interface Design. TAMODIA 2007. Lecture Notes in Computer Science, vol. 4849. pp. 84–97. Springer, Berlin (2007)Google Scholar
  63. 63.
    Petre, M.: UML in practice. In: Proceedings of the 2013 International Conference on Software Engineering, pp. 722–731. IEEE, Piscataway (2013)Google Scholar
  64. 64.
    Porter, A., Votta, L.: Comparing detection methods for software requirements inspections: a replication using professional subjects. Empir. Softw. Eng. 3(4), 355–379 (1998)Google Scholar
  65. 65.
    Ricca, F., Di Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans. Softw. Eng. 36(1), 96–118 (2010)Google Scholar
  66. 66.
    Runeson, P.: Using students as experiment subjects—an analysis on graduate and freshmen student data. In: Proceedings of the 7th International Conference on Empirical Assessment in Software Engineering, pp. 95–102. Keele, Staffordshire (2003)Google Scholar
  67. 67.
    Salman, I., Misirli, A.T., Juristo, N.: Are students representatives of professionals in software engineering experiments?. In: Proceedings of the ICSE, pp. 666–676. IEEE, Florence (2015)Google Scholar
  68. 68.
    Seddon, P.B.: A respecification and extension of the DeLone and McLean model of IS success. Inf. Syst. Res. 8(3), 240–253 (1997)Google Scholar
  69. 69.
    Seffah, A., Donyaee, M., Kline, R.B., Padda, H.K.: Usability measurement and metrics: a consolidated model. Softw. Qual. J. 14(2), 159–178 (2006)Google Scholar
  70. 70.
    Shin, S.S.: A study on the difficulties of learning phase transition in object-oriented analysis and design from the viewpoint of semantic distance. IEEE Trans. Educ. 58(2), 117–123 (2015)Google Scholar
  71. 71.
    Shin, S.S.: Evaluation of Model Driven Architecture-based instruction for understanding phase transitions in object-oriented analysis and design. ACM Trans. Comput. Educ. 16(4), 17 (2016)Google Scholar
  72. 72.
    Sjoberg, D.I., Anda, B., Arisholm, E., Dyba, T., Jorgensen, M., Karahasanovic, A., Koren, E.F., Vokác, M.: Conducting realistic experiments in software engineering. In: Proceedings of the International Symposium on Empirical Software Engineering, pp. 17–26. IEEE, Nara (2002)Google Scholar
  73. 73.
    Sjøberg, D.I., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N.K., Rekdal, A.C.: A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9), 733–753 (2005)Google Scholar
  74. 74.
    Sorokin, L., Montero, F., Märtin, C.: Flex RIA development and usability evaluation. In: Weske M., Hacid MS., Godart C. (eds.) Web Information Systems Engineering—WISE 2007 Workshops. WISE 2007. Lecture Notes in Computer Science, vol. 4832, pp. 447–452. Springer, Berlin (2007)Google Scholar
  75. 75.
    Stevens, P.: Bidirectional model transformations in QVT: semantic issues and open questions. Softw. Syst. Model. 9(1), 7–20 (2010)MathSciNetGoogle Scholar
  76. 76.
    Stoter, J., Visser, T., van Oosterom, P., Quak, W., Bakker, N.: A semantic-rich multi-scale information model for topography. Int. J. Geogr. Inf. Sci. 25(5), 739–763 (2011)Google Scholar
  77. 77.
    Thompson, R.L., Higgins, C.A., Howell, J.M.: Personal computing: toward a conceptual model of utilization. MIS Quart. 15(1), 124–143 (1991)Google Scholar
  78. 78.
    Uhl, A.: Model-driven development in the enterprise. IEEE Softw. 25(1), 46–49 (2008)Google Scholar
  79. 79.
    Vara, J.M., Marcos, E.: A framework for model-driven development of information systems: technical decisions and lessons learned. J. Syst. Softw. 85(10), 2368–2384 (2012)Google Scholar
  80. 80.
    Venkatesh, V., Morris, M.G., Davis, G.B., Davis, F.D.: User acceptance of information technology: toward a unified view. MIS Quart. 27(3), 425–478 (2003)Google Scholar
  81. 81.
    Wang, S., Ma, J., He, Q., Wan, J.: Formal behavior modeling and effective automatic refinement. Inf. Sci. 180(20), 3894–3913 (2010)Google Scholar
  82. 82.
    Whittle, J., Hutchinson, J., Rouncefield, M., Burden, H., Heldal, R.: A taxonomy of tool-related issues affecting the adoption of model-driven engineering. Softw. Syst. Model. 16(2), 313–331 (2017)Google Scholar
  83. 83.
    Whittle, J., Hutchinson, J., Rouncefield, M.: The state of practice in model-driven engineering. IEEE Softw. 31(3), 79–85 (2014)Google Scholar
  84. 84.
    Yeh, R.K.J., Teng, J.T.C.: Extended conceptualisation of perceived usefulness: empirical test in the context of information system use continuance. Behav. Inf. Technol. 31(5), 525–540 (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Information Science and Management SystemsNational Taitung UniversityTaitungTaiwan

Personalised recommendations