Assessing the Performance of Automated Model Extraction Rules

  • Jorge EcheverríaEmail author
  • Francisca Pérez
  • Óscar Pastor
  • Carlos Cetina
Conference paper
Part of the Lecture Notes in Information Systems and Organisation book series (LNISO, volume 26)


Automated Model Extraction Rules take as input requirements (in natural language) to generate domain models. Despite the existing work on these rules, there is a lack of evaluations in industrial settings. To address this gap, we conduct an evaluation in an industrial context, reporting the extraction rules that are triggered to create a model from requirements and their frequency. We also assess the performance in terms of recall, precision and F-measure of the generated model compared to the models created by domain experts of our industrial partner. Results enable us to identify new research directions to push forward automated model extraction rules: the inclusion of new knowledge sources as input for the extraction rules, and the development of specific experiments to evaluate the understanding of the generated models.


Conceptual models Natural language requirements Model extraction 



This work has been partially supported by the Ministry of Economy and Competitiveness (MINECO) through the Spanish National R+D+i Plan and ERDF funds under the project Model-Driven Variability Extraction for Software Product Line Adoption (TIN2015-64397-R).


  1. 1.
    Pohl, K., Rupp, C.: Requirements Engineering Fundamentals, 1st ed. Rocky Nook (2011)Google Scholar
  2. 2.
    Yue, T., Briand, L.C., Labiche, Y.: A systematic review of transformation approaches between user requirements and analysis models. Requir. Eng. 16(2), 75–99 (2011)CrossRefGoogle Scholar
  3. 3.
    Deeptimahanti, D.K., Sanyal, R.: Semi-automatic generation of UML models from natural language requirements. In: Proceedings of the 4th India Software Engineering Conference, pp. 165–174. ACM, New York, NY, USA (2011)Google Scholar
  4. 4.
    Elbendak, M., Vickers, P., Rossiter, B.N.: Parsed use case descriptions as a basis for object-oriented class model generation. J. Syst. Softw. 84, 1209–1223 (2011)CrossRefGoogle Scholar
  5. 5.
    Ibrahim, M., Ahmad, R.: Class diagram extraction from textual requirements using natural language processing (NLP) techniques. In: 2010 Second International Conference on Computer Research and Development, pp. 200–204, (2010)Google Scholar
  6. 6.
    Popescu, D., Rugaber, S., Medvidovic, N., Berry, D.M.: Reducing ambiguities in requirements specifications via automatically created object-oriented models. In: Monterey Workshop, pp. 103–124 (2007)Google Scholar
  7. 7.
    Sagar, V.B.R.V., Abirami, S.: Conceptual modeling of natural language functional requirements. J. Syst. Softw. 88, 25–41 (2014)CrossRefGoogle Scholar
  8. 8.
    Yue, T., Briand, L.C., Labiche, Y.: aToucan: an automated framework to derive uml analysis models from use case models. ACM Trans. Softw. Eng. Methodol. 24(3), 13:1–13:52 (2015)Google Scholar
  9. 9.
    Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated checking of conformance to requirements templates using natural language processing. IEEE Trans. Softw. Eng. 41(10), 944–968 (2015)CrossRefGoogle Scholar
  10. 10.
    Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Extracting domain models from natural-language requirements: approach and industrial evaluation. In: Proceedings of the ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems, pp. 250–260. ACM, New York, NY, USA (2016)Google Scholar
  11. 11.
    Akbik, A., Broß, J.: Wanderlust: Extracting semantic relations from natural language text using dependency grammar patterns. In: WWW Workshop (2009)Google Scholar
  12. 12.
    Rosadini, B., Ferrari, A., Gori, G., Fantechi, A., Gnesi, S., Trotta, I., Bacherini, S.: Using NLP to detect requirements defects: an industrial experience in the railway domain. In: Requirements Engineering: Foundation for Software Quality—23rd International Working Conference, REFSQ 2017, Essen, Germany, Feb 27–Mar 2, 2017, Proceedings, pp. 344–360 (2017)Google Scholar
  13. 13.
    Fanmuy, G., Fraga, A., Lloréns, J.: Requirements verification in the industry. In: Proceedings of the Second International Conference on Complex Systems Design and Management, CSDM 2011, Paris, 7–9 Dec 2011, pp. 145–160 (2011)Google Scholar
  14. 14.
    Lucas, F.J., Molina, F., Toval, A.: A systematic review of UML model consistency management. Inf. Softw. Technol. 51(12), 1631–1645 (2009)CrossRefGoogle Scholar
  15. 15.
    Zave, P.: Classification of research efforts in requirements engineering. ACM Comput. Surv. 29(4), 315–321 (1997)CrossRefGoogle Scholar
  16. 16.
    Capobianco, G., Lucia, A. De, Oliveto, R., Panichella, A., Panichella, S.: On the role of the nouns in IR-based Traceability Recovery. In: Proceedings of the International Conference on Program Comprehension, pp. 148–157. IEEE (2009)Google Scholar
  17. 17.
    Runeson, P., Höst, M.: Guidelines for conducting and reporting case study research in software engineering. Empir. Softw. Eng. 14(2), 131–164 (2009)CrossRefGoogle Scholar
  18. 18.
    Robeer, M., Lucassen, G., v. d. Werf, J.M.E.M., Dalpiaz, F., Brinkkemper, S.: Automated extraction of conceptual models from user stories via NLP. In: 2016 IEEE 24th International Requirements Engineering Conference (RE), pp. 196–205, (2016)Google Scholar
  19. 19.
    Ferrari, A., Dell’Orletta, F., Spagnolo, G.O., Gnesi, S.: Measuring and improving the completeness of natural language requirements. LNCS (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics). 8396 LNCS, pp. 23–38 (2014)Google Scholar
  20. 20.
    Ambriola, V., Gervasi, V.: On the systematic analysis of natural language requirements with CIRCE. Autom. Softw. Eng. 13(1), 107–167 (2006)CrossRefGoogle Scholar
  21. 21.
    Ben Abdessalem Karaa, W., Ben Azzouz, Z., Singh, A., Dey, N., S. Ashour, A., Ben Ghazala, H.: Automatic Builder of Class Diagram ABCD: An Application of UML Generation from Functional Requirements. Softw. Pr. Exper. 46(11), 1443–1458 (2016)Google Scholar
  22. 22.
    Thakur, J.S., Gupta, A.: Identifying domain elements from textual specifications. In: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, pp. 566–577. ACM, New York, NY, USA (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Jorge Echeverría
    • 1
    Email author
  • Francisca Pérez
    • 1
  • Óscar Pastor
    • 2
  • Carlos Cetina
    • 1
  1. 1.Universidad San JorgeZaragozaSpain
  2. 2.Universitat Politècnica de ValènciaValenciaSpain

Personalised recommendations