An NLP approach for cross-domain ambiguity detection in requirements engineering

  • Alessio FerrariEmail author
  • Andrea Esuli


During requirements elicitation, different stakeholders with diverse backgrounds and skills need to effectively communicate to reach a shared understanding of the problem at hand. Linguistic ambiguity due to terminological discrepancies may occur between stakeholders that belong to different technical domains. If not properly addressed, ambiguity can create frustration and distrust during requirements elicitation meetings, and lead to problems at later stages of development. This paper presents a natural language processing approach to identify ambiguous terms between different domains, and rank them by ambiguity score. The approach is based on building domain-specific language models, one for each stakeholders’ domain. Word embeddings from each language model are compared in order to measure the differences of use of a term, thus estimating its potential ambiguity across the domains of interest. We evaluate the approach on seven potential elicitation scenarios involving five domains. In the evaluation, we compare the ambiguity rankings automatically produced with the ones manually obtained by the authors as well as by multiple annotators recruited through Amazon Mechanical Turk. The rankings produced by the approach lead to a maximum Kendall’s Tau of 88%. However, for several elicitation scenarios, the application of the approach was unsuccessful in terms of performance. Analysis of the agreement among annotators and of the observed inaccuracies offer hints for further research on the relationship between domain knowledge and natural language ambiguity.


Natural language processing NLP Requirements engineering Ambiguity Domain knowledge Language models Word embeddings 



  1. Agirre, E., Edmonds, P.: Word Sense Disambiguation: Algorithms and Applications, vol. 33. Springer, Berlin (2007).
  2. Agresti, A.: Analysis of Ordinal Categorical Data, vol. 656. Wiley, Hoboken (2010)CrossRefzbMATHGoogle Scholar
  3. Ambriola, V., Gervasi, V.: On the systematic analysis of natural language requirements with CIRCE. ASE 13(1), 107–167 (2006).
  4. Arora, C., Sabetzadeh, M., Briand, L., Zimmer, F.: Automated checking of conformance to requirements templates using natural language processing. TSE 41(10), 944–968 (2015). Google Scholar
  5. Avramidis, E.: Rankeval: open tool for evaluation of machine-learned ranking. Prague Bull. Math. Linguist. 100, 63–72 (2013). CrossRefGoogle Scholar
  6. Berry, D.M., Kamsties, E.: Ambiguity in requirements specification. In: do Prado Leite, J.C.S., Doorn, J.H. (eds) Perspectives on Software Requirements. The Springer International Series in Engineering and Computer Science, vol. 753, pp. 7–44. Springer, Boston, MA (2004).
  7. Berry, D.M., Kamsties, E.: The syntactically dangerous all and plural in specifications. IEEE Softw. 22(1), 55–57 (2005). CrossRefGoogle Scholar
  8. Berry, D.M., Kamsties, E., Krieger, M.M.: From contract drafting to software specification: linguistic sources of ambiguity. (2003). Accessed 10 June 2019
  9. Blei, D.M.: Probabilistic topic models. Commun. ACM 55(4), 77–84 (2012). CrossRefGoogle Scholar
  10. Camacho-Collados, J., Pilehvar, T.: From word to sense embeddings: a survey on vector representations of meaning. arXiv preprint arXiv:1805.04032 (2018)
  11. Casamayor, A., Godoy, D., Campo, M.: Functional grouping of natural language requirements for assistance in architectural software design. KBS 30, 78–86 (2012). Google Scholar
  12. Chantree, F., Nuseibeh, B., De Roeck, A.N., Willis, A.: Identifying nocuous ambiguities in natural language requirements. In: RE’06, pp. 56–65 (2006).
  13. Chen, X., Liu, Z., Sun, M.: A unified model for word sense representation and disambiguation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1025–1035 (2014).
  14. Cleland-Huang, J.: Mining domain knowledge [requirements]. IEEE Softw. 32(3), 16–19 (2015). CrossRefGoogle Scholar
  15. Nattoch Dag, J., Gervasi, V., Brinkkemper, S., et al.: A linguistic-engineering approach to large-scale requirements management. IEEE Softw. 1, 32–39 (2005). CrossRefGoogle Scholar
  16. Dalpiaz, F., van der Schalk, I., Lucassen, G.: Pinpointing ambiguity and incompleteness in requirements engineering via information visualization and NLP. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, pp. 119–135 (2018).
  17. Dieste, O., Juristo, N.: Systematic review and aggregation of empirical studies on elicitation techniques. IEEE Trans. Softw. Eng. 37(2), 283–304 (2011). CrossRefGoogle Scholar
  18. Evans, M.C., Bhatia, J., Wadkar, S., Breaux, T.D.: An evaluation of constituency-based hyponymy extraction from privacy policies. In: Requirements Engineering Conference (RE), 2017 IEEE 25th International. IEEE, pp. 312–321 (2017).
  19. Falessi, D., Cantone, G., Canfora, G.: Empirical principles and an industrial case study in retrieving equivalent requirements via natural language processing techniques. IEEE Trans. Softw. Eng. 39(1), 18–44 (2013). CrossRefGoogle Scholar
  20. Femmer, H., Kučera, J., Vetrò, A.: On the impact of passive voice requirements on domain modelling. In: ESEM. ACM, p. 21 (2014).
  21. Femmer, H., Fernández, D.M., Wagner, S., Eder, S.: Rapid quality assurance with requirements smells. JSS 123, 190–213 (2017). Google Scholar
  22. Fernández, D.M., Wagner, S., Kalinowski, M., Felderer, M., Mafra, P., Vetrò, A., Conte, T., Christiansson, M.T., Greer, D., Lassenius, C., Männistö, T., Nayabi, M., Oivo, M., Penzenstadler, B., Pfahl, D., Prikladnicki, R., Ruhe, G., Schekelmann, A., Sen, S., Spinola, R., Tuzcu, A., de la Vara, J.L., Wieringa, R.: Naming the pain in requirements engineering. Empir. Softw. Eng. 22(5), 2298–2338 (2017). CrossRefGoogle Scholar
  23. Ferrari, A., Gnesi, S.: Using collective intelligence to detect pragmatic ambiguities. In: RE’12. IEEE, pp. 191–200 (2012).
  24. Ferrari, A., Spoletini, P., Gnesi, S.: Ambiguity and tacit knowledge in requirements elicitation interviews. REJ 21(3), 333–355 (2016). Google Scholar
  25. Ferrari, A., Dell’Orletta, F., Esuli, A., Gervasi, V., Gnesi, S.: Natural language requirements processing: a 4D vision. IEEE Softw. 6, 28–35 (2017a). CrossRefGoogle Scholar
  26. Ferrari, A., Donati, B., Gnesi, S.: Detecting domain-specific ambiguities: an NLP approach based on Wikipedia crawling and word embeddings. In: 2017 IEEE 25th International Requirements Engineering Conference Workshops (REW). IEEE, pp. 393–399 (2017b).
  27. Ferrari, A., Esuli, A., Gnesi, S.: Identification of cross-domain ambiguity with language models. In: Groen, E.C., Harrison, R., Murukannaiah, P.K., Vogelsang, A. (eds) 5th International Workshop on Artificial Intelligence for Requirements Engineering, AIRE@RE 2018, Banff, AB, Canada, 21 Aug 2018. IEEE, pp. 31–38 (2018a).
  28. Ferrari, A., Gori, G., Rosadini, B., Trotta, I., Bacherini, S., Fantechi, A., Gnesi, S.: Detecting requirements defects with NLP patterns: an industrial experience in the railway domain. Empir. Softw. Eng. 23(6), 3684–3733 (2018b). CrossRefGoogle Scholar
  29. Firth, J.R.: Selected Papers of JR Firth, 1952–59. Indiana University Press, Bloomington (1968)Google Scholar
  30. Flekova, L., Gurevych, I.: Supersense embeddings: a unified model for supersense interpretation, prediction, and utilization. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), vol. 1, pp. 2029–2041 (2016).
  31. Gacitua, R., Sawyer, P., Gervasi, V.: On the effectiveness of abstraction identification in requirements engineering. In: Proceedings of the 18th IEEE International Requirements Engineering Conference (RE’10). IEEE, pp. 5–14 (2010).
  32. Gleich, B., Creighton, O., Kof, L.: Ambiguity detection: Towards a tool explaining ambiguity sources. In: REFSQ’10, vol. 6182. Springer, LNCS, pp. 218–232 (2010).
  33. Guo, J., Cheng, J., Cleland-Huang, J.: Semantically enhanced software traceability using deep learning techniques. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, pp. 3–14 (2017).
  34. Harris, Z.S.: Distributional structure. Word 10(2–3), 146–162 (1954). Google Scholar
  35. Hoffman, R.R., Shadbolt, N.R., Burton, A.M., Klein, G.: Eliciting knowledge from experts: a methodological analysis. Organ. Behav. Hum. Decis. Process. 62(2), 129–158 (1995). CrossRefGoogle Scholar
  36. Honnibal, M., Montani, I.: spaCy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing (2017) (to appear)Google Scholar
  37. Jha, N., Mahmoud, A.: Mining user requirements from application store reviews using frame semantics. In: International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, pp. 273–287 (2017).
  38. Kendall, M.G.: The treatment of ties in ranking problems. Biometrika 33(3), 239–251 (1945). MathSciNetCrossRefzbMATHGoogle Scholar
  39. Kof, L.: From requirements documents to system models: a tool for interactive semi-automatic translation. In: 2010 18th IEEE International Requirements Engineering Conference, pp. 391–392 (2010).
  40. Krisch, J., Houdek, F.: The myth of bad passive voice and weak words an empirical investigation in the automotive industry. In: 2015 IEEE 23rd International Requirements Engineering Conference (RE). IEEE, pp. 344–351 (2015).
  41. Lami, G., Gnesi, S., Fabbrini, F., Fusani, M.: The linguistic approach to the natural language requirements quality: Benefit of the use of an automatic tool. In: Proceedings 26th Annual NASA Goddard Software Engineering Workshop(SEW), vol. 00, p. 97 (2001).
  42. Landis, J.R., Koch, G.G.: An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 33(2), 363–374 (1977). CrossRefzbMATHGoogle Scholar
  43. Lee, Y.K., Ng, H.T.: An empirical evaluation of knowledge sources and learning algorithms for word sense disambiguation. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing-Volume 10. Association for Computational Linguistics, pp. 41–48 (2002).
  44. Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Advances in Neural Information Processing Systems, pp. 2177–2185. (2014). Accessed 10 June 2019
  45. Lian, X., Rahimi, M., Cleland-Huang, J., Zhang, L., Ferrai, R., Smith, M.: Mining requirements knowledge from collections of domain documents. In: 2016 IEEE 24th International Requirements Engineering Conference (RE). IEEE, pp. 156–165 (2016).
  46. Maalej, W., Nabil, H.: Bug report, feature request, or simply praise? on automatically classifying app reviews. In: Proceedings of the 23rd IEEE International Requirements Engineering Conference, (RE’15). IEEE, pp. 116–125 (2015).
  47. Manning, C.D.: Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In: International Conference on Intelligent Text Processing and Computational Linguistics. Springer, pp. 171–189 (2011)Google Scholar
  48. Massey, A.K., Rutledge, R.L., Anton, A.I., Swire, P.P.: Identifying and classifying ambiguity for regulatory requirements. In: RE’14. IEEE, pp. 83–92 (2014).
  49. Mavin, A., Wilkinson, P., Harwood, A., Novak, M.: Easy approach to requirements syntax (EARS). In: RE’09. IEEE, pp. 317–322 (2009).
  50. Mich, L.: NL-OOPS: from natural language to object oriented requirements using the natural language processing system LOLITA. NLE 2(2), 161–187 (1996). Google Scholar
  51. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013). arXiv:1310.4546
  52. Miller, G.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  53. Navigli, R.: Word sense disambiguation: A survey. ACM Comput. Surv. (CSUR) 41(2), 10 (2009). CrossRefGoogle Scholar
  54. Pedersen, T.: A simple approach to building ensembles of naive bayesian classifiers for word sense disambiguation. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 63–69. Association for Computational Linguistics. (2000). Accessed 10 June 2019
  55. Pedersen, T.: Unsupervised corpus-based methods for WSD. In: Agirre, E., Edmonds, P. (eds) Word Sense Disambiguation. Text, Speech and Language Technology, vol. 33, pp. 133–166. Springer, Dordrecht (2007).
  56. Pohl, K., Rupp, C.: Requirements Engineering Fundamentals. Rocky Nook Inc, San Rafael (2011)Google Scholar
  57. Quirchmayr, T., Paech, B., Kohl, R., Karey, H.: Semi-automatic software feature-relevant information extraction from natural language user manuals. In: Proceedings of the 23rd International Working Conference on Requirements Engineering: Foundation for Software Quality (REFSQ’17), pp. 255–272, Springer (2017).
  58. Raganato, A., Bovi, C.D., Navigli, R.: Neural sequence learning models for word sense disambiguation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1156–1167 (2017).
  59. Robeer, M., Lucassen, G., van der Werf, J.M.E., Dalpiaz, F., Brinkkemper, S.: Automated extraction of conceptual models from user stories via nlp. In: Proceedings of the 24th IEEE International Requirements Engineering Conference (RE’16), pp. 196–205. IEEE (2016).
  60. Rodriguez, D.V., Carver, D.L., Mahmoud, A.: An efficient wikipedia-based approach for better understanding of natural language text related to user requirements. In: 2018 IEEE Aerospace Conference, pp. 1–16. IEEE (2018)
  61. Rosadini, B., Ferrari, A., Gori, G., Fantechi, A., Gnesi, S., Trotta, I., Bacherini, S.: Using NLP to detect requirements defects: an industrial experience in the railway domain. In: REFSQ, pp. 344–360. Springer (2017)
  62. Sleimi, A., Sannier, N., Sabetzadeh, M., Briand, L., Dann, J.: Automated extraction of semantic legal metadata using natural language processing. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 124–135. IEEE (2018).
  63. Sultanov, H., Hayes, J.H.: Application of reinforcement learning to requirements engineering: requirements tracing. In: Proceedings of the 21st IEEE International Requirements Engineering Conference (RE’13), pp. 52–61. IEEE (2013).
  64. Taghipour, K., Ng, H.T.: Semi-supervised word sense disambiguation using word embeddings in general and specific domains. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 314–323 (2015)
  65. Tjong, S., Berry, D.: The design of SREE a prototype potential ambiguity finder for requirements specifications and lessons learned. In: REFSQ’13, LNCS, vol. 7830, pp. 80–95 (2013).
  66. Wang, W., Niu, N., Liu, H., Niu, Z.: Enhancing automated requirements traceability by resolving polysemy. In: 2018 IEEE 26th International Requirements Engineering Conference (RE), pp. 40–51. IEEE (2018).
  67. Yang, H., De Roeck, A.N., Gervasi, V., Willis, A., Nuseibeh, B.: Analysing anaphoric ambiguity in natural language requirements. Requir. Eng. 16(3), 163–189 (2011). CrossRefGoogle Scholar
  68. Yuan, D., Richardson, J., Doherty, R., Evans, C., Altendorf, E.: Semi-supervised word sense disambiguation with neural models. arXiv preprint arXiv:1603.07012 (2016)
  69. Yue, T., Briand, L.C., Labiche, Y.: aToucan: an automated framework to derive UML analysis models from use case models. ACM Trans. Softw. Eng. Methodol. (TOSEM) 24(3), 13 (2015). CrossRefGoogle Scholar
  70. Zowghi, D., Coulin, C.: Requirements elicitation: a survey of techniques, approaches, and tools. In: Engineering and Managing Software Requirements, pp. 19–46. Springer (2005).

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.CNR-ISTIPisaItaly

Personalised recommendations