The software engineering field is continuously making an effort to improve the effectiveness of the software development process. This improvement is performed by developing quantitative measures that can be used to enhance the quality of software products and to more accurately describe, better understand and manage the software development life cycle. Even if the ontology engineering field is constantly adopting practices from software engineering, it has not yet reached a state in which metrics are an integral part of ontology engineering processes and support making evidence-based decisions over the process and its outputs. Up to now, ontology metrics are mainly focused on the ontology implementation and do not take into account the development process or other artefacts that can help assessing the quality of the ontology, e.g. its requirements. This work envisions the need for a metrics-driven ontology engineering process and, as a first step, presents a set of metrics for ontology engineering which are obtained from artefacts generated during the ontology development process and from the process itself. The approach is validated by measuring the ontology engineering process carried out in a research project and by showing how the proposed metrics can be used to improve the efficiency of the process by making predictions, such as the effort needed to implement an ontology, or assessments, such as the coverage of the ontology according to its requirements.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
The authors of the analysed paper refer to the knowledge base as the set of TBox and Abox.
The link to all the Github repositories are indicated in the VICINITY ontology portal: http://vicinity.iot.linkeddata.es/ which due to its version control allows the ontology engineers to be aware of the evolution of the artefacts during the development iterations. Test suites are also stored in the GitHub repository of its associated ontology.
Albrecht AJ (1979) Measuring application development productivity. In: Proceedings of the joint SHARE/GUIDE/IBM application development symposium, pp 83–92
Baader F, Horrocks I, Sattler U (2008) Description logics. Found Artif Intell 3:135–179
Blomqvist E, Sepour AS, Presutti V (2012) Ontology testing-methodology and tool. In: Proceedings of the 18th international conference on knowledge engineering and knowledge management, Galway City, Ireland, October 8–12. Springer, Berlin, pp 216–226
Boehm B, Clark B, Horowitz E, Westland C, Madachy R, Selby R (1995) Cost models for future software life cycle processes: COCOMO 2.0. Ann Softw Eng 1(1):57–94
Brickley D, Guha RV, McBride B (2014) RDF Schema 1.1. W3C Recommendation 25 February 2014. Available at https://www.w3.org/TR/rdf-schema/
Costello RJ, Liu DB (1995) Metrics for requirements engineering. J Syst Softw 29(1):39–63
Davis A, Overmyer S, Jordan K, Caruso J, Dandashi F, Dinh A, Kincaid G, Ledeboer G, Reynolds P, Sitaram P, Ta A, Theofanos M (1993) Identifying and measuring quality in a software requirements specification. In: Proceedings of the 1st international software metrics symposium, Baltimore, MD, USA, May 21–22. IEEE, pp 141–152
De Leenheer P, Debruyne C (2008) DOGMA-MESS: a tool for fact-oriented collaborative ontology evolution. In: Proceedings of the 2008 international conference on On the move to meaningful internet systems: OTM 2008 Workshops, Monterrey, Mexico, November 9–14. Springer, Berlin, pp 797–806
De Moor A, De Leenheer P, Meersman R (2006) DOGMA-MESS: a meaning evolution support system for interorganizational ontology engineering. In: Proceedings of the 14th international conference on conceptual structures, Aalborg, Denmark, July 16–21. Springer, Berlin, pp 189–202
De Nicola A, Missikoff M, Navigli R (2005) A proposal for a unified process for ontology building: Upon. In: Proceedings of the 16th international conference on database and expert systems applications, Copenhagen, Denmark, August 22–26. Springer, Berlin, pp 655–664
Debruyne C, Tran TK, Meersman R (2013) Grounding ontologies with social processes and natural language. J Data Semant 2(2–3):89–118
DeMarco T (1979) Structured analysis and system specification. Yourdon Press, Berlin
Duque-Ramos A, Fernández-Breis JT, Iniesta M, Dumontier M, Aranguren ME, Schulz S, Aussenac-Gilles N, Stevens R (2013) Evaluation of the OQuaRE framework for ontology quality. Expert Syst Appl 40(7):2696–2703
Fenton N, Bieman J (1997) Software metrics: a rigorous and practical approach. PWS Publishing Company, Boston
Fenton NE, Neil M (2000) Software metrics: roadmap. In: Proceedings of the conference on the future of software engineering, Limerick, Ireland, June 04–11. ACM, pp 357–370
Fernández-López M, Gómez-Pérez A (2002) The integration of OntoClean in WebODE. In: Proceedings of the OntoWeb-SIG3 workshop at the 13th international conference on knowledge engineering and knowledge management, Siguenza, Spain, 30th September. CEUR-WS.org, CEUR Workshop Proceedings, vol 62, pp 38–52
Fernández-López M, Gómez-Pérez A, Juristo N (1997) Methontology: from ontological art towards ontological engineering. In: Proceedings of the ontological engineering AAAI97 spring symposium series. Stanford University, EEUU, March 24–26. AAAI Press, pp 33–40
Gangemi A, Presutti V (2009) Ontology design patterns. Handbook on ontologies. Springer, Berlin, pp 221–243
Gangemi A, Catenacci C, Ciaramita M, Lehmann J (2006) Modelling ontology evaluation and validation. In: Proceedings of the 3rd European Semantic Web Conference, Budva, Montenegro, June 11–14. Springer, Berlin, pp 140–154
García-Ramos S, Otero A, Fernández-López M (2009) OntologyTest: A tool to evaluate ontologies through tests defined by the user. In: Proceedings of the 10th international work-conference on artificial neural networks on artificial neural networks, Salamanca, Spain, June 10–12. Springer, Berlin, pp 91–98
Gunning R (1952) The technique of clear writing. McGraw-Hill, New York
Hitzler P, Krötzsch M, Parsia B, Patel-Schneider PF, Rudolph S (2009) OWL 2 Web Ontology Language Primer (Second Edition) W3C Recommendation 11 December 2012. Available at https://www.w3.org/TR/owl2-primer/
Iqbal S, Naeem M, Khan A (2012) Yet another set of requirement metrics for software projects. Int J Softw Eng Its Appl 6(1):19–28
Kan SH (2002) Metrics and models in software quality engineering. Addison-Wesley, London
Kang YB, Li YF, Krishnaswamy S (2012) Predicting reasoning performance using ontology metrics. In: Proceedings of the 11th international semantic web conference, Boston, MA, USA, November 11–15, Springer, Berlin, pp 198–214
Keet CM, Ławrynowicz A (2016) Test-driven development of ontologies. In: Proceedings of 13th European semantic web conference, Heraklion, Crete, Greece, May 29–June 2. Springer, Berlin, pp 642–657
Kirch W (ed) (2008) Pearson’s correlation coefficient. Springer, Netherlands
Kotis K, Vouros GA, Alonso JP (2004) HCOME: A tool-supported methodology for engineering living ontologies. In: Proceedings of the 2nd international workshop on semantic web and databases, Toronto, Canada, August 29–30. Springer, Berlin, pp 155–166
Lantow B (2016) OntoMetrics: application of on-line ontology metric calculation. In: Joint proceedings of the BIR 2016 workshops and doctoral consortium co-located with 15th international conference on perspectives in business informatics research, Prague, Czech Republic, September 14–16
Ma Y, Jin B, Feng Y (2010) Semantic oriented ontology cohesion metrics for ontology-based systems. J Syst Softw 83(1):143–152
Mall R (2014) Fundamentals of software engineering. PHI Learning Pvt Ltd, Delhi
McCabe TJ (1976) A complexity measure. IEEE Trans Softw Eng 4:308–320
Moser R, Pedrycz W, Succi G (2008) A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction. In: Proceedings of the 30th international conference on software engineering, Leipzig, Germany, May 10–18. ACM, pp 181–190
Muller JZ (2018) The Tyranny of Metrics. Princeton University Press, Princeton
Noy N, Rector A, Hayes P, Welty C (2006) Defining n-ary relations on the semantic web. W3C working group note 12(4). Available at https://www.w3.org/TR/swbp-n-aryRelations/
Orme AM, Tao H, Etzkorn LH (2006) Coupling metrics for ontology-based system. IEEE Softw 23(2):102–108
Peroni S (2016) A simplified agile methodology for ontology development. In: Proceedings of the 13th OWL: experiences and directions workshop and 5th OWL reasoner evaluation workshop, Bologna, Italy, November 20. Springer, Berlin, pp 55–69
Pinto HS, Staab S, Tempich C (2004) DILIGENT: Towards a fine-grained methodology for Distributed, Loosely-controlled and evolvInG. In: Proceedings of the 16th European conference on artificial intelligence, Valencia, Spain, August 22–27, vol 110, p 393
Poveda-Villalón M, Gómez-Pérez A, Suárez-Figueroa MC (2014) OOPS! (OntOlogy Pitfall Scanner!): an on-line tool for ontology evaluation. Int J Seman Web Inform Syst 10(2):7–34
Pressman RS (2005) Software engineering: a practitioner’s approach. Palgrave Macmillan, London
Presutti V, Daga E, Gangemi A, Blomqvist E (2009) eXtreme design with content ontology design patterns. In: Proceedings of the workshop on ontology patterns, collocated with the 8th international semantic web conference, Washington DC, USA, 25 October, CEUR Workshop series, pp 83–97
Rahman F, Devanbu P (2013) How, and why, process metrics are better. In: Proceedings of the 35th international conference on software engineering, San Francisco, USA, May 18–26. IEEE, pp 432–441
Ren Y, Parvizi A, Mellish C, Pan JZ, van Deemter K, Stevens R (2014) Towards competency question-driven ontology authoring. In: Proceedings of the 11th European semantic web conference, Crete, Greece, May 25–29. Springer, Berlin, pp 752–767
Schober D, Tudose I, Svatek V, Boeker M (2012) OntoCheck: verifying ontology naming conventions and metadata completeness in protégé 4. J Biomed Semant 3(S-2):S4
Shatnawi R, Li W (2008) The effectiveness of software metrics in identifying error-prone classes in post-release software evolution process. J Syst Softw 81(11):1868–1882
Smith B, Ashburner M, Rosse C, Bard J, Bug W, Ceusters W, Goldberg LJ, Eilbeck K, Ireland A, Mungall CJ et al (2007) The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol 25(11):1251
Sommerville I (2010) Software engineering, 9th edn. Addison-Wesley, London
Suárez-Figueroa M, Gómez-Pérez A, Villazón-Terrazas B (2009) How to write and use the ontology requirements specification document. In: Proceedings of the international conference on on the move to meaningful internet systems, Ilamoura, Portugal, November 1–6. Springer, Berlin, pp 966–982
Suárez-Figueroa MC, Gómez-Pérez A, Fernández-López M (2012) The NeOn methodology for ontology engineering. In: Ontology engineering in a networked world. Springer, Berlin, pp 9–34
Suárez-Figueroa MC, Aguado de Cea G, Gómez-Pérez A (2013) Lights and shadows in creating a glossary about ontology engineering. Terminology 19(2):202–236
Tartir S, Arpinar IB, Moore M, Sheth AP, Aleman-Meza B (2005) OntoQA: Metric-based ontology quality analysis. In: Proceedings of IEEE workshop on knowledge acquisition from distributed, autonomous, semantically heterogeneous data and knowledge sources at 2005 IEEE international conference on data mining, Houston, USA, November 27, pp 45–53
Uschold M, Gruninger M (1996) Ontologies: Principles, methods and applications. Knowl Eng Rev 11(2):93–136
Vrandečić D, Gangemi A (2006) Unit tests for ontologies. In: Proceedings of the 2006 international conference on the move to meaningful internet systems: OTM 2006 workshops, Montpellier, France, October 29–November 3. Springer, Berlin, pp 1012–1020
Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85
Vrandečić D, Sure Y (2007) How to design better ontology metrics. Springer, Berlin, pp 311–325
Wilsdon J (2016) The metric tide: independent review of the role of metrics in research assessment and management. Sage, Thousand Oaks
Yao H, Orme AM, Etzkorn L (2005) Cohesion metrics for ontology design and application. J Comput Sci 1(1):107–113
Zhe Y, Zhang D, Chuan Y (2006) Evaluation metrics for ontology complexity and evolution analysis. In: Proceedings of the IEEE international conference on e-business engineering, Shanghai, China, October 24–26. IEEE Computer Society, pp 162–170
This work is partially supported by the H2020 project VICINITY: Open virtual neighbourhood network to connect intelligent buildings and smart objects (H2020-688467) and by a Predoctoral grant from the I+D+i program of the Universidad Politécnica de Madrid. We are very grateful to María Navas-Loro for her formula revisions and comments.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Fernández-Izquierdo, A., Poveda-Villalón, M., Gómez-Pérez, A. et al. Towards metrics-driven ontology engineering. Knowl Inf Syst (2021). https://doi.org/10.1007/s10115-021-01545-9
- Ontology engineering
- Ontology development