Abstract
Over the last decade, several ontology reasoners have been proposed to overcome the computational complexity of inference tasks on expressive ontology languages such as OWL 2 DL. Nevertheless, it is well-accepted that there is no outstanding reasoner that can outperform in all input ontologies. Thus, deciding the most suitable reasoner for an ontology based application is still a time and effort consuming task. In this paper, we suggest to develop a new system to provide user support when looking for guidance over ontology reasoners. At first, we will be looking at automatically predict a single reasoner empirical performances, in particular its robustness and efficiency, over any given ontology. Later, we aim at ranking a set of candidate reasoners in a most preferred order by taking into account information regarding their predicted performances. We conducted extensive experiments covering over 2500 well selected real-world ontologies and six state-of-the-art of the most performing reasoners. Our primary prediction and ranking results are encouraging and witnessing the potential benefits of our approach.
This work is an extension of our previous publication at the 7th International Conference on Knowledge Engineering and Ontology Development KEOD 2015 [2].
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
It’s important to notice that “DL classification” stands for a reasoning task aiming at inferring the subsumption relations between the classes and the properties in given ontology, however the “ML classification” denotes the process of training a predictive model from a labelled dataset.
- 2.
It’s important to notice that “DL classification” designs a reasoning task aiming at inferring the class and property subsumption relation in a given ontology, however the “ML classification” denotes the process of training a predictive model from a labeled dataset.
- 3.
For further details about OWL 2 profiles, the reader is kindly referred to http://www.w3.org/TR/owl2-profiles/.
- 4.
The ORE Framework is available at https://github.com/andreas-steigmiller/ore-2014-competition-framework/.
- 5.
The ORE corpus is available at http://zenodo.org/record/10791.
- 6.
The size corresponds to the number of the logical axioms.
- 7.
All ORE’2014 reasoners are available at https://zenodo.org/record/11145/.
- 8.
Results of our experiments are available at https://github.com/PhdStudent2015/Classification_Results_2016.
- 9.
The Weka API is available at www.cs.waikato.ac.nz/ml/weka/downloading.html.
- 10.
RLF will be used with a ranking threshold equal to 0.01.
- 11.
Accuracy places more weight on the majority class(es) than the minority one(s). Consequently, high accuracy rates would be reported, even if the predictive model is not necessarily a good one.
- 12.
The alternatives are the objects, or elements to be ranked.
- 13.
However, the reasoner results correctness may be approved by a different checking method.
- 14.
We recall that each of the buckets \(\mathcal {B}_C\prec \mathcal {B}_U \prec \mathcal {B}_T \prec \mathcal {B}_H\) corresponds to a predictable label \(\{C, U, T, H\}\) computed by the robustness classification model.
- 15.
It is to be noted that when \(K=1\), P@1 is equal to AP@1.
References
Abbott, D.: Applied Predictive Analytics: Principles and Techniques for the Professional Data Analyst, 1st edn. Wiley, Hoboken (2014)
Alaya, N., Ben Yahia, S., Lamolle, M.: Predicting the empirical robustness of the ontology reasoners based on machine learning techniques. In: Proceedings of the 7th International Conference on Knowledge Engineering and Ontology Development KEOD 2015, Lisbon, Portugal, pp. 61–73 (2015)
Alaya, N., Lamolle, M., Ben Yahia, S.: Towards unveiling the ontology key features altering reasoner performances. Technical report, IUT of Montreuil, France (2015). http://arxiv.org/abs/1509.08717
Baader, F., Calvanese, D., McGuinness, D.L., Nardi, D., Patel-Schneider, P.F.: The Description Logic Handbook: Theory Implementation and Applications, 2nd edn. Cambridge University Press, New York (2010)
Baeza-Yates, R.A., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley Longman Publishing Co., Inc., Boston (1999)
Caruso, J.C., Cliff, N.: Empirical size, coverage, and power of confidence intervals for Spearman’s rho. J. Educ. Psychol. Meas. 57, 637–654 (1997)
Chandrashekar, G., Sahin, F.: A survey on feature selection methods. Comput. Electr. Eng. 40(1), 16–28 (2014)
Dumontier, M., Glimm, B., Gonçalves, R.S., Horridge, M., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Stamou, G.B., Stoilos, G. (eds.): Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation (ORE-2015), Athens, Greece, vol. 1387. CEUR-WS.org (2015)
Emond, E.J., Mason, D.: A new rank correlation coefficient with application to consensus ranking problem. J. Multicriteria Decis. Anal. 11, 17–28 (2002)
Fagin, R., Kumar, R., Mahdian, M., Sivakumar, D., Vee, E.: Comparing partial rankings. SIAM J. Discret. Math. 20, 628–648 (2006)
Garcia, S., Luengo, J., Saez, J., Lopez, V., Herrera, F.: A survey of discretization techniques: taxonomy and empirical analysis in supervised learning. IEEE Trans. Knowl. Data Eng. 25, 734–750 (2013)
Gardiner, T., Tsarkov, D., Horrocks, I.: Framework for an automated comparison of description logic reasoners. In: Cruz, I., Decker, S., Allemang, D., Preist, C., Schwabe, D., Mika, P., Uschold, M., Aroyo, L.M. (eds.) ISWC 2006. LNCS, vol. 4273, pp. 654–667. Springer, Heidelberg (2006). doi:10.1007/11926078_47
Gomes, C.P., Selman, B.: Algorithm portfolios. Artif. Intell. 126, 43–62 (2001)
Gonçalves, R.S., Matentzoglu, N., Parsia, B., Sattler, U.: The empirical robustness of description logic classification. In: Informal Proceedings of the 26th International Workshop on Description Logics, Ulm, Germany, pp. 197–208 (2013)
Gonçalves, R.S., Bail, S., Jiménez-Ruiz, E., Matentzoglu, N., Parsia, B., Glimm, B., Kazakov, Y.: Owl reasoner evaluation (ORE) workshop 2013 results: short report. In: ORE, pp. 1–18 (2013)
Gonçalves, R.S., Parsia, B., Sattler, U.: Performance heterogeneity and approximate reasoning in description logic ontologies. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 82–98. Springer, Heidelberg (2012). doi:10.1007/978-3-642-35176-1_6
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P.: The weka data mining software: an update. SIGKDD Explor. Newsl. 11, 10–18 (2009)
Hu, B., Dong, W.: A study on cost behaviors of binary classification measures in class-imbalanced problems. CoRR abs/1403.7100 (2014)
Kang, Y.-B., Krishnaswamy, S., Li, Y.-F.: R\(_2\)O\(_2\): an efficient ranking-based reasoner for OWL ontologies. In: Arenas, M., et al. (eds.) ISWC 2015. LNCS, vol. 9366, pp. 322–338. Springer, Heidelberg (2015). doi:10.1007/978-3-319-25007-6_19
Kang, Y.-B., Li, Y.-F., Krishnaswamy, S.: Predicting reasoning performance using ontology metrics. In: Cudré-Mauroux, P., et al. (eds.) ISWC 2012. LNCS, vol. 7649, pp. 198–214. Springer, Heidelberg (2012). doi:10.1007/978-3-642-35176-1_13
Kang, Y.B., Li, Y.F., Krishnaswamy, S.: How long will it take? Accurate prediction of ontology reasoning performance. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence, pp. 80–86 (2014)
Lee, M., Matentzoglu, N., Sattler, U., Parsia, B.: Verifying reasoner correctness - a justication based method. In: Informal Proceedings of the 4th International Workshop on OWL Reasoner Evaluation (ORE-2015), pp. 46–52 (2015)
LePendu, P., Noy, N.F., Jonquet, C., Alexander, P.R., Shah, N.H., Musen, M.A.: Optimize first, buy later: analyzing metrics to ramp-up very large knowledge bases. In: Patel-Schneider, P.F., Pan, Y., Hitzler, P., Mika, P., Zhang, L., Pan, J.Z., Horrocks, I., Glimm, B. (eds.) ISWC 2010. LNCS, vol. 6496, pp. 486–501. Springer, Heidelberg (2010). doi:10.1007/978-3-642-17746-0_31
Matentzoglu, N., Leo, J., Hudhra, V., Sattler, U., Parsia, B.: A survey of current, stand-alone OWL reasoners. In: Proceedings of the 4th International Workshop on OWL Reasoner Evaluation, pp. 68–79 (2015)
Mikolà\({\tilde{\rm s}}\)ek, V.: Dependability and robustness: state of the art and challenges. In: Proceedings of the First International Workshop on Software Technologies for Future Dependable Distributed Systems (STFSSD), pp. 25–31 (2009)
W3C OWL Working Group: OWL 2 Web Ontology Language: Document Overview. W3C Recommendation, October 2009. http://www.w3.org/TR/owl2-overview/
Pandit, V., Kenkre, S., Khan, A.: On discovering bucket orders from preference data. In: Proceedings of the Eleventh SIAM International Conference on Data Mining, Arizona, USA, pp. 872–883 (2011)
Prudêncio, R.B.C., de Souto, M.C.P., Ludermir, T.B.: Selecting machine learning algorithms using the ranking meta-learning approach. In: Jankowski, N., Duch, W., Gra̧bczewski, K. (eds.) Meta-Learning in Computational Intelligence. SCI, vol. 358, pp. 225–243. Springer, Heidelberg (2011)
Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)
Rosen, K.H.: Discrete Mathematics and Its Applications. McGraw Hill Higher Education, New York (1991)
Sazonau, V., Sattler, U., Brown, G.: Predicting performance of OWL reasoners: locally or globally? In: Proceedings of the Fourteenth International Conference on Principles of Knowledge Representation and Reasoning (2014)
Weithöner, T., Liebig, T., Luther, M., Böhm, S., Henke, F., Noppens, O.: Real-world reasoning with OWL. In: Franconi, E., Kifer, M., May, W. (eds.) ESWC 2007. LNCS, vol. 4519, pp. 296–310. Springer, Heidelberg (2007). doi:10.1007/978-3-540-72667-8_22
Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Satzilla: portfolio-based algorithm selection for SAT. J. Artif. Int. Res. 32, 565–606 (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Alaya, N., Yahia, S.B., Lamolle, M. (2016). Ranking with Ties of OWL Ontology Reasoners Based on Learned Performances. In: Fred, A., Dietz, J., Aveiro, D., Liu, K., Filipe, J. (eds) Knowledge Discovery, Knowledge Engineering and Knowledge Management. IC3K 2015. Communications in Computer and Information Science, vol 631. Springer, Cham. https://doi.org/10.1007/978-3-319-52758-1_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-52758-1_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-52757-4
Online ISBN: 978-3-319-52758-1
eBook Packages: Computer ScienceComputer Science (R0)