Abstract
In recent years a huge amount of effort, and money has been invested in the area of semantic service discovery and presented approaches have become more sophisticated and mature. Nevertheless surprisingly little effort is being put into the evaluation of these approaches. We argue that the lack of established and theoretically well-founded methodologies and test beds for comparative evaluation of semantic service discovery is a major blocker of the advancement of the field. To lay the ground for a comprehensive treatment of this problem we discuss the applicability of well-known evaluation methodologies from information retrieval and provide an exhaustive survey of the current evaluation approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cécile Åberg. An Evaluation Platform for Semantic Web Technology. PhD thesis, Department of Computer and Information Science, Linköpings Universitet Sweden, 2007.
Anonymous. RW2 project deliverable D2.3: prototype implementation of the discovery component, 2006.
M. Brian Blake, William Cheung, Michael C. Jaeger, and Andreas Wombacher. WSC-06: the web service challenge. In Proceedings of the Eighth IEEE International Conference on E-Commerce Technology (CEC 2006), and Third IEEE International Conference on Enterprise Computing, E-Commerce and E-Services (EEE 2006), Palo Alto, California, USA, June 2006.
Thomas Fischer. Entwicklung einer Evaluationsmethodik für Semantic Web Services und Anwendung auf die DIANE Service Descriptions (in German). Master’s thesis, IPD, University Karlsruhe, August. 2005.
Donna Harman. Overview of the first Text REtrieval Conference (TREC-1). In Proceedings of TREC-1, Gaithersbury, Maryland, USA, November 1992.
Mahboob Alam Khalid, Benedikt Fries, and Patrick Kapahnke. OWLS-TC-OWL-S service retrieval test collection version 2.1 user manual, October 2006.
Matthias Klusch, Benedikt Fries, and Katia Sycara. Automated semantic web service discovery with OWLS-MX. In Proceedings of the 5th Intern. Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2006), Hakodate, Japan May 2006.
Sheila A. McIlraith, Tran Cao Son, and Honglei Zeng. Semantic web services. IEEE Intelligent Systems, 16(2):46–53, 2001.
Charles Petrie, Tiziana Margaria, Ulrich Küster, Holger Lausen, and Michal Zaremba, SWS Challenge: status, perspectives and lessons learned so far. In Proceedings of the 9th International Conference on Enterprise Information Systems (ICEIS2007), Special Session on Comparative Evaluation of Semantic Web Service Frameworks, Funchal, Madeira-Portugal, June 2007.
Tefko Saracevic. Evaluation of evaluation in information retrieval. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR95), pages 138–146, Seattle, Washington, USA, July 1995.
Adina Sirbu. DIP deliverable D4.14: discovery module prototype, June 2006.
Adina Sirbu, Ioan Toma, and Dumitru Roman. A logic based approach for service discovery with composition support. In Proceedings of the ECOWS06 Workshop on Emerging Web Services Technology, Zürich, Switzerland, December 2006.
Walter F. Tichy. Should computer scientists experiment more? IEEE Computer, 31(5):32–40, May 1998.
Walter F. Tichy, Paul Lukowicz, Lutz Prechelt, and Ernst A. Heinz. Experimental evaluation in computer science: a quantitative study. Journal of Systems and Software, 28(1), 1995.
Ioan Toma, Kashif Iqbal, Dumitru Roman, Thomas Strang, Dieter Fensel, Brahmananda Sapkota, Matthew Moran, and Juan Miguel Gomez. Discovery in grid and web services environments: A survey and evaluation. International Journal on Multiagent and Grid Systems, 3(3), 2007.
Vassileios Tsetsos, Christos Anagnostopoulos, and Stathes Hadjiefthymiades. On the evaluation of semantic web service matchmaking systems. In Proceedings of the 4th IEEE European Conference on Web Services (ECOWS2006) Zürich, Switzerland, December 2006.
Ellen M. Voorhees. The philosophy of information retrieval evaluation. In Evaluation of Cross-Language Information Retrieval Systems, Second Workshop of the Cross-Language Evaluation Forum (CLEF 2001) pages 355–370, Darmstadt, Germany, September 2001.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Birkhäuser Verlag, Basel/Switzerland
About this chapter
Cite this chapter
Küster, U., Lausen, H., König-Ries, B. (2008). Evaluation of Semantic Service Discovery—A Survey and Directions for Future Research. In: Gschwind, T., Pautasso, C. (eds) Emerging Web Services Technology, Volume II. Whitestein Series in Software Agent Technologies and Autonomic Computing. Birkhäuser Basel. https://doi.org/10.1007/978-3-7643-8864-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-7643-8864-5_4
Publisher Name: Birkhäuser Basel
Print ISBN: 978-3-7643-8863-8
Online ISBN: 978-3-7643-8864-5
eBook Packages: Computer ScienceComputer Science (R0)