Abstract
In previous work, Bogaerts and Leake [1,2] introduced the rank quality measure for the evaluation of conversational case-based reasoning (CCBR) systems. Rank quality assesses how well a system copes with the limited problem information available in an ongoing dialog, giving useful evaluation information not readily available from standard precision and efficiency measures. However, that work also revealed surprising challenges for developing rank quality measures, restricting the proposed measures’ applicability. This paper explores two open questions from that work: 1) how to define a rank quality measure immune to the previous pitfalls, and 2) how to assess the meaningfulness of any proposed rank quality measure. The paper establishes formal requirements for a rank quality measure, presents a new formulation of the measure, and provides a formal proof and empirical evidence to support that the new measure avoids previous pitfalls and meets the formal requirements.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bogaerts, S., Leake, D.: What evaluation criteria are right for CCBR? Considering rank quality. In: Proceedings of the Eighth European Conference on Case-Based Reasoning (2006)
Bogaerts, S.: Rank Quality for Evaluating CCBR System Performance. PhD thesis, Indiana University, Bloomington, IN, USA (2007)
Aha, D., Breslow, L.: Refining conversational case libraries. In: Proceedings of the Second International Conference on Case-Based Reasoning, pp. 267–278. Springer, Berlin (1997)
Kontkanen, P., Myllymäki, P., Silander, T., Tirri, H.: On BAYESIAN case matching. In: Cunningham, P., Smyth, B., Keane, M. (eds.) Proceedings of the Fourth European Workshop on Case-Based Reasoning, pp. 13–24. Springer, Berlin (1998)
Bogaerts, S., Leake, D.: Facilitating CBR for incompletely-described cases: Distance metrics for partial problem descriptions. In: Funk, P., González, P. (eds.) ECCBR 2004. LNCS (LNAI), vol. 3155, pp. 62–76. Springer, Heidelberg (2004)
Kendall, M., Gibbons, J.D.: Rank Correlation Methods, 5th edn. Oxford University Press, New York (1990)
Mosteller, F., Rourke, R.E.K.: Sturdy Statistics: Nonparametrics and Order Statistics. Addison-Wesley Publishing Company, Reading (1973)
Ha, V., Haddawy, P.: Similarity of personal preferences: Theoretical foundations and empirical analysis. Artificial Intelligence 146(2), 149–173 (2003)
Bogaerts, S., Leake, D.: IUCBRF: A framework for rapid and modular CBR system development. Technical Report TR 617, Computer Science Department, Indiana University, Bloomington, IN (2005)
Blake, C., Merz, C.: UCI repository of machine learning databases (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Bogaerts, S., Leake, D. (2008). Formal and Experimental Foundations of a New Rank Quality Measure. In: Althoff, KD., Bergmann, R., Minor, M., Hanft, A. (eds) Advances in Case-Based Reasoning. ECCBR 2008. Lecture Notes in Computer Science(), vol 5239. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-85502-6_5
Download citation
DOI: https://doi.org/10.1007/978-3-540-85502-6_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-85501-9
Online ISBN: 978-3-540-85502-6
eBook Packages: Computer ScienceComputer Science (R0)