On representation of source reliability in weight of evidence

Abstract

Developers of artificial intelligence-based systems have made frequent use of likelihood ratios. Those ratios have been used to represent the uncertainty associated with events and hypotheses on rules in expert systems and they have been used to establish rankings of resulting diagnoses in other systems. This paper discusses the representation of source reliability through those likelihood ratios, for use in artificial intelligence systems, such as expert systems, influence diagrams and other systems that employ a Bayesian-based approach to the representation of uncertainty to assess the weight of evidence.

This paper presents a means by which that reliability can be captured using a likelihood ratio format. Reliability can have a substantial impact on the value of the likelihood ratio. One example presented in the paper results in about a 60% decrease in the value of the likelihood ratio, with only a 10% decrease in reliability. Then this paper investigates the impact of accounting for reliability in the likelihood ratios. A monotonic property is established for the reliability embedded likelihood ratios. That property provides insight both into the behavior of weights on rules in systems that employ such an approach and into the use of likelihood ratios to rank diagnoses.

Then the impact of reliability-adjusted likelihood ratios is examined for their impact on rank ordering of the ratio. It is found that in some cases where likelihood ratios are examined in terms of the same evidence that reliability does not change the rankings. However, if the likelihood ratios are developed for comparison across different evidence, then the rankings do not remain the same. As a result, accounting for reliability of evidence can be critical to the ultimate success of systems employing this approach.