An Incentive Approach to Eliciting Probabilities

Abstract

A decision maker (e. g., the Nuclear Regulatory Commission) seeks an expert’s probabilities for uncertain quantities of interest (e. g., a seismologist’s forecast of earthquakes), and wants the expert’s reward to depend on the accuracy of the predictions. Assume that the expert compares compensation schemes on the basis of the expected utility of the dollar payoffs, and is willing to reveal his utility function for money. A reward is called “proper” if the expert is never encouraged to state probabilities he does not truly believe. It is “strictly proper” if he is, in fact, encouraged to state his beliefs.

The reward procedure suggested in this paper uses the expert’s stated probabilities and utility function to select from a set of possible payoffs. This procedure is always proper, but may not be strictly proper. If the preferred payoff is independent of the outcome whenever the decision maker and expert agree on the probabilities, then they are said to be “jointly risk-averse.” (For example, if the decision maker agrees to play “bookie” to a risk-averse expert, then they are jointly risk-averse.) In this case, the reward is shown to be strictly proper, as long as they don’t disagree too much, so the expert can gain from researching the problem and carefully assessing his probabilities. In addition, the expert would prefer to make the bet more detailed, distinguishing between finer grain events, whenever such detail exposes new differences of opinion.