On Modeling Risk in Markov Decision Processes

Abstract

Markov decision processes are solved recursively, using the Bellman optimality principle,

$$V(s,t): = \mathop {\max }\limits_{a \in A(s)} \left\{ {r(s,a) + \alpha \sum\limits_{j \in S} {{p_{s,j}}(a)} V(j,t + 1)} \right\}$$

(A)

where V(s, t) is the optimal value of state s at stage t, r(s, a) is the instantaneous profit from action a at state s, S is the state space, A(s) the set of feasible actions at state s and p _i,j (a) the transition probabilities from i to j. This solution maximizes the expected value of the discounted sum of future profits (the right side of (A)), and assumes risk neutrality, i.e. the decision maker is indifferent between a random variable and its expected value.

We propose an alternative solution, with explicit modeling of risk, using the recursion

$$V(s,t): = \mathop {\max }\limits_{a \in A(s)} \left\{ {r(s,a) + \alpha {S_\beta }(V(Z(s,a),t + 1))} \right\} $$

(B)

where Z(s, a) is the next state, S _β is the quadratic certainty equivalent

$${S_\beta }(X): = EX - \frac{\beta }{2}VarX$$

and β is a parameter modeling the attitude of the decision maker towards risk: β > 0 if risk-averse, β < 0 if risk seeking and β = 0 if risk-neutral (in which case (B) reduces to (A)).

We apply our model to solve two problems of maintenance and inventory and compare with the classical solution.