Actor-critic multi-objective reinforcement learning for non-linear utility functions

Abstract

We propose a novel multi-objective reinforcement learning algorithm that successfully learns the optimal policy even for non-linear utility functions. Non-linear utility functions pose a challenge for SOTA approaches, both in terms of learning efficiency as well as the solution concept. A key insight is that, by proposing a critic that learns a multi-variate distribution over the returns, which is then combined with accumulated rewards, we can directly optimize on the utility function, even if it is non-linear. This allows us to vastly increase the range of problems that can be solved compared to those which can be handled by single-objective methods or multi-objective methods requiring linear utility functions, yet avoiding the need to learn the full Pareto front. We demonstrate our method on multiple multi-objective benchmarks, and show that it learns effectively where baseline approaches fail.

Appendices

Appendix A: MO policy gradient proofs

MOCAC is inspired by single-objective actor-critic methods, which make use of the Policy Gradient theorem to provide convergence guarantees towards a local optimum via gradient descent. This theorem assumes that the policy maximizes the sum of rewards, which does not hold when using non-linear utility functions. In this section, we follow the original Policy Gradient theorem to provide convergence guarantees for MOCAC.

Given a performance measure $J(\pi _{\theta })$, with $\theta$ the parameters of the policy $\pi$, we would like to use its gradient $\nabla _\theta J(\pi _{\theta })$ to update the policy using gradient ascent:

$$\begin{aligned} \theta _{k+1} = \theta _k + \alpha \nabla _\theta J(\pi _{\theta _k}) \end{aligned}$$

where $\alpha$ is the learning rate.

We now derive this gradient for MOAC, the baseline that does not use a distributional critic in the ablation study, as well as for MOCAC, our main contribution.

1.1 A.1 Proof under SER

In Sect. 4.1 we perform an ablation study to show that using a distributional critic is necessary to optimize under the ESR criterion. Our ablation, which we call MOAC, does not use a distributional critic. MOAC thus optimizes under the SER criterion, as the utility function is applied on the V-value. We prove that MOAC indeed converges under SER. We define the performance measure we aim to maximize as:

$$\begin{aligned} J(\pi _{\theta })&\doteq u({{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \vec {R(\tau )} \right] ) \nonumber \\ &= u(V^{\pi _\theta }(s_0)) \end{aligned}$$

(A1)

where $\tau$ are trajectories sampled by following $\pi$. The gradient of the policy performance, i.e. the policy gradient, is:

$$\begin{aligned} \nabla _\theta J(\pi _{\theta }) &= \nabla _\theta u(V^{\pi _\theta }(s_0)) \nonumber \\ &= \sum _{i=0}^n \frac{\partial u}{\partial V^{\pi _\theta }_i}\frac{\partial V^{\pi _\theta }_i}{\partial \theta } \nonumber \\ &= \sum _{i=0}^n \frac{\partial u}{\partial V^{\pi _\theta }_i}\nabla _\theta V^{\pi _\theta }_i(s_0) \end{aligned}$$

(A2)

with $V^{\pi _\theta }_i$ the V-value for the i-th objective. Since this is a scalar, $\nabla _\theta V^{\pi _\theta }_i(s_0)$ follows the original Policy Gradient proof (in our case with baseline):

$$\begin{aligned} \nabla _\theta V^{\pi _\theta }_i(s_0) = {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \sum _{t=0}^T \nabla _\theta \log \pi _\theta (a_t \mid s_t) (Q^{\pi _\theta }_i(s_t, a_t) - V^{\pi _\theta }_i(s_t)) \right] \end{aligned}$$

(A3)

Intuitively, our MOAC policy gradient is a weighted sum over the individual single-objective policy gradients where each weight defines the importance of its objective, which is measured using the utility function. Concretely, the MOAC policy gradient equals to the sum of the single-objective policy gradients multiplied by the gradient of the utility function when evaluating the utility function for $s_0$. Since, in our setting, u is known, we assume we can compute its derivative:

$$\begin{aligned} \nabla _\theta J(\pi _{\theta }) = \sum _{i=0}^n \frac{\partial u}{\partial V^{\pi _\theta }_i} {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \sum _{t=0}^T \nabla _\theta \log \pi _\theta (a_t \mid s_t) (Q^{\pi _\theta }_i(s_t, a_t) - V^{\pi _\theta }_i(s_t)) \right] \end{aligned}$$

(A4)

We use this gradient to update the policy of MOAC with gradient ascent.

1.2 A.2 Proof under ESR

While MOAC optimizes under SER, we are interested in the ESR setting, which is what MOCAC optimizes under. In this case, the performance measure we aim to maximize is:

$$\begin{aligned} J(\pi _\theta ) \doteq {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ u(\vec {R}(\tau )) \right] \end{aligned}$$

(A5)

To derive its gradient, we use the log-derivative trick:

$$\begin{aligned}{} & {} \frac{\textrm{d}}{\textrm{d}x}\log f(x) = \frac{1}{f(x)}\frac{\textrm{d}}{\textrm{d}x} f(x) \nonumber \\{} & {} \frac{\textrm{d}}{\textrm{d}x} f(x) = f(x)\frac{\textrm{d}}{\textrm{d}x}\log f(x) \end{aligned}$$

(A6)

We derive the gradient $\nabla _\theta J(\pi _\theta )$ of $J(\pi _\theta )$ as follows:

$$\begin{aligned} \nabla _\theta J(\pi _\theta )&= \nabla _\theta {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ u(\vec {R}(\tau )) \right] \nonumber \\ &= \nabla _\theta \int _\tau P(\tau \mid \pi _\theta ) u(\vec {R}(\tau )) \nonumber \\ &= \int _\tau \nabla _\theta P(\tau \mid \pi _\theta ) u(\vec {R}(\tau )) \nonumber \\ &{\mathop {=}\limits ^{({\rm A}6)}}\int _\tau P(\tau \mid \pi _\theta ) \nabla _\theta \log P(\tau \mid \pi _\theta ) u(\vec {R}(\tau )) \nonumber \\= & {} {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \nabla _\theta \log P(\tau \mid \pi _\theta ) u(\vec {R}(\tau )) \right] \nonumber \\ &= {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \sum _{t=0}^T \nabla _\theta \log \pi _\theta (a_t \mid s_t) u(\vec {R}(\tau )) \right] \end{aligned}$$

(A7)

Using the law of iterated expectations, we split the trajectory $\tau$ into two parts: $\tau _{:t}, \tau _{t:}$ the parts of the trajectories before timestep t and from t onwards, respectively. We also use our definition of accrued rewards (Sect. 3.1) to split the episodic return $\vec {R}(\tau )$ into the accrued return $\vec {R}^{-}_t$ and future return $\vec {R}_t$.

$$\begin{aligned} \nabla _\theta J(\pi _\theta )= & {} {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \sum _{t=0}^T \nabla _\theta \log \pi _\theta (a_t \mid s_t) u(\vec {R}(\tau )) \right] \nonumber \\= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) u(\vec {R}(\tau )) \right] \nonumber \\= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) u(\vec {R}(\tau )) \mid \tau _{:t} \right] \right] \nonumber \\= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) u(\vec {R}^{-}_t + \vec {R}_t) \mid \tau _{:t} \right] \right] \end{aligned}$$

(A8)

Since $\nabla _\theta \log \pi _\theta (a_t \mid s_t)$ is constant with respect to the inner expectation, we can take it out:

$$\begin{aligned} \nabla _\theta J(\pi _\theta ) = \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t)\ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ u(\vec {R}^{-}_t + \vec {R}_t) \mid \tau _{:t} \right] \right] \end{aligned}$$

(A9)

Because we augment the state-space with the accrued reward, $\vec {R}^{-}_t$ depends only on $s_t$. Moreover, due to the Markovian property of (MO)MDPs, the future return $\vec {R}_t$ only depends on $s_t, a_t$. Given that $\vec {R}^{-}_t$ depends only on $s_t$ and $\vec {R}_t$ only on $s_t, a_t$, $u(\vec {R}^{-}_t + \vec {R}_t)$ does not depend on the past:

$$\begin{aligned} {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ u(\vec {R}^{-}_t + \vec {R}_t) \mid \tau _{:t} \right] = {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ u(\vec {R}^{-}_t + \vec {R}_t) \mid s_t, a_t \right] \end{aligned}$$

(A10)

Since a MOMDP is composed of a discrete number of states and actions, there are a finite number of possible returns $\vec {R}$. Thus, expanding the expectation of the previous equation results in:

$$\begin{aligned} {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ u(\vec {R}^{-}_t + \vec {R}_t) \mid s_t, a_t \right] = \sum _{\vec {R}} u(\vec {R}^{-}_t + \vec {R}) Pr\{\vec {R} \mid s_t, a_t \} \end{aligned}$$

(A11)

where $Pr\{\vec {R} \mid s_t, a_t \}$ is the probability of obtaining future return $\vec {R}$ from $s_t, a_t$. The MOCAC policy gradient is thus:

$$\begin{aligned} \nabla _\theta J(\pi _\theta ) = \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) \sum _{\vec {R}} u(\vec {R}^{-}_t + \vec {R}) Pr\{\vec {R} \mid s_t, a_t \} \right] \end{aligned}$$

(A12)

MOCAC approximates the distribution over future returns as a categorical distribution $Z_\psi$. Replacing $Pr\{\vec {R} \mid s_t, a_t \}$ with this approximation yields Eq. (13) but without a baseline $b(s_t)$. However, since the baseline only depends on $s_t$ (and not on $a_t$), we have the property that:

$$\begin{aligned} {{\,\mathrm{{\mathbb {E}}}\,}}_{a_t \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) b(s_t) \right]= & {} {{\,\mathrm{{\mathbb {E}}}\,}}_{a_t \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) \right] b(s_t)\nonumber \\= & {} \sum _{a \in {\mathcal {A}}} \pi _\theta (a_t \mid s_t) \nabla _\theta \log \pi _\theta (a_t \mid s_t) b(s_t) \nonumber \\&{\mathop {=}\limits ^{({\rm A}6)}}\sum _{a \in {\mathcal {A}}} \nabla _\theta \pi _\theta (a_t \mid s_t) b(s_t) \nonumber \\= & {} \nabla _\theta \sum _{a \in {\mathcal {A}}} \pi _\theta (a_t \mid s_t) b(s_t) = \nabla _\theta 1 b(s_t) = 0 \end{aligned}$$

(A13)

We can use the exact same proof as for the original policy gradient with baseline theorem:

$$\begin{aligned} \nabla _\theta J(\pi _\theta )= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t)\ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ (u(\vec {R}^{-}_t + \vec {R}_t) - b(s_t)) \mid s_t, a_t \right] \right] \nonumber \\= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t)\ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ u(\vec {R}^{-}_t + \vec {R}_t) \mid s_t, a_t \right] \right. \nonumber \\{} & {} \left. -\nabla _\theta \log \pi _\theta (a_t \mid s_t)\ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ b(s_t) \mid s_t, a_t \right] \right] \nonumber \\= & {} \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t)\ {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{t:} \sim \pi _\theta } \left[ (u(\vec {R}^{-}_t + \vec {R}_t)) \mid s_t, a_t \right] - 0 \right] \end{aligned}$$

(A14)

Thus, we can use a baseline that only depends on $s_t$ as it does not affect the expectation ${{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta }$. In the case of MOCAC, we use the expected utility for state $s_t$, i.e. $b(s_t) = \sum _{\vec {R}} u(\vec {R}^{-}_t + \vec {R}) Pr\{\vec {R} \mid s_t \}$. Plugging this in Eq. (A12) results in:

$$\begin{aligned} \nabla _\theta J(\pi _\theta ) = \sum _{t=0}^T {{\,\mathrm{{\mathbb {E}}}\,}}_{\tau _{:t} \sim \pi _\theta } \left[ \nabla _\theta \log \pi _\theta (a_t \mid s_t) A(s_t, a_t) \right] \end{aligned}$$

(A15)

with

$$\begin{aligned} A(s_t, a_t) = \sum _{\vec {R}} u(\vec {R}^{-}_t + \vec {R}) Pr\{\vec {R} \mid s_t, a_t \} - \sum _{\vec {R}} u(\vec {R}^{-}_t + \vec {R}) Pr\{\vec {R} \mid s_t \} \end{aligned}$$

(A16)

We use this gradient to update the policy of MOCAC with gradient ascent.

Appendix B: Environment details

1.1 B.1 Split

In the Split environment, the agent chooses between two hallways of length 11 to traverse. Including the start- and end-states, there are 26 states. The states are one-hot encoded, resulting in a 26-sized vector that is given as input to any of the actor and critic estimators.

Table 1 Hyperparameters for the split, deep-sea-treasure and Minecart environments

1.2 B.2 Deep sea treasure

Deep sea treasure is a grid-world environment, where the submarine moves on a $11 \times 12$, resulting in 132 different states. The state number is one-hot encoded, resulting in a 132-sized vector that is given as input to any of the actor and critic estimators. The rewards provided by each treasure are made in such a way that every optimal $\texttt {treasure}\times \texttt {fuel}$ combination is evenly spread out on the convex coverage set. Treasure values are displayed on Fig. 8.

Fig. 8

The Deep Sea Treasure environment. The agent starts on the top-left corner and tries to reach any of the treasures. Further treasures are worth more

1.3 B.3 Minecart

The state is a top-down visual representation of the mining area. It displays the different mine locations, the cart and the base station. A frame of the environment can be seen in Fig. 9. The frames are pre-processed as follows: they are rescaled to $42\times 42$, converted to grayscale and normalized. Moreover, we keep a history of 2 frames as observation. This is similar to the frame-preprocessing done by [16] on the Arcade Learning Environment suite (Table 1).

Fig. 9

The Minecart environment. The base is located at the top-left corner, while the 5 mines are spread around the environment

Appendix C: Hyperparameters

All hyperparameters used for all these experiments, including neural network architectures are listed in Table 1. All hyperparameters used for the ablative study are listed in Table 2. Figure 11 shows results for Deep-Sea-Treasure using a linear utility function with different weight-parameters. Figure 12 shows results for diverse utility functions on Fishwood, with an additional single-objective baseline.

Fig. 10

The base of each actor and critic for the convolutional neural network used for the Minecart environment

Table 2 Hyperparameters for the experiments on MiniRandom and Fishwood. Since the input-size of the neural networks depend on the conditioning, the neurons rows are variable

Fig. 11

Results for Deep-Sea-Treasure using a linear function with diverse sets of weights

Fig. 12

Results for diverse utility functions as in Fig. 4, but with the added single-objective A2C (terminal) baseline