User Preferences in Bayesian Multi-objective Optimization: The Expected Weighted Hypervolume Improvement Criterion

Abstract

In this article, we present a framework for taking into account user preferences in multi-objective Bayesian optimization in the case where the objectives are expensive-to-evaluate black-box functions. A novel expected improvement criterion to be used within Bayesian optimization algorithms is introduced. This criterion, which we call the expected weighted hypervolume improvement (EWHI) criterion, is a generalization of the popular expected hypervolume improvement to the case where the hypervolume of the dominated region is defined using a user-defined absolutely continuous measure instead of the Lebesgue measure. The EWHI criterion takes the form of an integral for which no closed form expression exists in the general case. To deal with its computation, we propose an importance sampling approximation method. A sampling density that is optimal for the computation of the EWHI for a predefined set of points is crafted and a sequential Monte-Carlo (SMC) approach is used to obtain a sample approximately distributed from this density. The ability of the criterion to produce optimization strategies oriented by user preferences is demonstrated on a simple bi-objective test problem in the cases of a preference for one objective and of a preference for certain regions of the Pareto front.

A Approximate Variance of the EI Estimator

We derive in this appendix the variance of the SMC estimator for $\rho _n$. In the SMC procedure that we consider, the particles $\left( y_{n,i}\right) _{1 \le i \le m}$ are obtained from a sequence of densities $(\pi _{n,t})_{0 \le t \le T}$, where $\pi _{n,0}$ is an easy-to-sample initial density and $\pi _{n,T} = \pi _n$ is the target density. Let $(\gamma _{n,t})_{0 \le t \le T}$ and $(Z_{n,t})_{0 \le t \le T}$ denote the corresponding sequences of un-normalized densities and normalizing constants.

First, observe that, for $1\le t \le T$,

$$\begin{aligned} \begin{array}{lcl} Z_{n,t} &{}=&{} \displaystyle \int _{H_{n}^\mathrm{c}} \gamma _{n,t}(y)\,\mathrm {d}y\\ &{}=&{} \displaystyle Z_{n,t-1} \int _{G_n} \frac{\gamma _{n,t}(y)}{\gamma _{n,t-1}(y)}\, \pi _{n,t-1}(y)\, \mathrm {d}y. \end{array} \end{aligned}$$

(24)

Thus, we can derive a sequence of approximations $\widehat{Z}_{n,t}$ of $Z_{n,t}$, $t\ge 1$, using the following recursion formula:

$$\begin{aligned} \left\{ \begin{array}{l} \widehat{Z}_{n,0} = Z_{n,0} = \int _{G_n} \gamma _{n,0}(y)\, \mathrm {d}y,\\ \widehat{Z}_{n,t} = \widehat{Z}_{n,t-1} \left( \frac{1}{m} \sum _{i=1}^{m} \frac{\gamma _{n,t}(y_{n,t-1,i})}{\gamma _{n,t-1}(y_{n,t-1,i})} \right) , \end{array}\right. \end{aligned}$$

(25)

where the particles $(y_{n,t-1,i})_{1 \le i \le m}\sim \pi _{n,t-1}$ are obtained using an SMC procedure (see, e.g., [6]). The estimator of $\rho _n(x)$ that we actually consider is then

(26)

where

(27)

and

$$\begin{aligned} \widehat{ Z}_n = \widehat{ Z}_{n,T} = Z_{n,0} \prod _{u=1}^T \widehat{\theta }_{n,\,u}, \end{aligned}$$

(28)

with

$$\begin{aligned} \widehat{\theta }_{n,t} = \frac{1}{m} \sum _{i=1}^{m} \frac{\gamma _{n,t}(y_{n,t-1,i})}{\gamma _{n,t-1}(y_{n,t-1,i})}\,. \end{aligned}$$

(29)

Now, assume the idealized setting, as usual in the SMC literature (see, e.g., [9]), where

• (i) $y_{n,t,i} {\mathop {\sim }\limits ^{\scriptstyle \text {i.i.d}}} \pi _{n,t}$, $1 \le i \le m$,

• (ii) the samples $\mathcal {Y}_{n,t} = (y_{n,t,i})_{1 \le i \le m}$ are independent, $0\le t \le T$.

Observe from  (19) and (24) that under (i), $\widehat{\alpha }_n(x)$ is an unbiased estimator of $\alpha _n(x) = \frac{\rho _n(x)}{Z_n}$, and $\widehat{\theta }_{n,t}$ is an unbiased estimator of $\theta _{n,t} = \frac{Z_{n,t}}{Z_{n,t-1}}$, $1 \le t \le T$. Moreover, under (ii), $\widehat{\alpha }_n(x)$ and the $(\widehat{\theta }_{n,t})_{1 \le t \le T}$ are independent. Thus,

We obtain the coefficient of variation of $\widehat{\rho }_n(x)$

$$\begin{aligned} \frac{\mathrm {Var}\, \widehat{\rho }_n(x) }{\rho _n(x)^2} = \varLambda _n(x)^2 + \left( 1+\varLambda _n(x)^2 \right) \varDelta _{n,T}^2, \end{aligned}$$

(30)

where $\varLambda _n(x)^{2} = \frac{\mathrm {Var}\,\widehat{\alpha }_n(x)}{\alpha _n(x)^2}$ and $\varDelta _{n,t}^2 = \frac{\mathrm {Var} \widehat{Z}_{n,t}}{Z_{n,t}^2}$ are the coefficients of variation of $\widehat{\alpha }_n(x)$ and $\widehat{Z}_{n,t}$ respectively.

Using the same ideas as above, we have

$$\begin{aligned} \varDelta _{n,t}^2 = \delta _{n,t}^2 + \left( 1+\delta _{n,t}^2 \right) \varDelta _{n,t-1}^2, \end{aligned}$$

(31)

where $\delta _{n,t}^2 = \frac{\mathrm {Var}\, \widehat{\theta }_{n,t}}{\theta _{n,t}^2}$ is the coefficient of variation of $\widehat{\theta }_{n,t}$.

Estimators of $\varLambda _{n}(x)^2$, $\varDelta _{n,t}^2$ and $\delta _{n,t}^2$ can be derived under (ii). For instance, observe that

(32)

Thus, an estimator of $\delta _{n,t}^2$ is

$$\begin{aligned} \widehat{\delta }_{n,t}^2 = \displaystyle \frac{ \sum _{i=1}^{m} \frac{\gamma _{n,t}(y_{n,t-1,i})^2}{\gamma _{n,t-1}(y_{n,t-1,i})^2} }{\biggl ( \sum _{i=1}^{m} \frac{\gamma _{n,t}(y_{n,t-1,i})}{\gamma _{n,t-1}(y_{n,t-1,i})} \biggr )^2} - \frac{1}{m}. \end{aligned}$$

(33)

Plugging (33) in (31), we obtain an estimator of $\varDelta _{n,t}^2$:

$$\begin{aligned} \widehat{\varDelta }_{n,t}^2 = \widehat{\delta }_{n,t}^2 + \left( 1+\widehat{\delta }_{n,t}^2 \right) \cdot \widehat{\varDelta }_{n,t-1}^2. \end{aligned}$$

(34)

Similarly, an estimator of $\varLambda _n(x)^2$ is

(35)

As a result, we obtain the following numerically tractable approximation of the variance of $\widehat{\rho }_n(x)$:

(36)

where $\widehat{Z}_{n,t}$ and $\widehat{\varDelta }_{n,t}^2$ are obtained recursively using (25) and (34), $\widehat{\varLambda }_n(x)^2$ is computed using  (35) and $\widehat{\rho }_n(x)$ is computed using (26).