On Nonuniqueness of Solutions of Hamilton–Jacobi–Bellman Equations

Abstract

An example of a nonunique solution of the Cauchy problem of Hamilton–Jacobi–Bellman (HJB) equation with surprisingly regular Hamiltonian is presented. The Hamiltonian H(t, x, p) is locally Lipschitz continuous with respect to all variables, convex in p and with linear growth with respect to p and x. The HJB equation possesses two distinct lower semicontinuous solutions with the same final conditions; moreover, one of them is the value function of the corresponding Bolza problem. The definition of lower semicontinuous solution was proposed by Frankowska (SIAM J. Control Optim. 31:257–272, 1993) and Barron and Jensen (Commun. Partial Differ. Equ. 15(12):1713–1742, 1990). Using the example an analysis and comparison of assumptions in some uniqueness results in HJB equations is provided.

1 Introduction

The classical Cauchy problem for the Hamilton–Jacobi–Bellman equation is a partial differential equation with the final condition

$$\begin{aligned} \begin{array}{rll} -U_{t}+ H(t,x,-U_{x})=0 &{}\,in\, &{} ]0,T[\,\times \,{\mathbb {R}}^n, \\ U(T,x)=g(x) &{} \,in\, &{} \,{\mathbb {R}}^n. \end{array}\qquad \quad \qquad \qquad \qquad \text {HJB} \end{aligned}$$

If the Hamiltonian H is convex in the gradient variable, then there are relations between solutions of HJB and optimization problems involving a function dual to H. This function, called the Lagrangian and denoted by L, is the Legendre–Fenchel transform of H in its gradient variable:

$$\begin{aligned} L(t,x,v)=\sup _{p\in {\mathbb {R}}^{n}}\,\{\,\langle v,p\rangle -H(t,x,p)\,\}. \end{aligned}$$

(1.1)

The HJB equation is related to the value function of the Bolza problem $V:[0,T]\times {\mathbb {R}}^n\rightarrow {\mathbb {R}}\cup \{+\infty \}$ which is defined as follows:

$$\begin{aligned} V(t_0,x_0)= \inf \left\{ \,g(x(T))+\int _{t_0}^TL(t,x(t),\dot{x}(t))\,dt\,\bigg |\, x(\cdot )\in \mathcal {A}[t_0,T],\, x(t_0)=x_0\,\right\} , \end{aligned}$$

(1.2)

where $\mathcal {A}[t_0,T]$ denotes the space of absolutely continuous functions from $[t_0,T]$ into ${\mathbb {R}}^n$. If the value function is differentiable, it is well-known that it satisfies HJB in the classical sense. However, in many situations the value function is not differentiable. Then the solution of the HJB equation must be defined in nonsmooth sense in such a way that under quite general assumptions on H and g, V is the unique solution of HJB. Since we use the nonsmooth analysis we need a notion of a subgradient. For a vector $v\in {\mathbb {R}}^n$ and a function $f:{\mathbb {R}}^n\rightarrow {\mathbb {R}}\cup \{+\infty \}$, v is a subgradient of f at $x\in {\mathrm {dom}}\,f$, written $v\in \partial f(x)$, if

$$\begin{aligned} \liminf \limits _{y\rightarrow x,\,y\not =x}\frac{f(y)-f(x)-\langle v,y-x\rangle }{|y-x|}\geqslant 0. \end{aligned}$$

(1.3)

In 1990 Baron and Jensen [3] and Frankowska [7] introduced extended viscosity solutions to semicontinuous functions for Hamiltonian that is convex in the gradient variable and provided a uniqueness result. Frankowska [7] called these solutions lower semicontinuous solutions.

Definition 1.1

A function $U:[0,T]\times {\mathbb {R}}^n\rightarrow {\mathbb {R}}\cup \{+\infty \}$ is lower semicontinuous solution of the HJB equation if it satisfies the following:

• (i) U is lower semicontinuous function and $U(T,x)=g(x)$;

• (ii) For every $(t,x)\in {\mathrm {dom}}\,U$, every $(p_{t},p_{x})\in \partial U(t,x)$ the following holds:

  $$\begin{aligned} \left\{ \begin{array}{lll} -p_{t}+H(t,x,-p_{x})\geqslant 0 &{} \,{if}\, &{} t\in [0,T[, \\ -p_{t}+H(t,x,-p_{x})\leqslant 0 &{} \,{if}\, &{} t\in \, ]0,T]. \end{array}\right. \end{aligned}$$

  (1.4)

The main goal of the paper is to present an example of two distinct lower semicontinuous solutions of the HJB equation with surprisingly regular Hamiltonian. Understanding the role that the Lipschitz-type condition plays in theorems about uniqueness of solution of HJB is also important. The proposed Hamiltonian $H\!:[0,T]\!\times \!{\mathbb {R}}\times \!{\mathbb {R}}\rightarrow {\mathbb {R}}$ satisfies the classical assumptions i.e. firstly, it is convex with respect to p, secondly, it increases linearly in p and x, i.e. $|H(t,x,p)|\leqslant 2|p|$ for any $t\in [0,T]$, $x,p\in {\mathbb {R}}$, thirdly, it is locally Lipschitz continuous, i.e.

$$\begin{aligned} \begin{array}{l} \forall \, r>0\,\, \exists \, k>0\,\,\forall \, t,s\in [0,T]\,\, \forall \, x,y\in r\mathbb {B}\,\, \forall \, p,q\in r\mathbb {B}\\ |H(t,x,p)-H(s,y,q)|\leqslant k(|t-s|+|x-y|+|p-q|), \end{array} \qquad \qquad \qquad \qquad \quad \text {LLC} \end{aligned}$$

where $\mathbb {B}$ is the closed unit ball. In addition, we show that one of the indicated lower semicontinuous solutions is the value function given by (1.2). In general, for the uniqueness of HJB solutions, one needs some stronger Lipschitz-type condition, that we shall study further in connection to the results of uniqueness.

Frankowska [7] proved that, the value function is the unique lower semicontinuous solution of the HJB equation if the Hamiltonian meets the classical assumptions and it is positively homogeneous in p, i.e. $\forall _{r>0}\,H(t,x,rp)=rH(t,x,p)$. Actually, the result of Frankowska does not require local Lipschitz continuity with respect to the triple (t, x, p). It is enough to assume it is satisfied with respect to state variable x only. The example of nonuniqueness of solution of HJB introduced in the current paper, does not contradict the result of Frankowska as the Hamiltonian in our example, fulfills the classical assumptions, but it is not positively homogeneous in p.

Earlier Ishii [11, Theorem 2.5] and Crandall and Lions [5, Theorem VI.1] had proved the uniqueness of viscosity solutions of HJB in the class of continuous functions. They had assumed instead of linear growth in p and x of Hamiltonian, the following condition

$$\begin{aligned} \begin{array}{l} \exists \, C>0\,\,\forall \, t\in [0,T]\,\, \forall \, x\in {\mathbb {R}}^n\,\,\forall \, p,q\in {\mathbb {R}}^n \\ |H(t,x,p)-H(t,x,q)|\,\leqslant \, C(1+|x|)|p-q|. \end{array} \end{aligned}$$

(1.5)

One can show that if the Hamiltonian is convex in p and possesses a linear growth of the form $H(t,x,p)\leqslant 2(1+|x|)|p|$, then condition (1.5) holds with constant $C=2$. Therefore the Hamiltonian from our example of nonuniqueness also satisfies (1.5). Next, the results from papers [11, Theorem 2.5] and [5, Theorem VI.1] require the Lipschitz-type condition for the Hamiltonian:

$$\begin{aligned} \begin{array}{l} \forall \, r>0\,\, \exists \, k>0\,\,\forall \, t,s\in [0,T]\,\, \forall \, x,y\in r\mathbb {B}\,\, \forall \, p\in {\mathbb {R}}^n \\ |H(t,x,p)-H(s,y,p)|\leqslant k(1+|p|)(|t-s|+|x-y|), \end{array} \qquad \qquad \qquad \qquad \qquad \text {SLC} \end{aligned}$$

that is derived from the optimal control problem. The meaning of (SLC) in the optimal control problems is discussed in the papers of Frankowska and Sedrakyan [8] and Rampazzo [18]. The results in [11, Theorem 2.5] and [5, Theorem VI.1] do not require the convexity of the Hamiltonian in p, but the uniqueness of HJB solutions is obtained in the class of continuous functions. Using these results, Bardi and Capuzzo-Dolcetta [2, Chapter V, Theorem 5.16] showed the uniqueness of HJB solutions in the class of lower semicontinuous functions assuming additionally the convexity of the Hamiltonian in p. Because the Hamiltonian in our example of nonuniqueness satisfies the classical assumptions, also the condition (1.5) is satisfied. It means that in the uniqueness results, the key point is (SLC), at least in the case of lower semicontinuous solutions of the HJB equation.

In order to understand better the reason for nonuniqueness of the solution of the HJB equation given in our example, we need to recall the Loewen–Rockafellar condition from [12] that is more general version of (SLC). This condition was used by Loewen and Rockafellar [13, 14] to study necessary conditions satisfied by optimal solutions of Bolza problem. Galbraith [9] proved that, the value function is the unique lower semicontinuous solution of HJB assuming the Loewen–Rockafellar condition [12]. This uniqueness result has, in general, the same nature as the uniqueness result of Dal Maso and Frankowska [6]. The Hamiltonian from our example of nonuniqueness is constructed using the function $\varphi$. In Sect. 6 we show that if $\varphi$ is replaced by a function that is sufficiently regular, then we obtain the Hamiltonian that does not satisfy the condition (SLC), while it satisfies the Loewen–Rockafellar condition [12]. Therefore, by virtue of classical results, we are not able to say if after the mentioned change, we get the uniqueness of the solution of HJB or not. However, using the Galbraith result [9], we know that HJB with this Hamiltonian has the unique solution. Thus, the HJB equation can have the unique solution even if the Hamiltonian does not satisfy the condition (SLC). Moreover, the Hamiltonian from our example of nonuniqueness shows the difference between (LLC), (SLC) and Loewen–Rockafellar condition [12]. We know that the condition (LLC) does not guarantee uniqueness of the solution to the HJB equation, so the natural question can be stated—what extra conditions the Hamiltonian should satisfy in order the HJB equation has the unique solution? The answer to this question is obtained when analysing conditions (LLC), (SLC) and Loewen–Rockafellar condition [12]. Namely, some extra conditions in the interdependence between space and subgradient for large values of the subgradient are mandatory (see Sect. 2).

In the literature, an example of nonuniqueness of the solution of the equation of HJB is known. Crandall and Lions in their fundamental article [4] give an example of nonuniqueness of viscosity solution of the transport equation

$$\begin{aligned} -U_{t}+b(x)\cdot (-U_x)=0,\quad U(T,x)=g(x). \end{aligned}$$

In this example, the function $b(\cdot )$ is bounded and continuous, but is not locally Lipschitz continuous. Therefore the Hamiltonian $H(t,x,p)=b(x)\,p$ is convex, continuous and satisfies the condition (1.5), but it does not satisfy (LLC). The solutions in the Crandall–Lions example are continuous, and in our example, they are lower semicontinuous. Now we consider an easy example of nonuniqueness of lower semicontinuous solutions of the transport equation. Let $b(x)=x^2$ and $g\equiv 0$. Then functions $V\equiv 0$ and $U(t,x)=0$, $(T-t)x\leqslant 1$, $U(t,x)=1$, $(T-t)x>1$ are lower semicontinuous solutions of the transport equation. In this example, the Hamiltonian $H(t,x,p)=x^2p$ satisfies (LLC), (SLC) and Loewen–Rockafellar condition [12], but it does not satisfy (1.5) or equivalently it does not have a linear growth in x. It means that these conditions play important roles in the uniqueness of lower semicontinuous solutions of HJB. Notice that Hamiltonians from the above two examples of nonuniqueness do not satisfy the classical assumptions, but the Hamiltonian from our example of nonuniqueness does. In the transport equation we cannot find the example of nonuniqueness as ours. Because on one hand the HJB equation with the Hamiltonian satisfying the classical assumptions and positively homogeneous in p has the unique solution (see [7]). On the other hand conditions (LLC), (SLC) and Loewen–Rockafellar condition [12] are equivalent, if the Hamiltonian is convex in p, positively homogeneous in p and continuous. Therefore, we cannot find differences between them while considering Hamiltonians from the transport equation.

Summarizing, the above examples of nonuniqueness show that the key role in the uniqueness of viscosity solutions of HJB is played by the Lipschitz-type conditions originated in optimization instead of local Lipschitz continuity originated in differential equations theory. Therefore our result is proper for optimization problems.

We have found the example of nonuniqueness presented here while generalizing a result of Plaskacz and Quincampoix [17]. This generalization was published in [16].

2 Existence and Uniqueness Theorem

In this section we present the well-known theorem on existence and uniqueness of lower semicontinuous solutions of the HJB equation. We discuss results of the paper on the basis of this theorem. First, we introduce basic assumptions on Hamiltonian:

• (H1) $H:[0,T]\times {\mathbb {R}}^{n}\times {\mathbb {R}}^{n}\rightarrow {\mathbb {R}}$ is continuous;

• (H2) H(t, x, p) is convex with respect to p for every $(t,x)\in [0,T]\times {\mathbb {R}}^n$;

• (H3) there exists a constant $C>0$ such that for all $(t,x,p)\in [0,T]\times {\mathbb {R}}^n\times {\mathbb {R}}^n$ the following inequality is satisfied $H(t,x,p)\leqslant C(1+|x|)|p|$.

Condition (H3) is called a linear growth in p and x of the Hamiltonian. Basing on (H1)–(H3) we can state the following theorem on existence and uniqueness:

Theorem 2.1

We suppose that H satisfies (H1)–(H3) and (SLC). Let g be lsc and bounded from below, and V be the value function associated with g and L, where L is given by (1.1). Then the value function V is bounded from below lower semicontinuous solution of HJB. Moreover, if U is bounded from below lower semicontinuous solution of HJB, then $U=V$ on $[0,T]\times {\mathbb {R}}^n$.

We shall show that Theorem 2.1 is a particular case of Theorem 2.2 from [9]. To this end, we need to discuss the subgradient characterization of the condition (SLC) and the Loewen–Rockafellar condition from [12]. A subgradient is defined for the function given on the entire Euclidean space (see Definition 1.3), so to use a subgradient of the Lagrangian L we extend it in the following way: $L(t,x,v):=L(0,x,v)$ for $t<0$ and $L(t,x,v):=L(T,x,v)$ for $t>T$.

$$\begin{aligned} \begin{array}{l} \text {For every}\, r>0\, \text {there exists}\, k>0\, \text {such that at every point}\, (t,x,v)\\ \in [0,T]\times r\mathbb {B}\times {\mathbb {R}}^n,\, {every}\, (w_1,w_2,p)\in \partial L(t,x,v)\, \text {the inequality}\\ |(w_1,w_2)|\leqslant k(1+|p|)\, \text {holds.} \end{array} \end{aligned}$$

(2.1)

Condition (2.1) is equivalent to (SLC), if Hamiltonian satisfies (H1)–(H3). Moreover, one can prove that the condition (2.1) is a subgradient characterization of Lipschitz continuity of the multifunction $(t,x)\rightarrow {\mathrm {epi}}\,L(t,x,\cdot )$ in the Hausdorff’s sense. Besides, it is easy to see that the condition (2.1) implies the following one:

$$\begin{aligned} \begin{array}{l} \text {For every}\, r>0\, \text {there exists}\, k>0\, \text {such that at every point}\, (t,x,v)\\ \in [0,T]\times r\mathbb {B}\times {\mathbb {R}}^n,\, \text {every}\, (w_1,w_2,p)\in \partial L(t,x,v)\, \text {we have}\\ |(w_1,w_2)|\leqslant k(1+|v|+|L(t,x,v)|)(1+|p|). \end{array} \end{aligned}$$

(2.2)

The condition (2.2) is a subgradient characterization of the Aubin type continuity of multifunction $(t,x)\rightarrow {\mathrm {epi}}\,L(t,x,\cdot )$. This kind of Aubin continuity was introduced by Loewen and Rockafellar in [12, Definition 2.3, (b)]. However, a subgradient characterization can be found in [10, Proposition 3.4]. Galbraith [9] obtained the uniqueness result assuming convexity of Hamiltonian with respect to p, a mild growth of Hamiltonian with respect to p (see (A1) from [9]) and slightly more general Lipschitz-type condition than (2.2) (see (A2) from [9]). Thus, if we replace the condition (SLC) by (2.2) in Theorem 2.1, then the claim of Theorem 2.1 still holds. In [9, 10, 12,13,14] the authors use a limiting subgradient that is larger than the regular subgradient we use. We know that replacing the regular subgradient by the limiting subgradient one the condition (2.2) is unchanged if we assume that Hamiltonian is continuous. We also know that the condition (2.2) implies (LLC) (see [9], Proposition 2.4). Therefore, assuming conditions (H1)–(H3) we obtain

$$\begin{aligned} {(SLC)}\,\, \Longrightarrow \,\,{({2.2})}\,\, \Longrightarrow \,\, {(LLC)}. \end{aligned}$$

(2.3)

Further on, we will show that the implications (2.3) cannot be reversed. Moreover, we prove that the Lipschitz continuity (LLC) raising from differential equations is not sufficient for the uniqueness of lower semicontinuous solution of HJB. However, conditions (SLC) and (2.2) of optimization problems are sufficient for uniqueness.

3 Example of Hamiltonian

In this section we define a Hamiltonian and discuss its regularity. In the next section we show that the HJB equality with this Hamiltonian does not have the unique lower semicontinuous solution. We define an auxiliary function $\varphi :[0,T]\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ by the formula

$$\begin{aligned} \varphi (t,x)=\sqrt{|t-x|}\,\exp \left( 2\sqrt{|t-x|}\right) . \end{aligned}$$

(3.1)

The Hamiltonian $H:[0,T]\times {\mathbb {R}}\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ is defined by the formula

$$\begin{aligned} H(t,x,p)=\left\{ \begin{array}{ccl} \,0 &{} \,{if}\,\, &{} 2|p|\leqslant \frac{1}{\varphi (t,x)},\,t \not =x, \\ \,2|p|-\frac{1}{\varphi (t,x)} &{} \,{if}\,\, &{} 2|p|>\frac{1}{\varphi (t,x)},\,t \not =x, \\ \,0 &{} \,{if}\,\, &{} p\in {\mathbb {R}},\, t=x. \end{array} \right. \end{aligned}$$

(3.2)

It is not difficult to see that the Hamiltonian H given by (3.2) is continuous on $[0,T]\times {\mathbb {R}}\times {\mathbb {R}}$, convex with respect to p for each $(t,x)\in [0,T]\times {\mathbb {R}}$ and has a linear growth in p and x, i.e. $|H(t,x,p)|\leqslant 2|p|$ for all $t\in [0,T]$, $x,p\in {\mathbb {R}}$, so it satisfies (H1)–(H3).

Theorem 3.1

The Hamiltonian H given by (3.2) satisfies locally the Lipschitz continuity, i.e. for each $(t_0,x_0,p_0)\in [0,T]\times {\mathbb {R}}\times {\mathbb {R}}$ there exist numbers $r,k>0$ such that

$$\begin{aligned} \begin{array}{l} \forall \, t,s\in \mathbb {B}(t_0,r)\,\, \forall \, x,y\in \mathbb {B}(x_0,r)\,\, \forall \, p,q\in \mathbb {B}(p_0,r)\\ |H(t,x,p)-H(s,y,q)|\leqslant k(|t-s|+|x-y|+|p-q|). \end{array} \end{aligned}$$

(3.3)

Proof

For $\theta \geqslant 0$ we define an auxiliary function f by the formula $f(\theta )=\sqrt{\theta }\exp (2\sqrt{\theta })$. We notice that the function f is increasing and $f(|t-x|)=\varphi (t,x)$ for each $t\in [0,T]$, $x\in {\mathbb {R}}$. Furthermore, the function f on [a, b] satisfies Lipschitz continuity if $0<a<b$. We fix $t_0\in [0,T]$, $x_0\in {\mathbb {R}}$ and $p_0\in {\mathbb {R}}$, and consider two cases.

Case 1 Let $p_0\in {\mathbb {R}}$ and $t_0=x_0$. We define $r:=1/[2\exp (4)(1+2|p_0|)^2]$ and $k:=1$. We notice that $|p|\leqslant r+|p_0|$ for each $p\in \mathbb {B}(p_0,r)$ and $|t-x|\leqslant 2r$ for each $t\in \mathbb {B}(t_0,r)$, $x\in \mathbb {B}(x_0,r)$, with $r\leqslant 1/2$. Therefore, for $t\not =x$ we obtain

$$\begin{aligned} 2|p|\leqslant 2(r+|p_0|)\leqslant 1+2|p_0|=\frac{1}{\sqrt{2r}\exp (2)}\leqslant \frac{1}{f(2r)}\leqslant \frac{1}{f(|t-x|)} =\frac{1}{\varphi (t,x)}. \end{aligned}$$

By the definition of the Hamiltonian $H(t,x,p)=0$ if $p\in {\mathbb {R}}$, $t=x$ and $2|p|\leqslant 1/\varphi (t,x)$, $t\not =x$. Therefore, $H(t,x,p)=0$ for each $t\in \mathbb {B}(t_0,r)$, $x\in \mathbb {B}(x_0,r)$, $p\in \mathbb {B}(p_0,r)$, so we have (3.3).

Case 2 Let $p_0\in {\mathbb {R}}$ and $t_0\not =x_0$. We define r by the formula $r:=|t_0-x_0|/3$, then $r\leqslant |t-x|\leqslant 5r$ for each $t\in \mathbb {B}(t_0,r)$, $x\in \mathbb {B}(x_0,r)$. Let l be the Lipschitz constant of the function f on [r, 5r]. We define k by $k:=2+l/f^2(r)$.

Since $t\not = x$ for each $t\in \mathbb {B}(t_0,r)$, $x\in \mathbb {B}(x_0,r)$, then by the definition of the Hamiltonian (3.2) we obtain the following relations:

• (a) For $2|p|\leqslant 1/\varphi (t,x)$ and $2|q|\leqslant 1/\varphi (s,y)$ we have ${LS}(3.3)= 0 \leqslant {RS}(3.3)$.

• (b) For $2|p|\geqslant 1/\varphi (t,x)$ and $2|q|\geqslant 1/\varphi (s,y)$ we have inequalities

  $$\begin{aligned} {LS}({3.3})\leqslant & {} 2|p-q|+\frac{1}{f(|t-x|)f(|s-y|)}|f(|t-x|)-f(|s-y|)|\\\leqslant & {} 2|p-q|+\frac{l}{f^2(r)}\left( |t-s|+|x-y|\right) \,\,\leqslant \,\, {RS}({3.3}). \end{aligned}$$

• (c) For $2|p|\geqslant 1/\varphi (t,x)$ and $2|q|\leqslant 1/\varphi (s,y)$ we have inequalities

  $$\begin{aligned} {LS}({3.3}) \,\leqslant \, 2|p|-\frac{1}{\varphi (t,x)}+\frac{1}{\varphi (s,y)} -2|q|\,\leqslant \, {RS}({3.3}). \end{aligned}$$

The consequence of cases (a)–(c) is the inequality (3.3). $\square$

The Lagrangian $L\!:\![0,T]\!\times \!{\mathbb {R}}\!\times \!{\mathbb {R}}\rightarrow {\mathbb {R}}\cup \{\!+\!\infty \}\!$ given by the formula (1.1) has the following form:

$$\begin{aligned} L(t,x,v)=\left\{ \begin{array}{ccl} +\infty &{}\qquad \mathrm{if}&{}\, |v|> 2,\,t\not =x, \\ \frac{|v|}{2\varphi (t,x)} &{}\qquad \mathrm{if} &{} \, |v|\leqslant 2,\,t\not =x, \\ 0 &{} \qquad \mathrm{if}&{}\, v=0,\, t=x,\\ +\infty &{} \qquad \mathrm{if}&{}\, v\not =0,\,t=x. \end{array} \right. \end{aligned}$$

(3.4)

Now we prove that Lagrangian (3.4) does not satisfy the condition (2.2). The proof of this fact shows how condition (2.2) is violated for large values of gradients. It implies that the second implication in (2.3) cannot be reversed. In Sect. 6 we will see that large values of gradients not necessarily violate condition (2.2), if $\varphi$ is sufficiently regular.

Proposition 3.2

Lagrangian (3.4) does not satisfy the condition (2.2).

Proof

Let the Lagrangian L be given by the formula (3.4). Then for $t>x$, $v\in \,]0,2[$ and $(w_1,w_2,p)\in \partial L(t,x,v)$ we have

$$\begin{aligned} p=\frac{1}{2\varphi (t,x)}, \qquad -w_1=w_2=\frac{v}{2\varphi (t,x)} \left[ \frac{1}{2(t-x)}+\frac{1}{\sqrt{(t-x)}}\right] . \end{aligned}$$

Therefore, the left and right hand sides of the inequality (2.2) are given by

$$\begin{aligned} {LS}(2.2)=\sqrt{2}\,w_2, \qquad {RS}(2.2)=k\left( 1+v+ \frac{v}{2\varphi (t,x)}\right) \left( 1+\frac{1}{2\varphi (t,x)}\right) . \end{aligned}$$

Let $t_n-x_n\rightarrow 0+$ and $v_n=2\varphi (t_n,x_n)$. If the inequality (2.2) is satisfied, then for large $n\in {\mathbb {N}}$

$$\begin{aligned} \sqrt{2}\left[ \frac{1}{2(t_n-x_n)}+\frac{1}{\sqrt{(t_n-x_n)}}\right]\leqslant & {} 2k(1+\varphi (t_n,x_n))\left( 1+\frac{1}{2\varphi (t_n,x_n)}\right) . \end{aligned}$$

Multiplying the above inequality by $2(t_n-x_n)$ we have the inequality

$$\begin{aligned} \sqrt{2}+2\sqrt{2(t_n-x_n)}\leqslant & {} 2k\left[ 1+\varphi (t_n,x_n)\right] \left( 2(t_n-x_n)+\frac{\sqrt{t_n-x_n}}{\exp \left( 2\sqrt{t_n-x_n}\, \right) }\right) . \end{aligned}$$

Passing to the limit, we obtain a contradiction. $\square$

4 An Example of Nonuniqueness

In this section we present two different, bounded, lower semicontinuous solutions of HJB with the Hamiltonian H given by (3.2) and the terminal condition $g:{\mathbb {R}}\rightarrow {\mathbb {R}}$ given by the formula

$$\begin{aligned} g(x)=\left\{ \begin{array}{ccl} \,\exp \left( -2\sqrt{x-T}\,\right) -1 &{} \quad \mathrm{if}\,\, &{} x\geqslant T, \\ \,1 &{} \quad \mathrm{if}\,\, &{} x<T. \end{array} \right. \end{aligned}$$

(4.1)

We are going to define some notions for the whole Euclidean space so we extend U(t, x) from $[0,T]\times {\mathbb {R}}$ to entire space by setting $+\infty$ for $(t,x)\notin [0,T]\times {\mathbb {R}}$.

4.1 First Solution

Let the function $U:[0,T]\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ be given by the formula

$$\begin{aligned} U(t,x)=\left\{ \begin{array}{ccl} \,\exp \left( -2\sqrt{x-t}\,\right) -1 &{} \,{if}\, &{} x\geqslant t, \\ \,1 &{} \,{if}\, &{} x<t. \end{array} \right. \end{aligned}$$

(4.2)

Theorem 4.1

The function U given by (4.2) is bounded lower semicontinuous solution of the HJB equation with the Hamiltonian (3.2) and the terminal condition (4.1).

Proof

It is not difficult to notice that the function U is lower semicontinuous, bounded and $U(T\!,x)\!\!=\!\!g(x)$. We will prove that the function U satisfies conditions (1.4). We consider five cases.

Case 1 Let $x>t$ and $t\in [0,T[$. Then $-p_t\geqslant -1/\varphi (t,x)$ and $-p_x =1/\varphi (t,x)$ for each $(p_t,p_x)\in \partial U(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)\geqslant 0$.

Case 2 Let $x>t$ and $t\in \,]0,T]$. Then $-p_t\leqslant -1/\varphi (t,x)$ and $-p_x =1/\varphi (t,x)$ for each $(p_t,p_x)\in \partial U(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)\leqslant 0$.

Case 3 Let $x=t$ and $t\in [0,T]$. Then $\partial U(t,x)=\emptyset$.

Case 4 Let $x<t$ and $t\in [0,T[$. Then $-p_t\geqslant 0$ and $-p_x =0$ for each $(p_t,p_x)\in \partial U(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)\geqslant 0$.

Case 5 Let $x<t$ and $t\in \,]0,T]$. Then $-p_t\leqslant 0$ and $-p_x =0$ for each $(p_t,p_x)\in \partial U(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)\leqslant 0$.

That finishes the proof. $\square$

4.2 Second Solution

Let the function $V:[0,T]\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ be given by the formula

$$\begin{aligned} V(t,x)=\left\{ \begin{array}{ccl} \,U(t,x) &{} \quad \mathrm{if}\,\, &{} x\geqslant t, \\ \,1-\exp \left( -2\sqrt{t-x}\,\right) &{} \quad \mathrm{if}\,\, &{} 2t-T\leqslant x< t,\\ \,1 &{} \quad \mathrm{if}\,\, &{} x<2t-T. \end{array} \right. \end{aligned}$$

(4.3)

Theorem 4.2

The function V given by (4.3) is bounded lower semicontinuous solution of the HJB equation with the Hamiltonian (3.2) and the terminal condition (4.1).

Proof

It is not difficult to notice that the function V is lower semicontinuous, bounded and $V(T,x)=g(x)$. We prove that the function V satisfies conditions (1.4). Since $\partial V(t,x)=\emptyset$ for $x=t$ and $V(t,x)=U(t,x)$ for $x\geqslant t$ and $x<2t-T$, then by the Theorem 4.1 it is sufficient to show that V satisfies conditions (1.4), when $2t-T\leqslant x< t$. To do it we consider two cases.

Case 1 Let $2t-T\leqslant x<t$ and $t\in \,]0,T[$. Then $-p_x\geqslant 1/\varphi (t,x)$ and $1/\varphi (t,x)= -p_t-2p_x$ for all $(p_t,p_x)\in \partial V(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)=0$.

Case 2 Let $2t-T\leqslant x<t$ and $t=0$. Then $-p_x\geqslant 1/\varphi (t,x)$ and $-p_t-2p_x\geqslant 1/\varphi (t,x)$ for all $(p_t,p_x)\in \partial V(t,x)$. By definition of H, we have $-p_t+H(t,x,-p_x)\geqslant 0$. $\square$

5 The Value Function

In this section we show that the function V given by (4.3) is the value function, in the sense of definition (1.2), corresponding to the Lagrangian (3.4) and the terminal condition (4.1). To this purpose, we use the methods of [15], whose scheme is as follows: We construct a sequence of Hamiltonians $(H_n)_{n\in {\mathbb {N}}}$ such that $H_n$ satisfy (H1)–(H3) together with (SLC) and $H_n\searrow H$. Then, by [15, Corollary 3.5], the value functions $V_n$ corresponding to HJB with Hamiltonians $H_n$ and terminal conditions $g_n=g$ converge to the value function V ($V_n\nearrow V$). Since Hamiltonians $H_n$ additionally, satisfy the condition (SLC), then by Theorem 2.1, the value functions $V_n$ are unique lower semicontinuous solutions. Therefore, to find the value function V, one needs to find solutions $U_n$ of equations HJB with Hamiltonians $H_n$ and the terminal condition g and then take the limit $U_n=V_n\rightarrow V$.

5.1 Approximation of the Hamiltonian

Let the function $\sigma :[0,+\infty [\rightarrow {\mathbb {R}}$ be given by $\sigma (z)=\sqrt{z}$, and functions $\sigma _n:[0,+\infty [\,\rightarrow \,{\mathbb {R}}$ by $\sigma _n(z)=\sqrt{z}$ if $z\geqslant 1/n$ and $\sigma _n(z)=1/\sqrt{n}$ if $0\leqslant z<1/n$. We notice that functions $\sigma _n$ satisfy locally the Lipschitz continuity and $\sigma _n\searrow \sigma$. Using functions $\sigma _n$ we define functions $\varphi _n:[0,T]\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ by

$$\begin{aligned} \varphi _n(t,x)=\sigma _n(|t-x|)\exp [2\sigma _n(|t-x|)]. \end{aligned}$$

Then functions $\varphi _n$ also satisfy locally the Lipschitz continuity and $\varphi _n\searrow \varphi$.

Hamiltonians $H_n:[0,T]\times {\mathbb {R}}\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ are defined by the formula

$$\begin{aligned} H_n(t,x,p)=\left\{ \begin{array}{ccl} \,0 &{} \quad \mathrm{if}\,\, &{} 2|p|\leqslant \frac{1}{\varphi _n(t,x)}, \\ \,2|p|-\frac{1}{\varphi _n(t,x)} &{} \quad \mathrm{if}\,\, &{} 2|p|>\frac{1}{\varphi _n(t,x)}. \end{array} \right. \end{aligned}$$

(5.1)

It is not difficult to notice that Hamiltonians $H_n$ given by (5.1) are continuous, convex with respect to p and have linear growth in p and x, so they satisfy conditions (H1)–(H3). Moreover $H_n\searrow H$, because $\varphi _n\searrow \varphi$. Similarly, for the proof of Case 2 in Theorem 3.1 we can prove that Hamiltonians (5.1) satisfy the condition (SLC).

5.2 Solutions $\pmb {U_n}$ of HJB with $\pmb {H_n}$

Let $U_{n}:[0,T]\times {\mathbb {R}}\rightarrow {\mathbb {R}}$ be given by formula:

                  $\pmb {U_n(t,x)=}$                         For $x\geqslant 2t-T$ we have

• (a) if $|t-x|\leqslant 1/n$ and $T-2t+x\geqslant 1/n$, then

  $$\begin{aligned} \,\,\,\,U_n(t,x)= & {} (t-x)\sqrt{n}\exp (-2/\sqrt{n}\,)+(1+1/\sqrt{n}\,) \exp (-2/\sqrt{n}\,)-1, \end{aligned}$$

• (b) if $|t-x|\leqslant 1/n$ and $T-2t+x\leqslant 1/n$, then

  $$\begin{aligned} \!\!U_n(t,x)= & {} \exp (-2\sqrt{T-2t+x}\,)+(T-t)\sqrt{n}\exp (-2/\sqrt{n}\,)-1, \end{aligned}$$

• (c) if $t-x\geqslant 1/n$ and $T-2t+x\geqslant 1/n$, then

  $$\begin{aligned} U_n(t,x)= & {} 2(1+1/\sqrt{n}\,)\exp (-2/\sqrt{n}\,)-\exp (-2\sqrt{t-x}\,)-1, \end{aligned}$$

• (d) if $t-x\geqslant 1/n$ and $T-2t+x\leqslant 1/n$, then

  $$\begin{aligned} U_n(t,x)=&[1+\sqrt{n}(T-2t+x)+1/\sqrt{n}\,]\exp (-2/\sqrt{n}\,)\\&\, +\exp (-2\sqrt{T-2t+x}\,)-\exp (-2\sqrt{t-x}\,)-1, \end{aligned}$$

• (e) if $1/n\leqslant x-t$, then $U_n(t,x)=\exp (-2\sqrt{x-t}\,)-1$.                                     For $x< 2t-T$ we have $U_n(t,x)=1$.

Proposition 5.1

Let Hamiltonians $H_n$ be given by (5.1), the terminal condition g by (4.1), and the function V by (4.3). Then functions $U_n$ given by the above formula are bounded lower semicontinuous solutions of equations HJB with Hamiltonians $H_n$ and the terminal condition g, moreover $U_n\rightarrow V$.

Proof

It is not difficult to notice that functions $U_n$ are bounded ( i.e. $-1\leqslant U_n(\,\cdot \,,\,\cdot \,)\leqslant 1$ ) and $U_n(T,x)=g(x)$. We can prove that functions $U_n$ are lower semicontinuous on $[0,T]\times {\mathbb {R}}$. Furthermore, functions $U_n$ are differentiable on $A=\{(t,x)\in \,]0,T[\times {\mathbb {R}}: x> 2t-T\}$ and $B=\{(t,x)\in \,]0,T[\times {\mathbb {R}}: x< 2t-T\}$, moreover $\partial U_n(t,x)=\emptyset$ for $x=2t-T$ and $t\in [0,T]$, in addition $U_n$ satisfy conditions (1.4) with Hamiltonians $H_n$ and $U_n\rightarrow V$. $\square$

6 Regularity of Hamiltonian

Let ${\mathcal {C}}$ be a family of continuous functions $\varphi :[0,T]\times {\mathbb {R}}\rightarrow [0,+\infty [$ that satisfy the condition $\varphi (t,x)=0\,\Leftrightarrow \, t=x$. Then it is easy to prove that the Hamiltonian H given by (3.2) with $\varphi \in {\mathcal {C}}$ is well-defined and satisfies (H1)–(H3). Next, by ${\mathcal {L}}$ we denote a subfamily of ${\mathcal {C}}$ that contains locally Lipschitz functions. Notice that $\varphi$ given by the formula (3.1) belongs to ${\mathcal {C}}$, but does not belong to ${\mathcal {L}}$. The example of a function, that is contained in ${\mathcal {L}},$ is $\varphi (t,x)=|t-x|$.

In this section we prove that the Lagrangian L given by (3.4) with $\varphi \in {\mathcal {L}}$ satisfies the Loewen–Rockafellar condition (2.2). However, it can be shown easily that its Hamiltonian does not satisfy the Lipschitz-type condition (SLC). It means that we cannot reverse the first implication in (2.3). Moreover, by the result of Galbraith [9] it follows that HJB equation with H given by (3.2) with $\varphi \in {\mathcal {L}}$ has the unique solution.

So, it could be said that the nonuniqueness of HJB is due to a particular choice of $\varphi \in {\mathcal {C}}$ and that choosing a different finction $\varphi \in {\mathcal {L}}\subset {\mathcal {C}}$ one can get the unique solution of HJB.

Let the function $\varphi$ belongs to ${\mathcal {C}}$ and $w(\,\cdot \,,r)$ be modulus of continuity of the $\varphi$ on the set $[0,T]\times r\mathbb {B}$. Then the following proposition holds.

Proposition 6.1

Let the Lagrangian L be given by (3.4) with the function $\varphi$ belongs to ${\mathcal {C}}$. Moreover, let $w(\,\cdot \,,r)$ be modulus of continuity of $\varphi$ . Then for every $t,s\in [0,T]$ and $x,y\in r\mathbb {B}$, every $v\in {\mathrm {dom}}\,L(t,x,\cdot )$ there exists $\nu \in {\mathrm {dom}}\,L(s,y,\cdot )$ such that

• (i) $|\nu -v|\leqslant 2(1+|v|+|L(t,x,v)|)\,w(|s-t|+|y-x|,r)$;

• (ii) $L(s,y,\nu )\leqslant L(t,x,v)+2(1+|v|+|L(t,x,v)|)\, w(|s-t|+|y-x|,r)$.

Proof

To prove the proposition we consider 3 cases.

Case 1 Let $t\not =x$ and $\varphi (s,y)/\varphi (t,x)\leqslant 1$. Then for $v\in {\mathrm {dom}}\,L(t,x,\cdot )$ we put $\nu =v\,\varphi (s,y)/\varphi (t,x)$. We notice that $\nu \in {\mathrm {dom}}\,L(s,y,\cdot )$. Moreover ${LS(ii)}\leqslant L(t,x,v)\leqslant {RS(ii)}$ and ${LS(i)}=2L(t,x,v)|\varphi (s,y)-\varphi (t,x)|\leqslant {RS(i)}$.

Case 2 Let $t\not =x$ and $\varphi (s,y)/\varphi (t,x)> 1$. Then for $v\in {\mathrm {dom}}\,L(t,x,\cdot )$ we put $\nu =v$. We notice that $\nu \in {\mathrm {dom}}\,L(s,y,\cdot )$. Moreover ${LS(i)}=0\leqslant {RS(i)}$ and ${LS(ii)}\leqslant L(t,x,v)\leqslant {RS(ii)}$.

Case 3 Let $t=x$. If $v\in {\mathrm {dom}}\,L(t,x,\cdot )$, then $v=0$. Put $\nu =0$. We notice that $\nu \in {\mathrm {dom}}\,L(s,y,\cdot )$. Moreover ${LS(i)}=0\leqslant {RS(i)}$ and ${LS(ii)}=0\leqslant {RS(ii)}$.

Therefore, the proposition is proven. $\square$

Proposition 6.2

The condition (2.2) holds, if the following condition is true:

• (A) For every $r>0$ there exists $k>0$ such that for every $t,s\in [0,T]$ and $x,y\in r\mathbb {B}$, every $v\in {\mathrm {dom}}\,L(t,x,\cdot )$ there exists $\nu \in {\mathrm {dom}}\,L(s,y,\cdot )$ such that

  • (i) $|\nu -v|\leqslant k(1+|v|+|L(t,x,v)|)(|s-t|+|y-x|)$;

  • (ii) $L(s,y,\nu )\leqslant L(t,x,v)+k(1+|v|+|L(t,x,v)|)(|s-t|+|y-x|)$.

Proof

We extend L in the following way: $L(t,x,v):=L(0,x,v)$ for $t<0$ and $L(t,x,v):=L(T,x,v)$ for $t>T$. Fix $r>0$ and choose $k>0$ for $1+r$ in such a way that the condition (A) holds. Let $t\in [0,T]$, $x\in r\mathbb {B}$, $v\in {\mathbb {R}}^n$ and $(w_1,w_2,p)\in \partial L(t,x,v)$. Without loss of generality we can assume that $(w_1,w_2)\not =0$. Let $(t_n,x_n):=(t,x)+(w_1,w_2)/[n|(w_1,w_2)|]$, then $x_n\in (1+r)\mathbb {B}\,$. Since $v\in {\mathrm {dom}}\,L(t,x,\cdot )$, then there exist $v_n\in {\mathrm {dom}}\,L(t_n,x_n,\cdot )$ such that

• (i) $|v_n-v|\leqslant 2k(1+|v|+|L(t,x,v)|)\,|(t_n,x_n)-(t,x)|$,

• (ii) $L(t_n,x_n,v_n)\leqslant L(t,x,v)+2k(1+|v|+|L(t,x,v)|)\, |(t_n,x_n)-(t,x)|$.

We put $b_n:=n(v_n-v)$ and notice that (i) implies that $|b_n|\leqslant 2k(1+|v|+|L(t,x,v)|)$. Therefore, a sequence $\{b_n\}_{n\in {\mathbb {N}}}$ is bounded, so there exists a subsequence (denoted again by) $b_n\rightarrow b$. Obviously, the following inequality is satisfied

$$\begin{aligned} |b|\leqslant 2k(1+|v|+|L(t,x,v)|). \end{aligned}$$

(6.1)

Since $(w_1,w_2,p)\in \partial L(t,x,v)$, then the property [19, s. 301] or [1, Chapter 6] and (ii) imply

$$\begin{aligned} \left\langle \,(w_1,w_2,p),\left( \frac{(w_1,w_2)}{|(w_1,w_2)|},b\right) \, \right\rangle\leqslant & {} d\,L(t,x,v)\left( \frac{(w_1,w_2)}{|(w_1,w_2)|}, b\right) \nonumber \\\leqslant & {} \liminf _{n}\frac{L(t_n,x_n,v_n)-L(t,x,v)}{|(t_n,x_n)-(t,x)|}\nonumber \\\leqslant & {} 2k(1+|v|+|L(t,x,v)|). \end{aligned}$$

(6.2)

From the inequality (6.2) and (6.1) for $(w_1,w_2,p)\in \partial L(t,x,v)$ we obtain

$$\begin{aligned} |(w_1,w_2)|\leqslant & {} 2k(1+|v|+|L(t,x,v)|)+|p||b|\\\leqslant & {} 2k(1+|v|+|L(t,x,v)|)(1+|p|). \end{aligned}$$

So the proposition is proven. $\square$

Remark 6.3

The only difference between the assertion of Proposition 6.1 and the condition (A) is contained in modulus. From Propositions 6.1 and 6.2 we obtain that the Lagrangian L given by (3.4) with $\varphi \in {\mathcal {L}}$ satisfies the Loewen–Rockafellar condition (2.2).