Introduction

Abstract

In this chapter we discuss turnpike properties and optimality criterions over infinite horizon for a class of convex dynamic optimization problems.

The study of optimal control problems and dynamic games defined on infinite intervals and on sufficiently large intervals has been a rapidly growing area of research [1, 4, 5, 9–16, 21, 23, 24, 26, 27, 29–33, 35, 36, 40, 41, 43, 44, 49–52, 54–56, 61] which has various applications in engineering [2, 19, 38, 70], in models of economic growth [3, 6, 17–20, 25, 28, 34, 37, 42, 48, 53, 59, 60, 62, 70, 79, 85, 94, 95], in infinite discrete models of solid-state physics related to dislocations in one-dimensional crystals [8, 57, 63], and in the theory of thermodynamical equilibrium for materials [22, 39, 45–47].

In this chapter we discuss turnpike properties and optimality criterions over infinite horizon for a class of convex dynamic optimization problems.

1.1 Convex Discrete-Time Problems

Let R ⁿ be the n-dimensional Euclidean space with the inner product 〈⋅ , ⋅ 〉 which induces the norm

$$\displaystyle{\vert x\vert = (\sum _{i=1}^{n}x_{ i}^{2})^{1/2},\;x = (x_{ 1},\ldots,x_{n}) \in R^{n}.}$$

Let K be a nonempty convex subset of R ⁿ. A function f: K → R ¹ is called convex (strictly convex respectively) if for all x, y ∈ K such that x ≠ y and all α ∈ (0, 1),

$$\displaystyle{f(\alpha x + (1-\alpha )y) \leq \alpha f(x) + (1-\alpha )f(y)}$$

$$\displaystyle{(f(\alpha x + (1-\alpha )y) <\alpha f(x) + (1-\alpha )f(y)}$$

respectively) [58].

Let v: R ⁿ × R ⁿ → R ¹ be a bounded from below function. We consider the minimization problem

$$\displaystyle{ \sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \min, }$$

(P0)

$$\displaystyle{\mbox{ such that }\{x_{i}\}_{i=0}^{T} \subset R^{n}\mbox{ and }x_{ 0} = z,\;x_{T} = y,}$$

where T is a natural number and the points y, z ∈ R ⁿ.

The interest in discrete-time optimal problems of type (P₀ ) stems from the study of various optimization problems which can be reduced to it, e.g., continuous-time control systems which are represented by ordinary differential equations whose cost integrand contains a discounting factor [37], tracking problems in engineering [2, 38], the study of Frenkel–Kontorova model [8, 63], and the analysis of a long slender bar of a polymeric material under tension in [22, 39, 45–47]. Optimization problems of the type ( P ₀ ) were considered in [64–66, 68, 69, 74].

In this section we suppose that the function v: R ⁿ × R ⁿ → R ¹ is strictly convex and differentiable and satisfies the growth condition

$$\displaystyle{ v(y,z)/(\vert y\vert + \vert z\vert ) \rightarrow \infty \mbox{ as }\vert y\vert + \vert z\vert \rightarrow \infty. }$$

(1.1)

We intend to analyze the structure of solutions of the problem ( P ₀ ) when the points y, z and the real number T vary and T is sufficiently large. More precisely, we are interested to study a turnpike property of solutions of ( P ₀ ) which is independent of the length of the interval T, for all sufficiently large intervals. To have this property means, roughly speaking, that solutions of the optimal control problems are determined mainly by the objective function v, and are essentially independent of T, y, and z. Turnpike properties are well known in mathematical economics (see, for example, [42, 48, 59, 60] and the references mentioned there). Many turnpike results are collected in [70, 81, 84, 93, 95].

In order to meet our goal we consider the auxiliary optimization problem

$$\displaystyle{ v(x,x) \rightarrow \min,\;x \in R^{n}. }$$

(P1)

In view of the strict convexity of v and (1.1), problem (P₁ ) has a unique solution $\bar{x}$. Let

$$\displaystyle{ \nabla v(\bar{x},\bar{x}) = (l_{1},l_{2}), }$$

(1.2)

where l ₁,  l ₂ ∈ R ⁿ. Since $\bar{x}$ is a solution of (P₁ ) it follows from (1.2) that for every h ∈ R ⁿ, we have

$$\displaystyle{\langle l_{1},h\rangle +\langle l_{2},h\rangle =\langle (l_{1},l_{2}),(h,h)\rangle }$$

$$\displaystyle{=\lim _{t\rightarrow 0^{+}}t^{-1}[v(\bar{x} + th,\bar{x} + th) - v(\bar{x},\bar{x})] \geq 0.}$$

Therefore

$$\displaystyle{\langle l_{1} + l_{2},h\rangle \geq 0\mbox{ for all }h \in R^{n},}$$

l ₂ = −l ₁ and

$$\displaystyle{ \nabla v(\bar{x},\bar{x}) = (l_{1},-l_{1}), }$$

(1.3)

For every (y, z) ∈ R ⁿ × R ⁿ define

$$\displaystyle{L(y,z) = v(y,z) - v(\bar{x},\bar{x}) -\langle \nabla v(\bar{x},\bar{x}),(y -\bar{ x},z -\bar{ x})\rangle }$$

$$\displaystyle{ = v(y,z) - v(\bar{x},\bar{x}) -\langle l_{1},y - z\rangle. }$$

(1.4)

It is easy to see that the function L: R ⁿ × R ⁿ → R ¹ is differentiable and strictly convex. By (1.1) and (1.4), we have

$$\displaystyle{ L(y,z)/(\vert y\vert + \vert z\vert ) \rightarrow \infty \mbox{ as }\vert y\vert + \vert z\vert \rightarrow \infty. }$$

(1.5)

Since the functions v and L are both strictly convex [58] it follows from (1.4) that

$$\displaystyle{ L(y,z) \geq 0\mbox{ for all }(y,z) \in R^{n} \times R^{n} }$$

(1.6)

and

$$\displaystyle{ L(y,z) = 0\mbox{ if and only if }y =\bar{ x},\;z =\bar{ x}. }$$

(1.7)

We show that the function L: R ⁿ × R ⁿ → R ¹ possesses the following property:

(C) If a sequence {(y _i , z _i )} _i = 1 ^∞ ⊂ R ⁿ × R ⁿ satisfies the equality

$$\displaystyle{\lim _{i\rightarrow \infty }L(y_{i},z_{i}) = 0,}$$

then

$$\displaystyle{\lim _{i\rightarrow \infty }(y_{i},z_{i}) = (\bar{x},\bar{x}).}$$

Assume that a sequence {(y _i , z _i )} _i = 1 ^∞ ⊂ R ⁿ × R ⁿ and that lim _{i → ∞} L(y _i , z _i ) = 0. It follows from (1.5) that the sequence {(y _i , z _i )} _i = 1 ^∞ is bounded. Let (y, z) be its limit point. Then it is not difficult to see that the equality

$$\displaystyle{L(y,z) =\lim _{i\rightarrow \infty }L(y_{i},z_{i}) = 0}$$

is true and in view of (1.7), we have $(y,z) = (\bar{x},\bar{x}).$ This implies that $(\bar{x},\bar{x}) =\lim _{i\rightarrow \infty }(y_{i},z_{i})$.

Therefore property (C) holds.

Consider an auxiliary minimization problem

$$\displaystyle{ \sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) \rightarrow \min, }$$

(P2)

$$\displaystyle{\mbox{ such that }\{x_{i}\}_{i=0}^{T} \subset R^{n}\mbox{ and }x_{ 0} = z,\;x_{T} = y,}$$

where T ≥ 1 is an integer and y, z ∈ R ⁿ.

By (1.4), for every natural number T and every sequence {x _i } _i = 0 ^T ⊂ R ⁿ,

$$\displaystyle{\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) =\sum _{ i=0}^{T-1}v(x_{ i},x_{i+1}) - Tv(\bar{x},\bar{x}) -\sum _{i=0}^{T-1}\langle l_{ 1},x_{i} - x_{i+1}\rangle }$$

$$\displaystyle{ =\sum _{ i=0}^{T-1}v(x_{ i},x_{i+1}) - Tv(\bar{x},\bar{x}) -\langle l_{1},x_{0} - x_{T}\rangle. }$$

(1.8)

It follows from (1.8) that problems ( P ₀ ) and (P₂ ) are equivalent. More precisely, {x _i } _i = 0 ^T ⊂ R ⁿ is a solution of problem ( P ₀ ) if and only if it is a solution of problem ( P ₂ ).

Let T ≥ 1 be an integer and Δ be a positive number. A sequence {x _i } _i = 0 ^T ⊂ R ⁿ is called (Δ)-optimal if for every sequence {x _i ′} _i = 0 ^T ⊂ R ⁿ which satisfies x _i  = x _i ′, i = 0, T the inequality

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \leq \sum _{i=0}^{T-1}v(x_{ i}',x_{i+1}')+\varDelta }$$

is valid. It is clear that if a sequence {x _i } _i = 0 ^T ⊂ R ⁿ is (0)-optimal, then it is a solution of problems (P₀ ) and (P₂ ) with z = x ₀ and y = x _T .

We prove the following existence result.

Proposition 1.1.

Let T > 1 be an integer and y,z ∈ R ⁿ . Then problem  (P₀ ) possesses a solution.

Proof.

In order to prove the proposition it is sufficient to show that problem ( P ₂ ) possesses a solution. Consider a sequence {x _i ′} _i = 0 ^T ⊂ R ⁿ which satisfies x ₀′ = z, x _T ′ = y. Put

$$\displaystyle{M_{1} =\sum _{ i=0}^{T-1}L(x_{ i}',x_{i+1}')}$$

and

$$\displaystyle{ M_{2} =\inf \{\sum _{ i=0}^{T-1}L(x_{ i},x_{i+1}):\;\{ x_{i}\}_{i=0}^{T} \subset R^{n},\;x_{ 0} = z,\;x_{T} = y\}. }$$

(1.9)

Evidently,

$$\displaystyle{0 \leq M_{2} \leq M_{1}.}$$

We may assume without loss of generality that

$$\displaystyle{ M_{2} <M_{1}. }$$

(1.10)

There exists a sequence {x _i ^(k)} _i = 0 ^T ⊂ R ⁿ, k = 1, 2, … such that for every integer k ≥ 1, we have

$$\displaystyle{ x_{0}^{(k)} = z,\;x_{ T}^{(k)} = y }$$

(1.11)

and

$$\displaystyle{ \lim _{k\rightarrow \infty }\sum _{i=0}^{T-1}L(x_{ i}^{(k)},x_{ i+1}^{(k)}) = M_{ 2}. }$$

(1.12)

By (1.10), (1.11), and (1.12), we may assume that

$$\displaystyle{ \sum _{i=0}^{T-1}L(x_{ i}^{(k)},x_{ i+1}^{(k)}) <M_{ 1}\mbox{ for all integers }k \geq 1. }$$

(1.13)

It follows from (1.13) and (1.5) that there exists a positive number M ₃ such that

$$\displaystyle{ \vert x_{i}^{(k)}\vert \leq M_{ 3}\mbox{ for all }i = 0,\ldots,T\mbox{ and all integers }k \geq 1. }$$

(1.14)

By (1.14), extracting subsequences, using diagonalization process and re-indexing, if necessary, we may assume without loss of generality that for every integer i ∈ { 0, …, T} there exists

$$\displaystyle{ \widehat{x}_{i} =\lim _{k\rightarrow \infty }x_{i}^{(k)}. }$$

(1.15)

In view of (1.15) and (1.11), we have

$$\displaystyle{ \widehat{x}_{0} = z,\;\widehat{x}_{T} = y. }$$

(1.16)

Relations (1.15) and (1.12) imply that

$$\displaystyle{\sum _{i=0}^{T-1}L(\widehat{x}_{ i},\widehat{x}_{i+1}) = M_{2}.}$$

Combined with (1.16) and (1.9) this implies that the finite sequence $\{\widehat{x}_{i}\}_{i=0}^{T}$ is a solution of problem (P₂ ). This completes the proof of Proposition 1.1.

Denote by Card(A) the cardinality of a set A.

The following result establishes a turnpike property for approximate solutions of the problem ( P ₀ ).

Proposition 1.2.

Let M ₁ ,M ₂ ,ε > 0. Then there exists an integer k ₀ ≥ 1 such that for every natural number T > 1 and every (M ₁ )-optimal sequence {x _i } _i=0 ^T ⊂ R ⁿ which satisfies

$$\displaystyle{ \vert x_{0}\vert \leq M_{2},\;\vert x_{T}\vert \leq M_{2} }$$

(1.17)

the inequality

$$\displaystyle{\mbox{ Card}(\{i \in \{ 0,\ldots,T - 1\}:\; \vert x_{i} -\bar{ x}\vert + \vert x_{i+1} -\bar{ x}\vert>\epsilon \}) \leq k_{0}}$$

is valid.

Proof.

Condition (C) implies that there exists a positive number δ such that for every point (y, z) ∈ R ⁿ × R ⁿ which satisfies

$$\displaystyle{ L(y,z) \leq \delta }$$

(1.18)

the inequality

$$\displaystyle{ \vert y -\bar{ x}\vert + \vert z -\bar{ x}\vert \leq \epsilon }$$

(1.19)

is true. Set

$$\displaystyle{ M_{3} =\sup \{ L(y,z):\; y,z \in R^{n}\mbox{ and }\vert y\vert + \vert z\vert \leq \vert \bar{x}\vert + M_{ 2}\} }$$

(1.20)

and fix a natural number

$$\displaystyle{ k_{0}>\delta ^{-1}(M_{ 1} + 2M_{3}). }$$

(1.21)

Assume that an integer T > 1 and that an (M ₁)-optimal sequence {x _i } _i = 0 ^T ⊂ R ⁿ satisfies (1.17). Define

$$\displaystyle{ y_{0} = x_{0},\;y_{T} = x_{T},\;y_{i} =\bar{ x},\;i = 1,\ldots,T - 1. }$$

(1.22)

Since the sequence {x _i } _i = 0 ^T is (M ₁)-optimal it follows from (1.22) that

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \leq \sum _{i=0}^{T-1}v(y_{ i},y_{i+1}) + M_{1}.}$$

Combined with (1.7), (1.8), and (1.22) this implies that

$$\displaystyle{\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) \leq \sum _{i=0}^{T-1}L(y_{ i},y_{i+1}) + M_{1} = L(x_{0},\bar{x}) + L(\bar{x},x_{T}) + M_{1}.}$$

Together with (1.17) and (1.20) this implies that

$$\displaystyle{\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) \leq M_{1} + 2M_{3}.}$$

In view of the choice of δ (see (1.18) and (1.19)), (1.21) and the inequality above, we have

$$\displaystyle{\mbox{ Card}(\{i \in \{ 0,\ldots,T - 1\}:\; \vert x_{i} -\bar{ x}\vert + \vert x_{i+1} -\bar{ x}\vert>\epsilon \})}$$

$$\displaystyle{\leq \mbox{ Card}(\{i \in \{ 0,\ldots,T - 1\}:\; L(x_{i},x_{i+1})>\delta \})}$$

$$\displaystyle{\leq \delta ^{-1}\sum _{ i=0}^{T-1}L(x_{ i},x_{i+1}) \leq \delta ^{-1}(M_{ 1} + 2M_{3}) \leq k_{0}.}$$

Proposition 1.2 is proved.

Proposition 1.2 implies the following turnpike result for exact solutions of the problem ( P ₀ ).

Proposition 1.3.

Let M,ε > 0. Then there exists an integer k ₀ ≥ 1 such that for every natural number T > 1, every pair of points y,z ∈ R ⁿ which satisfies |y|, |z|≤ M and every optimal solution {x _i } _i=0 ^T ⊂ R ⁿ of problem  (P₀ ) the inequality

$$\displaystyle{\mbox{ Card}(\{i \in \{ 0,\ldots,T - 1\}:\; \vert x_{i} -\bar{ x}\vert + \vert x_{i+1} -\bar{ x}\vert>\epsilon \}) \leq k_{0}}$$

is valid.

Now it is clear that the optimal solution {x _i } _i = 0 ^T of problem ( P ₀ ) spends most of the time in an ε-neighborhood of the point $\bar{x}$. In view of Proposition 1.3, the number of all integers i ∈ { 0, …, T − 1} for which that x _i does not belong to this ε-neighborhood, does not exceed the constant k ₀ which depends only on M, ε and does not depend on T. Following the tradition, the point $\bar{x}$ is called the turnpike. Moreover we can show that the set

$$\displaystyle{\{i \in \{ 0\ldots,T\}: \vert x_{i} -\bar{ x}\vert>\epsilon \}}$$

is contained in the union of two intervals [0, k ₁] ∪ [T − k ₁, T], where k ₁ is a constant depending only on M, ε.

We also study the infinite horizon problem associated with problem ( P ₀ ).

By (1.1) there is M _∗ > 0 such that

$$\displaystyle{ v(y,z)> \vert v(\bar{x},\bar{x})\vert + 1 }$$

(1.23)

$$\displaystyle{\mbox{ for any }(y,z) \in R^{n} \times R^{n}\mbox{ satisfying }\vert y\vert + \vert z\vert \geq M_{ {\ast}}.}$$

We suppose that the sum over empty set is zero.

Proposition 1.4.

Let M ₀ be a positive number. Then there exists a positive number M ₁ such that for every natural number T and every finite sequence {x _i } _i=0 ^T ⊂ R ⁿ which satisfies |x ₀ |≤ M ₀ the inequality

$$\displaystyle{ \sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \geq Tv(\bar{x},\bar{x}) - M_{1} }$$

(1.24)

holds.

Proof.

Set

$$\displaystyle{M_{1} = \vert l_{1}\vert (M_{0} + M_{{\ast}}).}$$

Assume that T is a natural number and a that a finite sequence {x _i } _i = 0 ^T ⊂ R ⁿ satisfies

$$\displaystyle{ \vert x_{0}\vert \leq M_{0}. }$$

(1.25)

If | x _i  | > M _∗, i = 1, …, T, then in view of (1.23), we have

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \geq Tv(\bar{x},\bar{x})}$$

and inequality (1.24) is true. Therefore we may assume that there exists an integer q ≥ 1 for which

$$\displaystyle{ q \leq T,\;\vert x_{q}\vert \leq M_{{\ast}}. }$$

(1.26)

We may assume without loss of generality that

$$\displaystyle{ \vert x_{i}\vert> M_{{\ast}}\mbox{ for all integers }i\mbox{ satisfying }q <i \leq T. }$$

(1.27)

In view of (1.23) and (1.27), we have

$$\displaystyle{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) =\sum _{ i=0}^{q-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))}$$

$$\displaystyle{+\sum \{v(x_{i},x_{i+1}) - v(\bar{x},\bar{x})):\; \mbox{ an integer }i\mbox{ satisfies }q \leq i <T\}}$$

$$\displaystyle{\geq \sum _{i=0}^{q-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})).}$$

By the equation above, (1.8), (1.6), (1.25), (1.26) and the choice of M ₁, we have

$$\displaystyle{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) \geq \sum _{i=0}^{q-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))}$$

$$\displaystyle{=\sum _{ i=0}^{q-1}L(x_{ i},x_{i+1}) +\langle l_{1},x_{0} - x_{q}\rangle \geq -\vert l_{1}\vert (\vert x_{0}\vert + \vert x_{q}\vert )}$$

$$\displaystyle{\geq -\vert l_{1}\vert (M_{0} + M_{{\ast}}) = -M_{1}.}$$

This completes the proof of Proposition 1.4.

Choose a positive number $\tilde{M}$ for which

$$\displaystyle{ \mbox{ Proposition <InternalRef RefID="FPar4">1.4</InternalRef> holds with }M_{0} = M_{{\ast}}\mbox{ and }M_{1} =\tilde{ M}. }$$

(1.28)

Proposition 1.5.

Let {x _i } _i=0 ^∞ ⊂ R ⁿ . Then either the sequence

$$\displaystyle{\{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))\}_{T=1}^{\infty }}$$

is bounded or

$$\displaystyle{ \lim _{T\rightarrow \infty }\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) = \infty. }$$

(1.29)

Proof.

In view of (1.23), if for all sufficiently large natural numbers i, we have | x _i  | ≥ M _∗, then equality (1.29) is valid. Therefore we may assume without loss of generality that there exists a strictly increasing sequence of natural numbers {t _k } _k = 1 ^∞ such that

$$\displaystyle{ \vert x_{t_{k}}\vert <M_{{\ast}}\mbox{ for all integers }k \geq 1. }$$

(1.30)

Proposition 1.4 implies that the sequence $\{\sum _{i=0}^{T-1}(v(x_{i},x_{i+1}) - v(\bar{x},\bar{x}))\}_{T=1}^{\infty }$ is bounded from below.

Assume that this sequence is not bounded from above. In order to complete the proof it is sufficient to show that equality (1.29) is valid.

Let Q > 0 be given. Then there exists an integer T ₀ ≥ 1 for which

$$\displaystyle{ \sum _{i=0}^{T_{0}-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))> Q +\tilde{ M}. }$$

(1.31)

Fix an integer k ≥ 1 such that

$$\displaystyle{ t_{k}> T_{0} + 4. }$$

(1.32)

Let an integer

$$\displaystyle{ T> t_{k}. }$$

(1.33)

In view of (1.30), (1.32), and (1.33), there exists an integer S such that

$$\displaystyle{ T> S \geq T_{0}, }$$

(1.34)

$$\displaystyle{ \vert x_{S}\vert \leq M_{{\ast}}, }$$

(1.35)

$$\displaystyle{ \vert x_{t}\vert> M_{{\ast}}\mbox{ for all integers }t\mbox{ satisfying } }$$

(1.36)

$$\displaystyle{S> t \geq T_{0}.}$$

By (1.31), (1.34), (1.36), (1.23), (1.35), (1.28), and Proposition 1.4, we have

$$\displaystyle{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) =\sum _{ i=0}^{T_{0}-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))}$$

$$\displaystyle{+\sum \{v(x_{i},x_{i+1}) - v(\bar{x},\bar{x}):\; i\mbox{ is an integer and }T_{0} \leq i <S\}}$$

$$\displaystyle{+\sum _{i=S}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))}$$

$$\displaystyle{> Q +\tilde{ M} +\sum _{ i=S}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))> Q.}$$

Thus for any integer T > t _k ,

$$\displaystyle{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x}))> Q.}$$

Since Q is any positive number (1.29) is valid. This completes the proof of Proposition 1.5.

A sequence {x _i } _i = 0 ^∞ ⊂ R ⁿ is called good [28, 70, 81] if the sequence $\{\sum _{i=0}^{T-1}(v(x_{i},x_{i+1}) - v(\bar{x},\bar{x}))\}_{T=1}^{\infty }$ is bounded.

Proposition 1.6.

1. A sequence {x _i } _i=0 ^∞ ⊂ R ⁿ is good if and only if

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) <\infty.}$$

2. If a sequence {x _i } _i=0 ^∞ ⊂ R ⁿ is good, then it converges to $\bar{x}$ .

Proof.

Assume that a sequence {x _i } _i = 0 ^∞ ⊂ R ⁿ is good. Then there exists a positive number M ₀ such that

$$\displaystyle{ \sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) <M_{0}\mbox{ for all integers }T \geq 1. }$$

(1.37)

In view of (1.37) and (1.23), there exists a strictly increasing sequence of natural numbers {t _k } _k = 1 ^∞ such that

$$\displaystyle{ \vert x_{t_{k}}\vert <M_{{\ast}}\mbox{ for all natural numbers }k. }$$

(1.38)

Let k ≥ 1 be an integer. It follows from (1.8), (1.37), and (1.38) that

$$\displaystyle{M_{0}>\sum _{ i=0}^{t_{k}-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) =\sum _{ i=0}^{t_{k}-1}L(x_{ i},x_{i+1}) +\langle l_{1},x_{0} - x_{t_{k}}\rangle }$$

$$\displaystyle{\geq \sum _{i=0}^{t_{k}-1}L(x_{ i},x_{i+1}) -\vert l_{1}\vert (\vert x_{0}\vert + \vert x_{t_{k}}\vert )}$$

$$\displaystyle{\geq \sum _{i=0}^{t_{k}-1}L(x_{ i},x_{i+1}) -\vert l_{1}\vert (\vert x_{0}\vert + M_{{\ast}})}$$

and

$$\displaystyle{\sum _{i=0}^{t_{k}-1}L(x_{ i},x_{i+1}) \leq M_{0} + \vert l_{1}\vert (\vert x_{0}\vert + M_{{\ast}}).}$$

Since the inequality above is valid for all natural numbers k we conclude that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) \leq M_{0} + \vert l_{1}\vert (\vert x_{0}\vert + M_{{\ast}}).}$$

Property (C) implies that the sequence {x _i } _i = 0 ^∞ converges to $\bar{x}$ and Assertion 2 is proved.

Assume that

$$\displaystyle{ M_{1}:=\sum _{ i=0}^{\infty }L(x_{ i},x_{i+1}) <\infty. }$$

(1.39)

In view of (1.5), there exists a positive number M ₂ such that

$$\displaystyle{ \vert x_{i}\vert <M_{2}\mbox{ for all integers }i \geq 0. }$$

(1.40)

By (1.8), (1.39), and (1.40), for all integers T ≥ 1, we have

$$\displaystyle{\sum _{i=0}^{T-1}(v(x_{ i},x_{i+1}) - v(\bar{x},\bar{x})) =\sum _{ i=0}^{T-1}L(x_{ i},x_{i+1}) +\langle l_{1},x_{0} - x_{T}\rangle }$$

$$\displaystyle{\leq M_{1} + 2\vert l_{1}\vert M_{2}.}$$

Combined with Proposition 1.5 this implies that the sequence {x _i } _i = 0 ^∞ is good and completes the proof of Proposition 1.6.

Proposition 1.7.

Let x ∈ R ⁿ . Then there exists a sequence {x _i } _i=0 ^∞ ⊂ R ⁿ such that x ₀ = x and for each sequence {y _i } _i=0 ^∞ ⊂ R ⁿ satisfying y ₀ = x the inequality

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) \leq \sum _{i=0}^{\infty }L(y_{ i},y_{i+1})}$$

holds.

Proof.

Set

$$\displaystyle{ M_{0} =\inf \{\sum _{ i=0}^{\infty }L(y_{ i},y_{i+1}):\;\{ y_{i}\}_{i=0}^{\infty }\subset R^{n}\mbox{ and }y_{ 0} = x\}. }$$

(1.41)

It is clear that M ₀ is well defined and nonnegative. There exists a sequence {x _i ^(k)} _i = 0 ^∞ ⊂ R ⁿ, k = 1, 2, … such that

$$\displaystyle{ x_{0}^{(k)} = x,\;k = 1,2,\ldots, }$$

(1.42)

$$\displaystyle{ \lim _{k\rightarrow \infty }\sum _{i=0}^{\infty }L(x_{ i}^{(k)},x_{ i+1}^{(k)}) = M_{ 0}. }$$

(1.43)

In view of (1.43) and (1.5), there exists a positive number M ₁ such that

$$\displaystyle{ \vert x_{i}^{(k)}\vert <M_{ 1}\mbox{ for all integers }i \geq 0\mbox{ for all integers }k \geq 1. }$$

(1.44)

By (1.44) using diagonalization process, extracting subsequences and re-indexing we may assume without loss of generality that for every nonnegative integer i there exists

$$\displaystyle{ x_{i} =\lim _{k\rightarrow \infty }x_{i}^{(k)}. }$$

(1.45)

It follows from (1.42) and (1.45) that

$$\displaystyle{ x_{0} = x. }$$

(1.46)

In view of (1.6), (1.43), and (1.45), for every integer T ≥ 1, we have

$$\displaystyle{\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) =\lim _{k\rightarrow \infty }\sum _{i=0}^{T-1}L(x_{ i}^{(k)},x_{ i+1}^{(k)}) \leq \lim _{ k\rightarrow \infty }\sum _{i=0}^{\infty }L(x_{ i}^{(k)},x_{ i+1}^{(k)}) = M_{ 0}.}$$

Since T is an arbitrary natural number we conclude that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) \leq M_{0}.}$$

Combined with (1.41) and (1.46) this implies that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) = M_{0}.}$$

Proposition 1.7 is proved.

In our study we use the following optimality criterion introduced in the economic literature [6, 28, 62] and used in the optimal control [19, 70, 81, 84].

A sequence {x _i } _i = 0 ^∞ ⊂ R ⁿ is called overtaking optimal if

$$\displaystyle{\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) -\sum _{i=0}^{T-1}v(y_{ i},y_{i+1})] \leq 0}$$

for any sequence {y _i } _i = 0 ^∞ ⊂ R ⁿ satisfying y ₀ = x ₀.

Proposition 1.8.

Let {x _i } _i=0 ^∞ ⊂ R ⁿ . Then the following assertions are equivalent:

1. the sequence {x _i } _i=0 ^∞ is overtaking optimal;

2.

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) \leq \sum _{i=0}^{\infty }L(y_{ i},y_{i+1})}$$

for every sequence {y _i } _i=0 ^∞ ⊂ R ⁿ which satisfies y ₀ = x ₀ .

Proof.

Assume that the sequence {x _i } _i = 0 ^∞ is overtaking optimal. Evidently, it is good. Proposition 1.6 implies that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) <\infty.}$$

Let a sequence {y _i } _i = 0 ^∞ ⊂ R ⁿ satisfy

$$\displaystyle{ y_{0} = x_{0}. }$$

(1.47)

We claim that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) \leq \sum _{i=0}^{\infty }L(y_{ i},y_{i+1}).}$$

We may assume that

$$\displaystyle{\sum _{i=0}^{\infty }L(y_{ i},y_{i+1}) <\infty.}$$

Property (C) implies that

$$\displaystyle{ \lim _{i\rightarrow \infty }y_{i} =\bar{ x},\;\lim _{i\rightarrow \infty }x_{i} =\bar{ x}. }$$

(1.48)

Since the sequence {x _i } _i = 0 ^∞ is overtaking optimal it follows from (1.47), (1.8), and (1.48) that

$$\displaystyle{0 \geq \limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) -\sum _{i=0}^{T-1}v(y_{ i},y_{i+1})]}$$

$$\displaystyle{=\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) +\langle l_{1},x_{0} - x_{T}\rangle -\sum _{i=0}^{T-1}L(y_{ i},y_{i+1}) -\langle l_{1},y_{0} - y_{T}\rangle ]}$$

$$\displaystyle{=\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) -\sum _{i=0}^{T-1}L(y_{ i},y_{i+1}) +\langle l_{1},y_{T} - x_{T}\rangle ]}$$

$$\displaystyle{=\sum _{ i=0}^{\infty }L(x_{ i},x_{i+1}) -\sum _{i=0}^{\infty }L(y_{ i},y_{i+1}).}$$

Thus assertion 2 holds.

Assume that assertion 2 holds. We claim that the sequence {x _i } _i = 0 ^∞ is overtaking optimal. It is clear that

$$\displaystyle{\sum _{i=0}^{\infty }L(x_{ i},x_{i+1}) <\infty.}$$

Proposition 1.6 implies that the sequence {x _i } _i = 0 ^∞ is good and that

$$\displaystyle{ \lim _{i\rightarrow \infty }x_{i} =\bar{ x}. }$$

(1.49)

Assume that a sequence {y _i } _i = 0 ^∞ ⊂ R ⁿ satisfies

$$\displaystyle{ y_{0} = x_{0}. }$$

(1.50)

We show that

$$\displaystyle{\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) -\sum _{i=0}^{T-1}v(y_{ i},y_{i+1})] \leq 0.}$$

We may assume without loss of generality that the sequence {y _i } _i = 0 ^∞ is good. In view of Proposition 1.6, we have

$$\displaystyle{ \lim _{i\rightarrow \infty }y_{i} =\bar{ x},\;\sum _{i=0}^{\infty }L(y_{ i},y_{i+1}) <\infty. }$$

(1.51)

In view of (1.8), (1.49), (1.50), (1.51), and assertion 2, we have

$$\displaystyle{\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) -\sum _{i=0}^{T-1}v(y_{ i},y_{i+1})]}$$

$$\displaystyle{=\limsup _{T\rightarrow \infty }[\sum _{i=0}^{T-1}L(x_{ i},x_{i+1}) +\langle l_{1},x_{0} - x_{T}\rangle -\sum _{i=0}^{T-1}L(y_{ i},y_{i+1}) -\langle l_{1},y_{0} - y_{T}\rangle ]}$$

$$\displaystyle{=\sum _{ i=0}^{\infty }L(x_{ i},x_{i+1}) -\sum _{i=0}^{\infty }L(y_{ i},y_{i+1}) +\langle l_{1},\lim _{T\rightarrow \infty }y_{T} -\lim _{T\rightarrow \infty }x_{T}\rangle ]}$$

$$\displaystyle{=\sum _{ i=0}^{\infty }L(x_{ i},x_{i+1}) -\sum _{i=0}^{\infty }L(y_{ i},y_{i+1}) \leq 0.}$$

Thus assertion 1 holds and Proposition 1.8 is proved.

Propositions 1.7 and 1.8 imply the following existence result.

Proposition 1.9.

For every point x ∈ R ⁿ there exists an overtaking optimal sequence {x _i } _i=0 ^∞ ⊂ R ⁿ such that x ₀ = x.

1.2 The Turnpike Phenomenon

In the previous section we proved the turnpike result and the existence of overtaking optimal solutions for rather simple class of discrete-time problems. The problems of this class are unconstrained and their objective functions are convex and differentiable. In this book our goal is to study the structure of approximate solutions over large intervals for a class of discrete-time constrained optimal control problems without convexity (concavity) assumptions. In particular, in Chaps. 2–4 we study the structure of approximate solutions of an autonomous discrete-time control system with a compact metric space of states X. This control system is described by a bounded upper semicontinuous function v: X × X → R ¹ which determines an optimality criterion and by a nonempty closed set Ω ⊂ X × X which determines a class of admissible trajectories (programs). We study the problems

$$\displaystyle{ \sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega,\;x_{ 0} = z,\;x_{T} = y, }$$

(P1)

$$\displaystyle{ \sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega,\;x_{ 0} = z }$$

(P2)

and

$$\displaystyle{ \sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega, }$$

(P3)

where T ≥ 1 is an integer and the points y, z ∈ X.

In the classical turnpike theory the objective function v possesses the turnpike property (TP) if there exists a point $\bar{x} \in X$ (a turnpike) such that the following condition holds:

For each positive number ε there exists an integer L ≥ 1 such that for each integer T ≥ 2L and each solution {x _i } _i = 0 ^T ⊂ X of the problem (P1) the inequality $\rho (x_{i},\bar{x}) \leq \epsilon$ is true for all i = L, …, T − L.

It should be mentioned that the constant L depends neither on T nor on y, z.

The turnpike phenomenon has the following interpretation. If one wishes to reach a point A from a point B by a car in an optimal way, then one should turn to a turnpike, spend most of time on it, and then leave the turnpike to reach the required point.

In the classical turnpike theory [28, 48, 59, 62] the space X is a compact convex subset of a finite-dimensional Euclidean space, the set Ω is convex, and the function v is strictly concave. Under these assumptions the turnpike property can be established and the turnpike $\bar{x}$ is a unique solution of the maximization problem v(x, x) → max, (x, x) ∈ Ω. In this situation it is shown that for each program {x _t } _t = 0 ^∞ either the sequence $\{\sum _{t=0}^{T-1}v(x_{t},x_{t+1}) - Tv(\bar{x},\bar{x})\}_{T=1}^{\infty }$ is bounded (in this case the program {x _t } _t = 0 ^∞ is called (v)-good) or it diverges to −∞. Moreover, it is also established that any (v)-good program converges to the turnpike $\bar{x}$. In the sequel this property is called as the asymptotic turnpike property.

Recently it was shown that the turnpike property is a general phenomenon which holds for large classes of variational and optimal control problems without convexity assumptions. (See, for example, [70] and the references mentioned therein). For these classes of problems a turnpike is not necessarily a singleton but may instead be a nonstationary trajectory (in the discrete time nonautonomous case) or an absolutely continuous function on the interval [0, ∞) (in the continuous time nonautonomous case) or a compact subset of the space X (in the autonomous case). Note that all of these results were obtained for unconstrained problems. In particular, the turnpike results for the problems of the type (P1) were obtained in the case Ω = X × X.

For classes of problems considered in [70], using the Baire category approach, it was shown that the turnpike property holds for a generic (typical) problem. In this book we are interested in individual (non-generic) results describing the structure of approximate solutions. We study the problems (P1)–(P3) with the constraint {(x _i , x _i+1)} _i = 0 ^T−1 ⊂ Ω where Ω is an arbitrary nonempty closed subset of X × X. Clearly, these constrained problems are more difficult and less understood than their unconstrained prototypes in the previous section and in [64–66, 68]. They are also more realistic from the point of view of mathematical economics. As we have mentioned before in general a turnpike is not necessarily a singleton. Nevertheless problems of the type (P1)–(P3) for which the turnpike is a singleton are of great importance because of the following reasons: there are many models of economic growth for which a turnpike is a singleton; if a turnpike is a singleton, then approximate solutions have very simple structure and this is very important for applications; if a turnpike is a singleton, then it can be easily calculated as a solution of the problem v(x, x) → max, (x, x) ∈ Ω.

The turnpike property is very important for applications. Suppose that our objective function v has the turnpike property and we know a finite number of “approximate” solutions of the problem (P1). Then we know the turnpike $\bar{x}$, or at least its approximation, and the constant L (see the definition of (TP)) which is an estimate for the time period required to reach the turnpike. This information can be useful if we need to find an “approximate” solution of the problem (P1) with a new time interval [m ₁, m ₂] and the new values z, y ∈ X at the end points m ₁ and m ₂. Namely instead of solving this new problem on the “large” interval [m ₁, m ₂] we can find an “approximate” solution of the problem (P1) on the “small” interval [m ₁, m ₁ + L] with the values $z,\bar{x}$ at the end points and an “approximate” solution of the problem (P1) on the “small” interval [m ₂ − L, m ₂] with the values $\bar{x},y$ at the end points. Then the concatenation of the first solution, the constant sequence $x_{i} =\bar{ x}$, i = m ₁ + L, …, m ₂ − L and the second solution is an “approximate” solution of the problem (P1) on the interval [m ₁, m ₂] with the values z, y at the end points. Sometimes as an “approximate” solution of the problem (P1) we can choose any admissible sequence $\{x_{i}\}_{i=m_{1}}^{m_{2}}$ satisfying

$$\displaystyle{x_{m_{1}} = z,\;x_{m_{2}} = y\mbox{ and }x_{i} =\bar{ x}\mbox{ for all }i = m_{1} + L,\ldots,m_{2} - L.}$$

1.3 Nonconcave (Nonconvex) Problems

In Chap. 2 we study the structure of approximate solutions of discrete-time optimal control problems introduced in Sect. 1.2.

Let (X, ρ) be a compact metric space, Ω be a nonempty closed subset of X × X, and let v: X × X → R ¹ be a bounded upper semicontinuous function.

A sequence {x _t } _t = 0 ^∞ ⊂ X is called an (Ω)-program (or just a program if the set Ω is understood) if (x _t , x _t+1) ∈ Ω for all nonnegative integers t. A sequence {x _t } _t = 0 ^T where T ≥ 1 is an integer is called an (Ω)-program (or just a program if the set Ω is understood) if (x _t , x _t+1) ∈ Ω for all integers t ∈ [0, T − 1].

We consider the problems

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega,\;x_{ 0} = y,\;x_{T} = z,}$$

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega,\;x_{ 0} = z}$$

and

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \rightarrow \max,\;\{(x_{i},x_{i+1})\}_{i=0}^{T-1} \subset \varOmega,}$$

where T ≥ 1 is an integer and the points y, z ∈ X.

We suppose that there exist a point $\bar{x} \in X$ and a positive number $\bar{c}$ such that the following assumptions hold:

(i) $(\bar{x},\bar{x})$ is an interior point of Ω;

(ii) $\sum _{t=0}^{T-1}v(x_{t},x_{t+1}) \leq Tv(\bar{x},\bar{x}) +\bar{ c}$ for any natural number T and any program {x _t } _t = 0 ^T.

The property (ii) implies that for each program {x _t } _t = 0 ^∞ either the sequence

$$\displaystyle{\{\sum _{t=0}^{T-1}v(x_{ t},x_{t+1}) - Tv(\bar{x},\bar{x})\}_{T=1}^{\infty }}$$

is bounded or $\lim _{T\rightarrow \infty }[\sum _{t=0}^{T-1}v(x_{t},x_{t+1}) - Tv(\bar{x},\bar{x})] = -\infty.$

A program {x _t } _t = 0 ^∞ is called (v)-good if the sequence

$$\displaystyle{\{\sum _{t=0}^{T-1}v(x_{ t},x_{t+1}) - Tv(\bar{x},\bar{x})\}_{T=1}^{\infty }}$$

is bounded.

We suppose that the following assumption holds.

(iii) (the asymptotic turnpike property) For any (v)-good program {x _t } _t = 0 ^∞, $\lim _{t\rightarrow \infty }\rho (x_{t},\bar{x}) = 0$.

Note that the properties (i)–(iii) hold for models of economic dynamics considered in the classical turnpike theory.

For each positive number M denote by X _M the set of all points x ∈ X for which there exists a program {x _t } _t = 0 ^∞ such that x ₀ = x and that for all natural numbers T the following inequality holds:

$$\displaystyle{\sum _{t=0}^{T-1}v(x_{ t},x_{t+1}) - Tv(\bar{x},\bar{x}) \geq -M.}$$

It is not difficult to see that ∪{ X _M :   M ∈ (0, ∞)} is the set of all points x ∈ X for which there exists a (v)-good program {x _t } _t = 0 ^∞ satisfying x ₀ = x.

Let T ≥ 1 be an integer and Δ ≥ 0. A program {x _i } _i = 0 ^T ⊂ X is called (Δ)-optimal if for any program {x _i ′} _i = 0 ^T satisfying x ₀ = x ₀′, the inequality

$$\displaystyle{\sum _{i=0}^{T-1}v(x_{ i},x_{i+1}) \geq \sum _{i=0}^{T-1}v(x_{ i}',x_{i+1}')-\varDelta }$$

holds.

The turnpike theory for problems (P1) and (P2) is presented in [84] which summaries our research [71–73, 75–77].

In particular, in Chap. 2 of [84] we prove the following turnpike result for approximate solutions of our second optimization problem stated above.

Theorem 1.10.

Let ε,M be positive numbers. Then there exist a natural number L and a positive number δ such that for each integer T > 2L and each (δ)-optimal program {x _t } _t=0 ^T which satisfies x ₀ ∈ X _M there exist nonnegative integers τ ₁ ,τ ₂ ≤ L such that $\rho (x_{t},\bar{x}) \leq \epsilon$ for all t = τ ₁ ,…,T −τ ₂ and if $\rho (x_{0},\bar{x}) \leq \delta$ , then τ ₁ = 0.

An analogous turnpike result for approximate solutions of our first optimization problem is also proved in Chap. 2 of [84].

A program {x _t } _t = 0 ^∞ is called (v)-overtaking optimal if for each program {y _t } _t = 0 ^∞ satisfying y ₀ = x ₀ the inequality limsup _{T → ∞} ∑ _t = 0 ^T−1[v(y _t , y _t+1) − v(x _t , x _t+1)] ≤ 0 holds.

In Chap. 2 of [84] we prove the following result which establishes the existence of an overtaking optimal program.

Theorem 1.11.

Assume that x ∈ X and that there exists a (v)-good program {x _t } _t=0 ^∞ such that x ₀ = x. Then there exists a (v)-overtaking optimal program {x _t ^∗ } _t=0 ^∞ such that x ₀ ^∗ = x.

In Chap. 2 of [84] for problems which satisfy concavity assumption common in the literature we study the structure of approximate solutions in the regions containing end points and obtain a full description of the structure of approximate solutions. More precisely, we study the structure of approximate solutions of our second optimization problem stated above in the regions [0, L] and [T − L, T] (see the definition of the turnpike property). We show that if {x _i } _i = 0 ^T ⊂ X is an approximate solution of our problem, then for all integers t = 0, …, L the state x _t is closed enough to z _t where {z _t } _t = 0 ^∞ ⊂ X is a unique solution of a certain infinite horizon optimal control problem satisfying z ₀ = z. We also show that if {x _i } _i = 0 ^T ⊂ X is an approximate solution of our second optimization problem, then for all integers t = 0, …, L the state x _T−t is closed enough to Λ _t where {Λ _t } _t = 0 ^∞ ⊂ X is a unique solution of a certain infinite horizon optimal control problem which does not depend on z. These results are established when the set X is a convex subset of the Euclidean space R ⁿ, the set Ω is convex, and the function v is strictly concave. In this case we obtain the full description of the structure of approximate solutions of our second optimization problem. Note that the structure of approximate solutions in the region [0, L] depends on z while their structure in the region [T − L, T] does not depend on z. Actually it depends only on v and Ω.

In Chap. 2 of the present book we prove the generalizations of the results of Chap. 2 of [84] on the structure of approximate solutions in the regions containing end points. These generalizations are established for problems (P1)–(P3) without concavity assumptions. The results of this chapter were obtained in [83, 86, 90].

In Chap. 3 we consider optimal control systems which are discrete-time analogs of Bolza problems in the calculus of variations. They are described by a pair of objective functions which determines an optimality criterion. We consider two classes of Bolza problems and obtain for each of them the full description of approximate solutions of these problems on large intervals. This description shows that on large intervals the approximate solutions are determined mainly by our optimality criterion and are essentially independent of the choice of time intervals and data. The results of Chap. 4 were obtained in [91, 92].

In Chap. 4 we continue to study the discrete-time analogs of Bolza problems in the calculus of variations. We show that the turnpike phenomenon and the structure of solutions on finite intervals in the regions close to the endpoints are stable under small perturbations of the objective functions and the set Ω. The results of this chapter are new.

1.4 Examples

Example 1.12.

Let (X, ρ) be a compact metric space, Ω be a nonempty closed subset of X × X, $\bar{x} \in X$, $(\bar{x},\bar{x})$ be an interior point of Ω, π: X → R ¹ be a continuous function, α be a real number, and L: X × X → [0, ∞) be a continuous function such that for each (x, y) ∈ X × X the equality L(x, y) = 0 holds if and only if $(x,y) = (\bar{x},\bar{x}).$ Set

$$\displaystyle{v(x,y) =\alpha -L(x,y) +\pi (x) -\pi (y)}$$

for all x, y ∈ X. It is not difficult to see that assumptions (i), (ii), and (iii) hold.

Example 1.13.

Let X be a compact convex subset of the Euclidean space R ⁿ with the norm | ⋅ | induced by the scalar product 〈⋅ , ⋅ 〉, let ρ(x, y) = | x − y | , x, y ∈ R ⁿ, Ω be a nonempty closed subset of X × X, a point $\bar{x} \in X$, $(\bar{x},\bar{x})$ be an interior point of Ω, and let v: X × X → R ¹ be a strictly concave continuous function such that

$$\displaystyle{v(\bar{x},\bar{x}) =\sup \{ v(z,z):\; z \in X\mbox{ and }(z,z) \in \varOmega \}.}$$

We assume that there exists a positive constant $\bar{r}$ such that

$$\displaystyle{\{(x,y) \in R^{n} \times R^{n}:\; \vert x -\bar{ x}\vert,\;\vert y -\bar{ x}\vert \leq \bar{ r}\} \subset \varOmega.}$$

It is a well-known fact of convex analysis [58] that there exists a point l ∈ R ⁿ such that

$$\displaystyle{v(x,y) \leq v(\bar{x},\bar{x}) +\langle l,x - y\rangle }$$

for any point (x, y) ∈ X × X. Set

$$\displaystyle{L(x,y) = v(\bar{x},\bar{x}) +\langle l,x - y\rangle - v(x,y)}$$

for all (x, y) ∈ X × X. It is not difficult to see that this example is a particular case of Example 1.12. Therefore assumptions (i), (ii), and (iii) hold.

Example 1.14.

Let X = [0, 1], Ω = { (x, y) ∈ [0, 1] × [0, 1]:   y ≤ x ^1∕2}, v(x, y) = x ^1∕2 − y ², x, y ∈ X. It is not difficult to see that the set Ω is convex, the function v is strictly concave, the optimization problem v(z, z) → max, z ∈ X, and (z, z) ∈ Ω has a unique solution 16^−1∕3 and (16^−1∕3, 16^−1∕3) is an interior point of Ω. Therefore this example is a particular case of Example 1.13 and assumptions (i), (ii), and (iii) hold.

Example 1.15.

Let X = [0, 1], Ω = { (x, y) ∈ [0, 1] × [0, 1]:   y ≤ x ^1∕2}, v(x, y) = x ^1∕2 − y, x, y ∈ X. It is not difficult to see that the set Ω is convex, the function v is concave but not strictly concave, the optimization problem v(z, z) → max, z ∈ X, and (z, z) ∈ Ω has a unique solution 4⁻¹ and (4⁻¹, 4⁻¹) is an interior point of Ω. Since the function v is concave for all x, y ∈ X,

$$\displaystyle{v(x,y) \leq v(4^{-1},4^{-1}) + x - y = 4^{-1} + x - y}$$

and

$$\displaystyle{4^{-1} + x - y - v(x,y) = (x^{1/2} - 2^{-1})^{2}}$$

is equal zero if and only if x = 4⁻¹. Now it is not difficult to see that assumptions (i), (ii), and (iii) hold.

Example 1.16.

Consider the sets X, Ω, and the function v defined in Example 1.15 and set u(x, y) = x ^1∕2 − x ² − y + y ², x, y ∈ X. The function u is strictly convex with respect to the variable y. Nevertheless assumptions (i), (ii), and (iii) hold for the function u because for any integer T and any program {x _t } _t = 0 ^T,

$$\displaystyle{\sum _{t=0}^{T-1}u(x_{ t},x_{t+1}) =\sum _{ t=0}^{T-1}v(x_{ t},x_{t+1}) + x_{T}^{2} - x_{ 0}^{2}.}$$

1.5 Two-Player Zero-Sum Games

In Chaps. 5–8 we prove turnpike results for classes of dynamic discrete-time two-player zero-sum games. These results describe the structure of approximate solutions, for all sufficiently large intervals. We also study the structure of approximate solutions on large intervals in the regions close to the endpoints of the intervals and examine the existence of a pair of overtaking equilibria strategies over an infinite horizon.

Let $X \subset R^{m_{1}}$ and $Y \subset R^{m_{2}}$ be nonempty convex compact sets. Denote by $\mathcal{M}$ the set of all continuous functions f: X × X × Y × Y → R ¹ such that:

for each point (y ₁, y ₂) ∈ Y × Y the function (x ₁, x ₂) → f(x ₁, x ₂, y ₁, y ₂), (x ₁, x ₂) ∈ X × X is convex;

for each point (x ₁, x ₂) ∈ X × X the function (y ₁, y ₂) → f(x ₁, x ₂, y ₁, y ₂), (y ₁, y ₂) ∈ Y × Y is concave.

The set $\mathcal{M}$ is equipped with a metric $\rho: \mathcal{M}\times \mathcal{M}\rightarrow R^{1}$ defined by

$$\displaystyle{\rho (f,g) =\sup \{ \vert f(x_{1},x_{2},y_{1},y_{2}) - g(x_{1},x_{2},y_{1},y_{2})\vert:}$$

$$\displaystyle{x_{1},x_{2} \in X,\quad y_{1},y_{2} \in Y \},\quad f,g \in \mathcal{M}.}$$

It is clearly that $(\mathcal{M},\rho )$ is a complete metric space.

Given $f \in \mathcal{M}$ and a natural number n we consider a discrete-time two-player zero-sum game over the interval [0, n]. For this game {{x _i } _i = 0 ⁿ: x _i  ∈ X, i = 0, … n} is the set of strategies for the first player, {{y _i } _i = 0 ⁿ: y _i  ∈ Y, i = 0, … n} is the set of strategies for the second player, and the objective function for the first player associated with the strategies {x _i } _i = 0 ⁿ, {y _i } _i = 0 ⁿ is given by ∑ _i = 0 ⁿ⁻¹ f(x _i , x _i+1, y _i , y _i+1).

Let $f \in \mathcal{M}$, n be a natural number and let M ∈ [0, ∞). A pair of sequences $\{\bar{x}_{i}\}_{i=0}^{n} \subset X,\;\{\bar{y}_{i}\}_{i=0}^{n} \subset Y$ is called (f, M)-good if the following properties hold:

(i) for each sequence {x _i } _i = 0 ⁿ ⊂ X satisfying $x_{0} =\bar{ x}_{0}$, $x_{n} =\bar{ x}_{n}$ the inequality

$$\displaystyle{M +\sum _{ i=0}^{n-1}f(x_{ i},x_{i+1},\bar{y}_{i},\bar{y}_{i+1}) \geq \sum _{i=0}^{n-1}f(\bar{x}_{ i},\bar{x}_{i+1},\bar{y}_{i},\bar{y}_{i+1})}$$

holds;

(ii) for each sequence {y _i } _i = 0 ⁿ ⊂ Y satisfying $y_{0} =\bar{ y}_{0}$, $y_{n} =\bar{ y}_{n}$ the inequality

$$\displaystyle{M +\sum _{ i=0}^{n-1}f(\bar{x}_{ i},\bar{x}_{i+1},\bar{y}_{i},\bar{y}_{i+1}) \geq \sum _{i=0}^{n-1}f(\bar{x}_{ i},\bar{x}_{i+1},y_{i},y_{i+1})}$$

holds.

If a pair of sequences {x _i } _i = 0 ⁿ ⊂ X,  {y _i } _i = 0 ⁿ ⊂ Y is (f, 0)-good then it is called (f)-optimal.

Let $f \in \mathcal{M}$. We say that the function f possesses the turnpike property if there exists a unique pair (x _f , y _f ) ∈ X × Y for which the following assertion holds:

For each positive number ε there exist an integer n ₀ ≥ 2 and a positive number δ such that for each integer n ≥ 2n ₀ and each (f, δ)-good pair of sequences {x _i } _i = 0 ⁿ ⊂ X,  {y _i } _i = 0 ⁿ ⊂ Y the inequalities | x _i − x _f  | ,   | y _i − y _f  | ≤ ε holds for all integers i ∈ [n ₀, n − n ₀].

In [67] we showed that the turnpike property holds for a generic $f \in \mathcal{M}$. Namely, in [67] we proved the existence of a set $\mathcal{F}\subset \mathcal{M}$ which is a countable intersection of open everywhere dense sets in $\mathcal{M}$ such that each $f \in \mathcal{F}$ has the turnpike property. Thus for most functions $f \in \mathcal{M}$ the turnpike property holds. Nevertheless it is very important to have conditions on $f \in \mathcal{M}$ which imply the turnpike property. These conditions are discussed in [78] and in Chap. 5 of [84].

In Chaps. 5–8 we study the structure of solutions of more general and complicated classes of dynamic discrete-time two-player zero-sum games. In Chap. 5 we study a class of unconstrained dynamic discrete-time two-player zero-sum games without using standard convexity–concavity assumptions and prove two turnpike results of [82]. Chapter 6 contains the study of the existence and turnpike properties of approximate solutions for a class of dynamic constrained discrete-time two-player zero-sum games without convexity–concavity assumptions. Its results were obtained in [88]. In Chap. 7 we study turnpike properties of approximate solutions for a class of dynamic discrete-time two-player zero-sum nonautonomous games with convexity–concavity assumptions which were established in [87]. In Chap. 8 we analyze the existence and turnpike properties of approximate solutions for a class of dynamic constrained discrete-time two-player zero-sum games which satisfy convexity–concavity assumptions. We describe the structure of approximate solutions which is independent of the length of the interval, for all sufficiently large intervals and show that approximate solutions are determined mainly by the objective function, and are essentially independent of the choice of intervals and endpoint conditions. We show that the turnpike phenomenon is stable under small perturbations of objective functions and analyze the structure of approximate solutions in regions closed to the endpoints of domains.