Optimal BESS Control in Microgrids

Abstract

With the wide penetration of the renewable energy generators in power grids, the high variability of renewable source output poses significant challenges to the power grid, including voltage instability and power generation cost. Utilising BESS in microgrids is considered an effective mechanism for absorbing the fluctuation of local energy generation and consumption, and thereby mitigating the detrimental impact of renewable energy sources on the main grid. In this chapter, we consider the optimal control of a microgrid, e.g., a factory or a commercial building, which is equipped with a finite-capacity BESS and renewable energy generators.

Appendix

4.1.1 Proof of Proposition 4.1

Note that 0 < η ≤ 1 by definition. We first prove the proposition for the case when 0 < η < 1 by contradiction. Let x′(t) > 0, y′(t) > 0 be an optimal solution to (4.9). If x′(t) ≥ y′(t), then, there exists another solution x″(t), y″(t) such that x″(t) = x′(t) − y′(t) ≥ 0, y″(t) = 0, which satisfies the constraint (4.9d)–(4.9e). Since 0 < η < 1, the following inequality always holds.

$$\displaystyle{ x'(t)/\eta -\eta y'(t) + l(t) - r(t)> x''(t)/\eta -\eta y''(t) + l(t) - r(t). }$$

(4.22)

Note that E(t) + x′(t) − y′(t) = E(t) + x″(t) − y″(t). Thus,

$$\displaystyle{ \begin{array}{rl} &\mathbf{E}\left [\lim _{T\rightarrow \infty }\sum _{t=1}^{T}\lambda ^{t}f\left (\frac{x'(t)} {\eta } -\eta y'(t) + l(t) - r(t)\right )\right ] \\>&\mathbf{E}\left [\lim _{T\rightarrow \infty }\sum _{t=1}^{T}\lambda ^{t}f\left (\frac{x''(t)} {\eta } -\eta y''(t) + l(t) - r(t)\right )\right ], \end{array} }$$

(4.23)

which contradicts the assumption that x′(t) > 0, y′(t) > 0 is the optimal solution. Similarly, we can show that x′(t) > 0, y′(t) > 0 cannot be the optimal solution when x′(t) < y′(t). Next, we prove the proposition when η = 1. In this case, the variables x(t), y(t) in (4.9) can be replaced by a single variable h(t), where

$$\displaystyle{ h(t) = x(t) - y(t),-u_{y} \leq h(t) \leq u_{x}. }$$

(4.24)

Let us denote by h ^∗(t) the optimal solution to (4.9). If h ^∗(t) ≤ 0, the x ^∗ = 0, y ^∗(t) = −h ^∗(t) is one of the optimal solutions to (4.9). If h ^∗(t) > 0, x ^∗ = h ^∗(t), y ^∗(t) = 0 is one of the optimal solutions to (4.9). This completes the proof. ■

4.1.2 Proof of Lemma 4.1

We show the lemma by induction. Define G ^k(E, p) as the cost-to-go value at kth value iteration in Algorithm 3. First, when k = 0,

$$\displaystyle{ G^{0}(E,p) = 0\ \forall E \in \mathcal{E},p \in \mathcal{P} }$$

(4.25)

and

$$\displaystyle{ \mathbf{E}_{p\in \mathcal{P}}[G^{0}(E,p)] = 0\ \forall E \in \mathcal{E}. }$$

(4.26)

Thus, it is trivial to see that G ⁰(E, p) is convex in $E \in \mathcal{E}$ $\forall p \in \mathcal{P}$. Now we show that if G ^k(E, p) is convex in $E \in \mathcal{E}$ $\forall p \in \mathcal{P}$, then G ^k+1(E, p) is also convex in $E \in \mathcal{E}$ $\forall p \in \mathcal{P}$. Suppose that G ^k(E, p) is convex in $E \in \mathcal{E}$ $\forall p \in \mathcal{P}$. Then, $\mathbf{E}_{p\in \mathcal{P}}[G^{k}(E,p)]$ is convex in $E \in \mathcal{E}$. At (k + 1)th value iteration, for any $E \in \mathcal{E}$,

$$\displaystyle{ \begin{array}{rl} &G^{k+1}(E,p) \\ =&\min _{x,y\in \mathcal{S}(E)}f\left (\frac{x} {\eta } -\eta y + p\right ) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E + x - y,q)].\end{array} }$$

(4.27)

For any $E_{1} \in \mathcal{E}$, define x ^∗(E ₁), y ^∗(E ₁) as optimal solution to (4.27) with E = E ₁. That is,

$$\displaystyle{ \begin{array}{rl} G^{k+1}(E_{1},p) =&f\left (\frac{x^{{\ast}}(E_{ 1})} {\eta } -\eta y^{{\ast}}(E_{ 1}) + p\right ) \\ & +\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{1} + x^{{\ast}}(E_{1}) - y^{{\ast}}(E_{1}),q)]. \end{array} }$$

(4.28)

Likewise, we define x ^∗(E ₂), y ^∗(E ₂) for any $E_{2} \in \mathcal{E}$. Now let

$$\displaystyle{ E_{3} =\alpha E_{1} + (1-\alpha )E_{2}. }$$

(4.29)

Note that E ₃ satisfies that $E_{3} \in \mathcal{E}$. Then, there must exist x _t (E ₃), y _t (E ₃) such that

$$\displaystyle{ \begin{array}{rl} &x(E_{3}) =\alpha x^{{\ast}}(E_{1}) + (1-\alpha )x^{{\ast}}(E_{2}), \\ &y(E_{3}) =\alpha y^{{\ast}}(E_{1}) + (1-\alpha )y^{{\ast}}(E_{2}). \end{array} }$$

(4.30)

Because $x^{{\ast}}(E_{1}),y^{{\ast}}(E_{1}) \in \mathcal{S}(E_{1})$ and $x^{{\ast}}(E_{2}),y^{{\ast}}(E_{2}) \in \mathcal{S}(E_{2})$, we have $x(E_{3}),y(E_{3}) \in \mathcal{S}(E_{3})$ due to the linearity of (4.8). Since

$$\displaystyle{ \begin{array}{rl} &E_{3} + x(E_{3}) - y(E_{3}) \\ =&\alpha (E_{1} + x^{{\ast}}(E_{1}) - y^{{\ast}}(E_{1})) + (1-\alpha )(E_{2} + x^{{\ast}}(E_{2}) - y^{{\ast}}(E_{2})), \end{array} }$$

(4.31)

where $E_{1} + x^{{\ast}}(E_{1}) - y^{{\ast}}(E_{1}),E_{2} + x^{{\ast}}(E_{2}) - y^{{\ast}}(E_{2}) \in \mathcal{E}.$ Then, we have $E_{3} + x(E_{3}) - y(E_{3}) \in \mathcal{E}.$ Let x ^∗(E ₃), y ^∗(E ₃) be the optimal solution to (4.27) with E = E ₃. Then, for all $p \in \mathcal{P}$,

$$\displaystyle\begin{array}{rcl} & & G^{k+1}(E_{ 3},p) \\ & & \!\!\!= f\left (\frac{x^{{\ast}}(E_{3})} {\eta } -\eta y^{{\ast}}(E_{ 3}) + p\right ) \\ & & +\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{ 3} + x^{{\ast}}(E_{ 3}) - y^{{\ast}}(E_{ 3}),q)]{}\end{array}$$

(4.32a)

$$\displaystyle{ \leq f(x(E_{3})/\eta -\eta y(E_{3}) + p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{ 3} + x(E_{3}) - y(E_{3}),q)] }$$

(4.32b)

$$\displaystyle\begin{array}{rcl} & \leq & \alpha f(x^{{\ast}}(E_{ 1})/\eta -\eta y^{{\ast}}(E_{ 1}) + p) + (1-\alpha )f(x^{{\ast}}(E_{ 2})/\eta -\eta y^{{\ast}}(E_{ 2}) \\ & & +p) +\alpha \lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{ 1} + x^{{\ast}}(E_{ 1}) - y^{{\ast}}(E_{ 1}),q)] \\ & & +(1-\alpha )\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{ 2} + x^{{\ast}}(E_{ 2}) - y^{{\ast}}(E_{ 2}),q)] {}\end{array}$$

(4.32c)

$$\displaystyle\begin{array}{rcl} & =& \alpha G^{k+1}(E_{ 1},p) + (1-\alpha )G^{k+1}(E_{ 2},p),{}\end{array}$$

(4.32d)

where (4.32a) and (4.32b) hold based on the definition of x ^∗(E ₃), y ^∗(E ₃) and x(E ₃), y(E ₃), (4.32b) holds based on the definition of x ^∗(E ₃), y ^∗(E ₃) and x(E ₃), y(E ₃), (4.32c) holds due to the convexity of $f(\frac{x(E_{3})} {\eta } -\eta y(E_{3}) + p)$ and $\lambda \mathbf{E}_{q\in \mathcal{P}}[G^{k}(E_{3} + x(E_{3}) - y(E_{3}),q)]$, and (4.32d) holds based on the definition of x ^∗(E ₁), y ^∗(E ₁) and x ^∗(E ₂), y ^∗(E ₂). Thus, G ^k+1(E, p) is convex in $E \in \mathcal{E}$ for all $p \in \mathcal{P}$. Therefore, we have shown that G ^k(E, p) is convex in $E \in \mathcal{E}$ for all $p \in \mathcal{P},k = 0,1,2,\cdots$. Similarly, we can show that $\mathbf{E}_{p\in \mathcal{P}}[G^{k}(E,p)]$ is convex in $E \in \mathcal{E}$ for each given $p \in \mathcal{P}$. Since

$$\displaystyle{ \lim _{k\rightarrow \infty }G^{k}(E,p) = G^{{\ast}}(E,p)\ \forall E \in \mathcal{E},p \in \mathcal{P}, }$$

(4.33)

it is suffices to show that G ^∗(E, p) is convex in $E \in \mathcal{E}$ for all $p \in \mathcal{P}$. Due to the linearity of the expectation operator, we see that $\mathbf{E}_{p\in \mathcal{P}}[G^{{\ast}}(E,p)]$ is convex in $E \in \mathcal{E}$ for all $p \in \mathcal{P}$. This completes the proof. ■

4.1.3 Proof of Theorem 4.1

We define an auxiliary variable e,

$$\displaystyle{ e = E + x - y,e \in \mathcal{E}. }$$

(4.34)

In the next proof, we use e instead of x, y in (4.11) since e and E uniquely determine x and y through the definition of e and the fact that x and y are not non-zero at the same time. To prove the theorem, we show in the following that the thresholds ξ ₁(E, p) and ξ ₂(E, p) are the target e that would minimize the function $\tilde{G} (E,p,x,y)$ in the case when x ≥ 0, y = 0 and the case when x = 0, y ≥ 0, respectively. When x ≥ 0, y = 0, the space of e, denoted by $\mathcal{H}_{1}$, is given by

$$\displaystyle{ \mathcal{H}_{1} =\{ e\vert e \in \mathcal{E},0 \leq e - E \leq u_{x}\} }$$

(4.35)

and $\tilde{G} (E,p,x,y)$ is computed by

$$\displaystyle{ \tilde{G} (E,p,x,y) = f((e - E)/\eta + p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G(e,q)]\ \forall e \in \mathcal{H}_{1}. }$$

(4.36)

When x = 0, y ≥ 0, the space of e, denoted by $\mathcal{H}_{2}$, is given by

$$\displaystyle{ \mathcal{H}_{2} =\{ e\vert e \in \mathcal{E},0 \leq E - e \leq u_{y}\} }$$

(4.37)

and $\tilde{G} (E,p,x,y)$ is computed by

$$\displaystyle{ \tilde{G} (E,p,x,y) = f(\eta (e - E) + p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G(e,q)]\ \forall e \in \mathcal{H}_{2}. }$$

(4.38)

For notation brevity, we define function $\hat{G}_{1}(E,p,e)$ and $\hat{G}_{2}(E,p,e)$ as

$$\displaystyle{ \hat{G}_{1}(E,p,e) = f((e - E)/\eta + p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G(e,q)] }$$

(4.39a)

$$\displaystyle{ \hat{G}_{2}(E,p,e) = f(\eta (e - E) + p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G(e,q)], }$$

(4.39b)

for all $e \in \mathcal{R}$ respectively. Thus, the cost-to-go for $e \in \mathcal{H}_{1}$, denoted by G ₁(E, p), and the cost-to-go for $e \in \mathcal{H}_{2}$, denoted by G ₂(E, p), are computed by

$$\displaystyle{ G_{1}(E,p) =\min _{e\in \mathcal{H}_{1}}\hat{G} _{1}(E,p,e), }$$

(4.40)

$$\displaystyle{ G_{2}(E,p) =\min _{e\in \mathcal{H}_{2}}\hat{G} _{2}(E,p,e), }$$

(4.41)

respectively. Specially, when x = 0, y = 0, we have e = E, and the cost-to-go, denoted by G ₃(E, p), is computed by

$$\displaystyle{ G_{3}(E,p) =\min _{e=E}f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G(e,q)]. }$$

(4.42)

The optimal charging policy to (4.11) can be obtained by solving (4.40)–(4.42) with variable e in different regions, and select the one that yields the minimum optimal values of (4.40)–(4.42), i.e.,

$$\displaystyle{ G^{{\ast}}(E,p) =\min \{ G_{ 1}(E,p),G_{2}(E,p),G_{3}(E,p)\}. }$$

(4.43)

Intuitively, the optimal solutions to (4.40), (4.41), (4.42) are the optimal solution if the battery is under charging mode, discharging mode and idle mode, respectively. By Lemma 4.1, both $\hat{G}_{1}(E,p,e)$ and $\hat{G}_{2}(E,p,e)$ are convex in e for all E, p. We denote the minimizer of $\hat{G} _{1}(E,p,e)$ and $\hat{G} _{2}(E,p,e)$ by ξ ₁(E, p) and ξ ₂(E, p) respectively. By definition, ξ ₁(E, p) satisfies the condition that the first order derivative of $\hat{G}_{1}(E,p,e)$ at e is equal to 0, i.e.,

$$\displaystyle{ f'\left ((\xi _{1}(E,p) - E)/\eta + p\right )/\eta +\lambda \left (\mathbf{E}_{q\in \mathcal{P}}[G(\xi _{1}(E,p),q)]\right )' = 0. }$$

(4.44)

Similarly, we have

$$\displaystyle{ \eta f'\left (-\eta (E -\xi _{2}(E,p)) + p\right ) +\lambda \left (\mathbf{E}_{q\in \mathcal{P}}[G(\xi _{2}(E,p),q)]\right )' = 0. }$$

(4.45)

Now we show that

$$\displaystyle{ \xi _{1}(E,p) \leq \xi _{2}(E,p)\forall E \in \mathcal{E},p \in \mathcal{P} }$$

(4.46)

by the contradiction. Assume that ξ ₁(E, p) > ξ ₂(E, p). Then,

$$\displaystyle{ (\xi _{1}(E,p) - E)/\eta + p> -\eta (E -\xi _{2}(E,p)) + p. }$$

(4.47)

Since f′(. ) is a non-decreasing function, then based on (4.47),

$$\displaystyle{ f'\left ((\xi _{1}(E,p) - E)/\eta + p\right )/\eta \geq \eta f'\left (-\eta (E -\xi _{2}(E,p)) + p\right ). }$$

(4.48)

Combining (4.44), (4.45), (4.48), we have

$$\displaystyle{ \left (\mathbf{E}_{q\in \mathcal{P}}[G(\xi _{1}(E,p),q)]\right )' \leq \left (\mathbf{E}_{q\in \mathcal{P}}[G(\xi _{2}(E,p),q)]\right )'. }$$

(4.49)

Since $\mathbf{E}_{q\in \mathcal{P}}[G(e,q)]$ is a convex function of e, as shown in Lemma 4.1, then, $\left (\mathbf{E}_{q\in \mathcal{P}}[G(e,q)]\right )'$ is a non-decreasing function of e. Thus, there is ξ ₁(E, p) ≤ ξ ₂(E, p), which leads to the contradiction with the assumption that ξ ₁(E, p) > ξ ₂(E, p). Next, we divide all possible relationships between ξ ₁(E, p), ξ ₂(E, p), E, 0, ζ into the following five cases to discuss the optimal charging policy.

Case 1: :

If 0 ≤ E ≤ ζ ≤ ξ ₁(E, p), we calculate the optimal e under three battery modes respectively. In other words, we calculate the optimal solution to (4.40), (4.41), (4.42) respectively. When the battery is under idle mode, i.e., e = E, the optimal solution to (4.42) is trivial to be E, and the optimal value is $G_{3}(E,p) = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)]$. When the battery is under charging mode, i.e., $e \in \mathcal{H}_{1}$, $\hat{G} _{1}(E,p,e)$ decreases as e increases within the domain (E, ζ]. Then, the optimal solution to (4.40) is given by e = min{ζ, E + u _x } and the optimal value is given by $G_{1}(E,p) = \hat{G} _{1}(E,p,\min \{\zeta,E + u_{x}\})$. When the battery is under discharging mode, i.e., $e \in \mathcal{H}_{2}$, $\hat{G} _{2}(E,p,e)$ increases as e decreases within the domain [0, E]. Then, the optimal solution to (4.41) is e = E and the optimal value is given by $G_{2}(E,p) = \hat{G} _{2}(E,p,E)$. Thus, based on the fact that the $\hat{G} _{1}(E,p,e)$ decreases within the domain [E, ζ], we have the following inequality.

$$\displaystyle{ G_{1}(E,p) = \hat{G} _{1}(E,p,\min \{\zeta,E + u_{x}\}) }$$

(4.50a)

$$\displaystyle{ \leq \hat{G} _{1}(E,p,E) }$$

(4.50b)

$$\displaystyle{ = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)] }$$

(4.50c)

$$\displaystyle{ = G_{3}(E,p). }$$

(4.50d)

On the other hand,

$$\displaystyle{ G_{2}(E,p) = \hat{G} _{2}(E,p,E) = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)] = G_{3}(E,p). }$$

(4.51)

Therefore, the optimal cost-to-go is given by

$$\displaystyle{ G^{{\ast}}(E,p) = G_{ 1}(E,p) = \hat{G} _{1}(E,p,\min \{\zeta,E + u_{x}\}) }$$

(4.52)

and the optimal charging policy is given by

$$\displaystyle{ x^{{\ast}} =\min \{\zeta -E,u_{ x}\},y^{{\ast}} = 0. }$$

(4.53)

Case 2: :

If 0 ≤ E ≤ ξ ₁(E, p) ≤ ζ, we calculate the optimal value under three battery modes respectively. When the charging is under idle or discharge mode, the results are the same as Case 1. When the battery is under charging mode, i.e., $e \in \mathcal{H}_{1}$, $\hat{G} _{1}(E,p,e)$ decreases as e increases within the domain (E, ξ ₁(E, p)]. Thus, the optimal solution to (4.40) is given by e = min{ξ ₁(E, p), E + u _x } and the optimal value is given by $G_{1}(E,p) = \hat{G} _{1}(E,p,\min \{\xi _{1}(E,p),E + u_{x}\})$. Since $\hat{G} _{1}(E,p,e)$ decreases within the domain [E, ξ ₁(E, p)],

$$\displaystyle{ G_{1}(E,p) = \hat{G} _{1}(E,p,\min \{\xi _{1}(E,p),E + u_{x}\}) }$$

(4.54a)

$$\displaystyle{ \leq \hat{G} _{1}(E,p,E) }$$

(4.54b)

$$\displaystyle{ = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)] }$$

(4.54c)

$$\displaystyle{ = G_{3}(E,p). }$$

(4.54d)

Thus, the optimal cost-to-go is given by

$$\displaystyle{ G^{{\ast}}(E,p) = G_{ 1}(E,p) = \hat{G} _{1}(E,p,\min \{\xi _{1}(E,p),E + u_{x}\}), }$$

(4.55)

and the optimal charging policy is given by

$$\displaystyle{ x^{{\ast}} =\min \{\xi _{ 1}(E,p) - E,u_{x}\},y^{{\ast}} = 0. }$$

(4.56)

Case 3: :

If ξ ₂(E, p) ≤ 0 ≤ E ≤ ζ, we still solve (4.40), (4.41), (4.42) respectively. When the battery is under idle mode, i.e., e = E, the optimal solution to (4.42) is e = E, and the optimal value is $G_{3}(E,p) = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)]$. When the battery is under discharging mode, i.e., $e \in \mathcal{H}_{2}$, $\hat{G} _{2}(E,p,e)$ decreases as e decreases within the domain (0, E]. Then, the optimal solution to (4.41) is given by e = max{0, E − u _y } and the optimal value is given by $G_{2}(E,p) = \hat{G} _{2}(E,p,\max \{0,E - u_{y}\})$. When the battery is under charging mode, i.e., $e \in \mathcal{H}_{1}$, $\hat{G} _{1}(E,p,e)$ increases as e increases within the domain [E, ζ] Then, the optimal solution to (4.40) is e = E and the optimal value is given by $G_{1}(E,p) = \hat{G} _{1}(E,p,E)$. By the fact that $\hat{G} _{2}(E,p,e)$ increases within the domain [0, E], we have the following inequality.

$$\displaystyle{ G_{2}(E,p) = \hat{G} _{2}(E,p,\max \{0,E - u_{y}\}) }$$

(4.57a)

$$\displaystyle{ \leq \hat{G} _{2}(E,p,E) }$$

(4.57b)

$$\displaystyle{ = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)] }$$

(4.57c)

$$\displaystyle{ = G_{3}(E,p). }$$

(4.57d)

On the other hand,

$$\displaystyle{ G_{1}(E,p) = \hat{G} _{1}(E,p,E) = f(p) +\lambda \mathbf{E}_{q\in \mathcal{P}}[G_{t}(E,q)] = G_{3}(E,p). }$$

(4.58)

Thus, the optimal cost-to-go is given by

$$\displaystyle{ G^{{\ast}}(E,p) = G_{ 2}(E,p) = \hat{G} _{2}(E,p,\max \{0,E - u_{y}\}), }$$

(4.59)

and the optimal charging policy is given by

$$\displaystyle{ x^{{\ast}} = 0,y^{{\ast}} =\min \{ E,u_{ y}\}. }$$

(4.60)

Case 4: :

If 0 ≤ ξ ₂(E, p) ≤ E ≤ ζ, similar to Case 3, the optimal cost-to-go is given by

$$\displaystyle{ G^{{\ast}}(E,p) = G_{ 2}(E,p) = \hat{G} _{2}(E,p,\max \{\xi _{2}(E,p),E - u_{y}\}), }$$

(4.61)

and the optimal charging policy is given by

$$\displaystyle{ x^{{\ast}} = 0,y^{{\ast}} =\min \{ E -\xi _{ 2}(E,p),u_{y}\}. }$$

(4.62)

Case 5: :

If ξ ₁(E, p) < E < ξ ₂(E, p), we can observe that under the charging mode $\hat{G} _{1}(E,p,e)$ increases as e increases within (E, ξ ₂(E, p)] and under the discharging mode $\hat{G} _{2}(E,p,e)$ increases as e decreases within [ξ ₁(E, p), E). That is, there is no incentive to charge or discharge. i.e.,

$$\displaystyle{ G_{1}(E,p) = \hat{G} _{1}(E,p,E) = G_{3}(E,p) }$$

(4.63)

and

$$\displaystyle{ G_{2}(E,p) = \hat{G} _{2}(E,p,E) = G_{3}(E,p). }$$

(4.64)

Therefore, the optimal cost-to-go is given by

$$\displaystyle{ G^{{\ast}}(E,p) = G_{ 1}(E,p) = G_{2}(E,p) = G_{3}(E,p), }$$

(4.65)

and the optimal charging policy is given by

$$\displaystyle{ x^{{\ast}} = 0,y^{{\ast}} = 0. }$$

(4.66)

Combining the results of Case 1 and Case 2, Case 3 and Case 4, we can get the first two equalities in (4.14) respectively, and Case 5 verifies the third equality in (4.14). This completes the proof. ■

4.1.4 Proof of Proposition 4.2

In order to show the proposition, it is sufficient to show that the function $\tilde{G}(E,p,x,y)$ is an increasing function of x and decreasing function of y for all $x,y \in \mathcal{S}(E),E \in \mathcal{E},p \in \mathcal{P},y \in \mathcal{S}(E)$. That is, for all $x_{1} \leq x_{2},x_{1},x_{2} \in \mathcal{S}(E),E \in \mathcal{E},p \in \mathcal{P},y \in \mathcal{S}(E)$, we have

$$\displaystyle{ \tilde{G} (E,p,x_{1},y) \leq \tilde{G} (E,p,x_{2},y), }$$

(4.67)

and for all $y_{1} \geq y_{2},y_{1},y_{2} \in \mathcal{S}(E),E \in \mathcal{E},p \in \mathcal{P},x \in \mathcal{S}(E)$, we have

$$\displaystyle{ \tilde{G} (E,p,x,y_{1}) \leq \tilde{G} (E,p,x,y_{2}). }$$

(4.68)

Now we show that (4.67) holds for all $x_{1} \leq x_{2},x_{1},x_{2} \in \mathcal{S}(E),E \in \mathcal{E},p \in \mathcal{P},y \in \mathcal{S}(E)$. Equation (4.67) is equivalent to the following inequality.

$$\displaystyle{ \begin{array}{rl} &\mathbf{E}_{q\in \mathcal{P}}[G(E + x_{1} - y,q) - G(E + x_{2} - y,q)] \leq \\ &(x_{2} - x_{1})\left ((x_{1} + x_{2})/\eta - 2\eta y + 2p\right )/(\eta \lambda ).\end{array} }$$

(4.69)

Since $x_{1},x_{2},y \in \mathcal{S}(E)$, then,

$$\displaystyle{ (x_{1} + x_{2})/\eta - 2\eta y + 2p \geq 2p - 2\eta u_{y} \geq 2p_{1} - 2\eta u_{y}. }$$

(4.70)

Combining (4.15) and (4.70), we have

$$\displaystyle{ (x_{2} - x_{1})\left ((x_{1} + x_{2})/\eta - 2\eta y + 2p\right )/(\eta \lambda ) \geq (x_{2} - x_{1})(2p_{2} + 2u_{x}/\eta ). }$$

(4.71)

Thus, to prove (4.69) holds, it suffices to show that the inequality

$$\displaystyle{ G(E + x_{1} - y,q) - G(E + x_{2} - y,q) \leq (x_{2} - x_{1})(2p_{2} + 2u_{x}/\eta ) }$$

(4.72)

holds for all $q \in \mathcal{P}$. Next, we show that (4.72) holds by the mathematical induction. Define G ^k(E, q) as the cost-to-go value at kth value iteration for all $E \in \mathcal{E},q \in \mathcal{P}$ in Algorithm 3. When k = 0,

$$\displaystyle{ \begin{array}{rl} &G^{0}(E + x_{1} - y,q) - G^{0}(E + x_{2} - y,q) \\ =&0 \leq (x_{2} - x_{1})(2p_{2} + 2u_{x}/\eta ).\end{array} }$$

(4.73)

Now assume that $\forall q \in \mathcal{P},k = 0,1,2,\cdots$,

$$\displaystyle{ G^{k}(E + x_{ 1} - y,q) - G^{k}(E + x_{ 2} - y,q) \leq (x_{2} - x_{1})(2p_{2} + 2u_{x}/\eta ). }$$

(4.74)

Let x ₁ ^k+1, y ₁ ^k+1 and x ₂ ^k+1, y ₂ ^k+1 be the solutions that yield the cost-to-go value at (k + 1)th value iteration, i.e., G ^k+1(E + x ₁ − y, q) and G ^k+1(E + x ₂ − y, q) respectively. Then,

$$\displaystyle\begin{array}{rcl} & & G^{k+1}(E + x_{ 1} - y,q) - G^{k+1}(E + x_{ 2} - y,q) \\ & =& {\biggl (\frac{x_{1}^{k+1}} {\eta } -\eta y_{1}^{k+1} + q\biggr )}^{2} -{\biggl (\frac{x_{2}^{k+1}} {\eta } -\eta y_{2}^{k+1} + q\biggr )}^{2} \\ & & +\lambda \mathbf{E}_{q'\in \mathcal{P}}[G^{k}(E + x_{ 1} - y + x_{1}^{k+1} - y_{ 1}^{k+1},q')] \\ & & -\lambda \mathbf{E}_{q'\in \mathcal{P}}[G^{k}(E + x_{ 2} - y + x_{2}^{k+1} - y_{ 2}^{k+1},q')]{}\end{array}$$

(4.75a)

$$\displaystyle\begin{array}{rcl} & =& {\biggl (\frac{x_{1}^{k+1} + x_{2}^{k+1}} {\eta } -\eta (y_{1}^{k+1} + y_{ 2}^{k+1}) + 2q\biggr )}\biggl (\frac{x_{1}^{k+1} - x_{ 2}^{k+1}} {\eta } \\ & & -\eta (y_{1}^{k+1} - y_{ 2}^{k+1})\biggr ) +\lambda \mathbf{E}_{ q'\in \mathcal{P}}[G^{k}(E + x_{ 1} - y + x_{1}^{k+1} \\ & & -y_{1}^{k+1},q') - G^{k}(E + x_{ 2} - y + x_{2}^{k+1} - y_{ 2}^{k+1},q')]{}\end{array}$$

(4.75b)

$$ \displaystyle\begin{array}{rcl} & \leq & {\biggl (\frac{x_{1}^{k+1} + x_{2}^{k+1}} {\eta } -\eta (y_{1}^{k+1} + y_{ 2}^{k+1}) + 2q\biggr )}\biggl (\frac{x_{1}^{k+1} - x_{ 2}^{k+1}} {\eta } \\ & & -\eta (y_{1}^{k+1} - y_{ 2}^{k+1})\biggr ) +\lambda (x_{ 2}^{k+1} - x_{ 1}^{k+1} -\eta (y_{ 2}^{k+1} - y_{ 1}^{k+1}) \\ & & +x_{2} - x_{1})(2p_{2} + 2u_{x}/\eta ), {}\end{array} $$

(4.75c)

where the last inequality holds based on (4.74). Since

$$\displaystyle{ x_{1}^{k+1} \leq u_{ x},x_{2}^{k+1} \leq u_{ x},y_{1}^{k+1} \geq 0,y_{ 2}^{k+1} \geq 0, }$$

(4.76)

then

$$\displaystyle{ \frac{x_{1}^{k+1} + x_{2}^{k+1}} {\eta } -\eta (y_{1}^{k+1} + y_{ 2}^{k+1}) + 2q \leq 2u_{ x}/\eta + 2q }$$

(4.77a)

$$\displaystyle{ \leq 2u_{x}/\eta + 2p_{2}, }$$

(4.77b)

where the last inequality holds because q ≤ p ₂. By substituting (4.77) into (4.75), we have

$$\displaystyle\begin{array}{rcl} & & G^{k+1}(E + x_{ 1} - y,q) - G^{k+1}(E + x_{ 2} - y,q) \\ & \leq & (2u_{x}/\eta + 2p_{2}){\biggl (\frac{x_{1}^{k+1} - x_{2}^{k+1}} {\eta } -\eta (y_{1}^{k+1} - y_{ 2}^{k+1})\biggr )} \\ & & +\lambda {\biggl (\frac{x_{2}^{k+1} - x_{1}^{k+1}} {\eta } -\eta (y_{2}^{k+1} - y_{ 1}^{k+1}) + x_{ 2} - x_{1}\biggr )}(2p_{2} \\ & & +2u_{x}/\eta ) {}\end{array}$$

(4.78a)

$$\displaystyle{ \leq (x_{2} - x_{1})(2u_{x}/\eta + 2p_{2}), }$$

(4.78b)

where the last inequality holds because 0 < η ≤ 1, λ ≤ 1. Since

$$\displaystyle{ \begin{array}{rl} &\lim _{k\rightarrow \infty }G^{k}(E + x_{1} - y,q) - G^{k}(E + x_{2} - y,q) \\ =&G(E + x_{1} - y,q) - G(E + x_{2} - y,q)\end{array} }$$

(4.79)

holds for all $q \in \mathcal{P}$, it is sufficient to show that (4.72) holds. Similarly, we can show that (4.68) holds for any $y_{1} \geq y_{2},y_{1},y_{2} \in \mathcal{S}(E)$. This completes the proof. ■

4.1.5 Proof of Proposition 4.2

For any ζ′ ≥ 0, define x ^∗(ζ′), y ^∗(ζ′) as optimal solution to (4.11) with ζ = ζ′. Likewise, we define x ^∗(ζ″), y ^∗(ζ″) for any ζ″ ≥ 0. Now let

$$\displaystyle{ \zeta ''' =\alpha \zeta ' + (1-\alpha )\zeta '',\forall \alpha \in [0,1]. }$$

(4.80)

Then, there must exist a group of x(ζ‴), y ^∗(ζ‴) such that

$$\displaystyle{ x(\zeta ''') =\alpha x^{{\ast}}(\zeta ') + (1-\alpha )x^{{\ast}}(\zeta ''),y(\zeta ''') =\alpha y^{{\ast}}(\zeta ') + (1-\alpha )y^{{\ast}}(\zeta ''). }$$

(4.81)

Note that x(ζ‴), y(ζ‴) still satisfies $x(\zeta '''),y(\zeta ''') \in \mathcal{S}(E)$ due to the linearity of (4.8). Based on (4.81), we have

$$\displaystyle{ \begin{array}{rl} &\alpha (x_{t}^{{\ast}}(\zeta ')/\eta -\eta y_{t}^{{\ast}}(\zeta ') + p_{t}) + (1-\alpha )(x_{t}^{{\ast}}(\zeta '')/\eta -\eta y_{t}^{{\ast}}(\zeta '') + p_{t}) \\ =&x_{t}(\zeta ''')/\eta -\eta y_{t}(\zeta ''') + p_{t},\alpha \in [0,1],\end{array} }$$

(4.82)

and

$$\displaystyle{ \begin{array}{rl} &\alpha (E + x^{{\ast}}(\zeta ') - y^{{\ast}}(\zeta ')) + (1-\alpha )(E + x^{{\ast}}(\zeta '') - y^{{\ast}}(\zeta '')) \\ =&E + x(\zeta ''') - y^{{\ast}}(\zeta '''),\forall \alpha \in [0,1]. \end{array} }$$

(4.83)

Then, we have

$$\displaystyle{ \begin{array}{rl} &f(x(\zeta ''')/\eta -\eta y(\zeta ''') + p) \leq \alpha f(x^{{\ast}}(\zeta ')/\eta -\eta y^{{\ast}}(\zeta ') + p_{t}) \\ & + (1-\alpha )f(x^{{\ast}}(\zeta '')/\eta -\eta y^{{\ast}}(\zeta '') + p_{t}) \end{array} }$$

(4.84)

and

$$\displaystyle{ \begin{array}{rl} &\mathbf{E}_{q\in \mathcal{P}}[G^{{\ast}}(E + x(\zeta ''') - y(\zeta '''),q)] \leq \alpha \mathbf{E}_{q\in \mathcal{P}}[G^{{\ast}}(E + x^{{\ast}}(\zeta ') \\ & - y^{{\ast}}(\zeta '),q)] + (1-\alpha )\mathbf{E}_{q\in \mathcal{P}}[G^{{\ast}}(E + x^{{\ast}}(\zeta '') - y^{{\ast}}(\zeta ''),q)] \end{array} }$$

(4.85)

for all α ∈ [0, 1], where inequality (4.84) and (4.85) hold due to the convexity of f(x(ζ‴)∕η −η y(ζ‴) + p) and $\mathbf{E}_{q\in \mathcal{P}}[G^{{\ast}}(E + x(\zeta ''') - y(\zeta '''),q)]$, respectively. Thus, Combining (4.84) and (4.85), we have

$$\displaystyle{ \begin{array}{rl} &\tilde{G}(E,p,x(\zeta '''),y(\zeta ''')) \\ \leq &\alpha \tilde{G}(E,p,x^{{\ast}}(\zeta '),y^{{\ast}}(\zeta ')) + (1-\alpha )\tilde{G} (E,p,x^{{\ast}}(\zeta ''),y^{{\ast}}(\zeta ''))\end{array} }$$

(4.86)

for all α ∈ [0, 1]. Let x ^∗(ζ‴), y ^∗(ζ‴) be the optimal solution to (4.11) with ζ = ζ‴. By definition,

$$\displaystyle{ \tilde{G} (E,p,x^{{\ast}}(\zeta '''),y^{{\ast}}(\zeta ''')) \leq \tilde{G} (E,p,x(\zeta '''),y(\zeta ''')). }$$

(4.87)

Based (4.86) and (4.87), we have

$$\displaystyle{ \begin{array}{rl} &\tilde{G} (E,p,x^{{\ast}}(\zeta '''),y^{{\ast}}(\zeta ''')) \\ \leq &\alpha \tilde{G} (E,p,x^{{\ast}}(\zeta '),y^{{\ast}}(\zeta ')) + (1-\alpha )\tilde{G} (E,p,x^{{\ast}}(\zeta ''),y^{{\ast}}(\zeta '')) \end{array} }$$

(4.88)

for all α ∈ [0, 1]. Thus, we have established the convexity of G ^∗(E, p) over ζ, ζ ≥ 0. Next, we show that G ^∗(E, p) is a decreasing function of ζ. Suppose that ζ′ ≤ ζ″. Then, by definition of x ^∗(ζ′) and y ^∗(ζ′), x ^∗(ζ′) and y ^∗(ζ′) satisfy

$$\displaystyle{ 0 \leq E + x^{{\ast}}(\zeta ') - y^{{\ast}}(\zeta ') \leq \zeta '\leq \zeta '', }$$

(4.89)

which indicates that x ^∗(ζ′), y ^∗(ζ′) are the feasible solutions to (4.11) with ζ = ζ″. Since x ^∗(ζ″), y ^∗(ζ″) are the optimal solutions to (4.11) with ζ = ζ″, then, we have

$$\displaystyle{ \tilde{G} (E,p,x^{{\ast}}(\zeta '),y^{{\ast}}(\zeta ')) \geq \tilde{G} (E,p,x^{{\ast}}(\zeta ''),y^{{\ast}}(\zeta '')). }$$

(4.90)

This completes the proof. ■