# Optimal EPO dosing in hemodialysis patients using a non-linear model predictive control approach

## Abstract

Anemia management with erythropoiesis stimulating agents is a challenging task in hemodialysis patients since their response to treatment varies highly. In general, it is difficult to achieve and maintain the predefined hemoglobin (Hgb) target levels in clinical practice. The aim of this study is to develop a fully personalizable controller scheme to stabilize Hgb levels within a narrow target window while keeping drug doses low to mitigate side effects. First in-silico results of this framework are presented in this paper. Based on a model of erythropoiesis we formulate a non-linear model predictive control (NMPC) algorithm for the individualized optimization of epoetin alfa (EPO) doses. Previous to this work, model parameters were estimated for individual patients using clinical data. The optimal control problem is formulated for a continuous drug administration. This is currently a hypothetical form of drug administration for EPO as it would require a programmable EPO pump similar to insulin pumps used to treat patients with diabetes mellitus. In each step of the NMPC method the open-loop problem is solved with a projected quasi-Newton method. The controller is successfully tested in-silico on several patient parameter sets. An appropriate control is feasible in the tested patients under the assumption that the controlled quantity is measured regularly and that continuous EPO administration is adjusted on a daily, weekly or monthly basis. Further, the controller satisfactorily handles the following challenging problems in simulations: bleedings, missed administrations and dosing errors.

## Keywords

Optimal control of hyperbolic equations Model predictive control PDE-constrained optimization Quasi-Newton methods Anemia Erythropoietin## Mathematics Subject Classification

35F45 49J24 49K20 65K10 90C30## 1 Introduction

According to the 2018 United States Renal Data System annual data report (USRDS 2018), approximately 2.5 million patients were treated world-wide for end-stage renal disease in 2016. In nearly all reporting countries hemodialysis (HD) was the predominant form of dialysis therapy. On December 31, 2016, there were 726,331 prevalent cases of end-stage kidney disease in the U.S., of which 63.1% were receiving HD. Due to reduced erythropoietin production in the kidneys almost all HD patients suffer from a chronically decreased number of circulating red blood cells (RBCs) and associated low hemoglobin (Hgb) levels. This condition is called anemia. Untreated anemia is associated with poor quality of life and increased morbidity and mortality. Therefore, physicians aim for a partial correction of anemia with erythropoiesis stimulating agents (ESA). In this paper we consider the treatment with epoetin alfa (EPO), a human recombinant erythropoietin produced in cell culture.

Currently, the recommended Hgb target range for anemia management is 10–12 g/dl (see Mactier et al. 2011). Albeit, the KDIGO (Kidney Disease Improving Global Outcomes) Work Group, which provides clinical practice guidelines for anemia in chronic kidney disease and guidance on diagnosis, evaluation, management and treatment of HD patients, recommends not to exceed the limit of 11.5 g/dl in general but suggests to individualize therapy for patients whose quality of life may be improved at Hgb levels above 11.5 g/dl. Usually, dialysis facilities use dosing protocols that work as follows: A starting dose gets specified and based on the resulting Hgb change and depending on whether the patient is within the Hgb target range the ESA dose gets adjusted. This one-size-fits-all approach results in about 65% of patients achieving the set Hgb target. The Dialysis Outcomes and Practice Patterns Study (DOPPS) Practice Monitor reports that in the US, since December 2014, the percentage of patients with Hgb above 12 g/dl is 14–15% while 18–20% of patients have a Hgb below 10 g/dl. Moreover, Hgb variability and cycling are well known to occur in HD patients treated with ESA (Berns et al. 2003; Fishbane and Berns 2005, 2007). According to Yang et al. (2007) a greater Hgb variability is independently associated with higher mortality. The challenge in anemia treatment is the patients’ difference in long-term Hgb response to ESA. The drug concentration in plasma influences the maturation, proliferation and apoptosis of cells in the erythroid lineage. However, these cells remain about two weeks unobservable in the bone marrow before being released into the bloodstream. Consequently, it is difficult to anticipate the resulting delayed effect of the drug administration.

The mathematical model of erythropoiesis presented in Fuertinger et al. (2013) predicts patients’ erythropoietic response. We utilize this model to design a model-based feedback controller. The successful use of control algorithms for drug dosing has been shown, for example, in Magni et al. (2007) and in Bequette (2013). Both works are about closed-loop insulin dosing (i.e., the artificial pancreas). The control strategy we use is called model predictive control (MPC), also known as moving or receding horizon control. In 2011, Brier and Gaweda have already published an MPC-based algorithm for improved anemia management that has been tested and validated in clinical studies. Unlike our approach their predictive model is based on the concept of artificial neural networks (see also Barbieri et al. 2016; Brier et al. 2010). One of the limitations of using an artificial neural network approach is the need for large training and validation data sets making it very difficult to fully tailor this approach to individual patients and to personalize target Hgb ranges for patients whose quality of life would improve from a Hgb level higher than 11.5 g/dl as suggested by the KDIGO work group.

The purpose of this work is to develop a controller scheme that is fully personalizable. Previous to the study, the used model was adapted to individual patients using clinical data on Hgb levels and EPO administration. The details of the parameter estimation procedure and its results were published in Fuertinger et al. (2018). Note that insufficient iron availability is not modelled explicitly. However, several bone marrow parameters of the erythropoiesis model that are influenced by iron availability were estimated on a per patient level. The presented feedback controller is tested on various patient parameter sets. Throughout this manuscript we assume that EPO is administered continuously. This is currently not done in clinical practice but could be achieved with an “EPO pump” similar to the programmable insulin pumps used to treat diabetes mellitus. The approach is chosen nonetheless since it yields a continuous control which provides the best situation for stabilizing a system. Having developed a functional controller scheme it can then be adapted to actual dosing schedules and the effect of reducing the dosing frequency on the achievable stability can be analyzed. The given model equations are coupled hyperbolic partial differential equations (PDEs) and the control variable enters these equations non-linearly. In the presence of a non-linear model MPC is referred to as non-linear MPC (NMPC). More details on MPC can be found, e.g., in the books by Grüne and Pannek (2011) and by Rawlings and Mayne (2009). The basic principle of MPC consists of repeatedly solving finite horizon open loop optimal control problems. In each step, an open loop problem is solved. Then, only the first component of the obtained optimal control is applied and the optimization horizon gets pushed. This allows to include measurements and to react to unforeseen disturbances or complications. Here, we assume to know the true model and that we are able to measure Hgb perfectly. We simulate (gastro-intestinal) bleedings which are a common complication in HD patients and consider a malfunction of the pump by simulating the complete failure to administer EPO for an entire day or by applying an incorrect dose. Further, we simulate different frequencies of rate change with which the pump is programmed, ranging from daily to only once a month.

The paper is organized as follows: In Sect. 2 we introduce the control variable and present the model equations. The numerical approximation of these so-called state equations is investigated in Sect. 3. Both the model and its numerical approximation are recalled from Fuertinger et al. (2013). In addition, we regularize the erythrocytes model equation to obtain differentiability required for numerically solving the optimal control problem utilizing first-order optimality conditions. In Sect. 4 we formulate the optimal control problem and the NMPC algorithm is described. In Sect. 5 we present our numerical results of the following in-silico experiments: bleedings, missed administrations or wrongly administered doses and the restriction of EPO administration rates to be constant over several weeks. We draw some conclusions in Sect. 6. Finally, all parameters used for simulations are presented in Appendix A.

## 2 The model of erythropoiesis

We start by describing the structure of the control process; compare Fig. 1. The *control* which is a vector of EPO administration rates can be altered to change a patient’s Hgb being the outcome. First, the EPO rates change the patient’s EPO concentration in plasma which affects the production of RBCs. The process of erythropoiesis is described by a mathematical model presented in Fuertinger et al. (2013). The different cell types during erythropoiesis are grouped into five population classes and the model allows to calculate their densities \(y_1,y_2,\ldots ,y_5\). From a control perspective these are our *states*. Given the density \(y_5\) of erythrocytes we can calculate their number and finally the Hgb.

### 2.1 The dosing of EPO as the control variable

*T*] with large final time \(T\gg 1\). We assume that EPO can be applied continuously, with a constant administration rate per day or per multiple days. The EPO rates are given in U/day, where U stands for units. The EPO concentration

*E*(

*t*), \(t\in [0,T]\), in plasma is separated into a constant summand \(E^\mathrm {end}>0\) modeling the patient’s remaining endogenous erythropoietin level and a time-dependent summand \(E^\mathrm {ex}(t)\) resulting from the administered EPO:

*C*([0,

*T*]), where

*C*([0,

*T*]) is the space of all continuous functions from [0,

*T*] to \({\mathbb {R}}\). From (4) and \(\lambda >0\) we infer that

### Lemma 1

*E*introduced in (4), has the following properties.

- (1)
For every \({{\varvec{u}}}\in {{\mathscr {U}}_\mathsf {ad}}\) the function \(E(\cdot ;{{\varvec{u}}}):[0,T]\rightarrow {\mathbb {R}}\) is continuously differentiable.

- (2)The mapping \(E(t;\cdot ):{{\mathscr {U}}_\mathsf {ad}}\rightarrow {\mathbb {R}}\) is twice continuously differentiable for any \(t\in [0,T]\). Its gradient is given asat \(t\in [t_u^j,t_u^{j+1})\), \(j=1,\ldots ,n_u\), and \({{\varvec{u}}}\in {{\mathscr {U}}_\mathsf {ad}}\). Further, the hessian matrix \(\nabla ^2_uE(t;{{\varvec{u}}})\in {\mathbb {R}}^{n_u\times n_u}\) is zero.$$\begin{aligned} \nabla _{{\varvec{u}}}E(t;{{\varvec{u}}})=\frac{e^{-\lambda t}}{c_\mathsf {tbv}\lambda }\left( \begin{array}{c} e^{\lambda t_2}-e^{\lambda t_1}\\ \vdots \\ e^{\lambda t_i}-e^{\lambda t_{i-1}}\\ e^{\lambda t} - e^{\lambda t_i}\\ 0\\ \vdots \\ 0 \end{array} \right) \in {\mathscr {U}} \end{aligned}$$

### Proof

The claims follow directly from formula (4). Since \(\nabla _{{\varvec{u}}}E(t;{{\varvec{u}}})\) is independent of \({{\varvec{u}}}\), the hessian \(\nabla ^2_uE(t;{{\varvec{u}}})\) is zero. \(\square \)

### 2.2 The PDE model of erythropoiesis

A schematic of the model is shown in Fig. 2. For the underlying assumptions and more details we refer to Fuertinger (2012), Fuertinger et al. (2013). Stem cells commit to the erythroid lineage at a constant rate \(S_0>0\). Once a stem cell has committed it passes the five shown cell classes/stages over time (if it does not die along the way): BFU-E (burst-forming unit erythroids), CFU-E (colony-forming unit erythroids), erythroblasts, marrow reticulocytes and erythrocytes (including blood reticulocytes). This means that there is a flux of cells from each population class to the subsequent one. For example, when a CFU-E cell has reached maximum age of that class it leaves the class and becomes an erythroblast with minimum maturity. The population densities depend on maturity and time. For each class an age-structured population model is given which describes the development of the respective class subject to a given EPO concentration in plasma. EPO has a direct effect on the rate of apoptosis of CFU-E cells (\(\alpha _2\)), the maturation velocity of marrow reticulocytes (\(\nu \)) and the mortality rate of erythrocytes (\(\alpha _5\)). For each class we are given an individual maturity interval \(\varOmega _i=({{\underline{x}}}_i,{{\overline{x}}}_i)\subset {\mathbb {R}}\), \(1 \le i \le 5\), in days. The interval boundaries are given by \({{\underline{x}}}_1=0\), \({{\overline{x}}}_1=3 = {{\underline{x}}}_2\), \({{\overline{x}}}_2 = 8 = {{\underline{x}}}_3\), \({{\overline{x}}}_3=13 = {{\underline{x}}}_4\), \({{\overline{x}}}_4=15.5\), \({{\underline{x}}}_5 =0\), whereas the RBC lifespan \({{\overline{x}}}_5\) is patient-dependent. Ma et al. (2017) have measured this shortened RBC lifespan in HD patients to be in the range of 37.7 to 115.8 days. Note that the first four cell classes are in bone marrow while the fifth class describes cells circulating in blood. That is why \({{\underline{x}}}_5\) is set to 0.

**S.i**) denotes the cell density of the respective cell population with maturity

*x*at time

*t*. The function \(v_i\) describes the maturation velocity and \(\kappa _i(\cdot )\) is of form \(\beta _i - \alpha _i(\cdot )\), where \(\beta _i > 0\) describes the profileration rate and \(\alpha _i\) the rate of apoptosis. Actually, the function \(\alpha _5\) and the sigmoid functions \(\alpha _2\) and \(\nu \) depend on the bounded (patient-dependent) parameter vector \({\varvec{\mu }}=(\mu _i)\in {\mathbb {R}}^{10}_+\) with \({\mathbb {R}}_+=\{s\in {\mathbb {R}}\,|\,s>0\}\). To simplify the notation we do not indicate dependencies on \({\varvec{\mu }}\). We refer to Appendix A, where all fixed and all individualized parameters, which we utilize in our numerical experiments, are listed. The individualized parameters are obtained via parameter estimation; see Fuertinger et al. (2018). The functions for the different classes read as follows:

### Lemma 2

The mappings \(\alpha _2,\nu :{{\mathscr {E}}_\mathsf {ad}}\rightarrow {\mathbb {R}}\) are continuously differentiable.

### Proof

The claim follows directly from (6). \(\square \)

We denote the coupled system (**S.1**)–(**S.5**) by (**S**).

### 2.3 Total RBC population

*total RBC population*\(P=P(t)\), \(t\in [0,T]\), is given as

**S**).

### 2.4 Hgb concentration

*P*, the Hgb concentration (in g/dl) is calculated as

### 2.5 Regularization of the equation for the erythrocytes

In Sect. 4 we formally introduce the non-linear optimal control problem. In order to solve it numerically by utilizing first-order necessary optimality conditions we have to differentiate the state system (**S**) with respect to the state variable \({{\varvec{y}}}=(y_i)_{1\le i\le 5}\) and the control variable \({{\varvec{u}}}=(u_j)_{1\le j\le n_u}\). From Lemma 1 we already know that \({{\varvec{u}}}\mapsto E(t;{{\varvec{u}}})\) is continuously differentiable for every \(t\in [0,T]\). Moreover, the mappings \(\alpha _2\) and \(\nu \) are continuously differentiable by Lemma 2. However, the mapping \({{\mathscr {E}}_\mathsf {ad}}\ni E\mapsto \alpha _5(x;E)\) is non-differentiable for every \(x\in {\widehat{\varOmega }}_5\). Therefore, we have to regularize \(\alpha _5\) in order to get smooth state equations.

*R*defined in (10) can be regularized as

*E*) mapping

### Lemma 3

For every \(x\in \varOmega _5\) the mapping \(\alpha _5^\varepsilon (x;\cdot ):{{\mathscr {E}}_\mathsf {ad}}\rightarrow {\mathbb {R}}\) is continuously differentiable.

### Proof

The claim follows directly from (11) because of \(E_\mathrm {min}>0\). \(\square \)

In the sequel we replace \(\alpha _5\) in (5) by \(\alpha _5^\varepsilon \) and hence \(\kappa _5\) by \(\kappa _5^\varepsilon \) to account for the regularized fifth state equation which we denote by (\(\mathbf {S.5}^\varepsilon \)). Let (\({\mathbf {S}}^\varepsilon \)) be the state system (**S.1**)–(**S.4**) and (\(\mathbf {S.5}^\varepsilon \)).

## 3 Numerical approximation of the state equations

The numerical solution of the age-structured population models is based on semigroup theory. We formulate the five state equations as abstract Cauchy problems which are then approximated by semigroups acting on finite dimensional subspaces. We have compared this discretization to an upwind finite difference scheme (cf. Strikwerda 2004). Due to computational speed while obtaining a similar approximation quality we have favored the semigroup based approach. We refer the reader to, e.g., Ito and Kappel (2002) and Kappel and Zhang (1993) for results on evolution operators and their approximation.

In the following we derive the discretization of the state equations by means of the general form (**S.i**) where we omit the dependency on *i* but replace \(\kappa \) by \(\kappa ^\varepsilon \).

### 3.1 The state equations as abstract Cauchy problems

*y*is the solution to the corresponding state equation (

**S.1**)–(

**S.4**) or (\(\mathbf {S.5}^\varepsilon \)).

### 3.2 Approximation of the abstract Cauchy problems

In the following we derive a discretization of the Cauchy problem (13) based on shifted Legendre polynomials. This approach was originally presented in Kappel and Zhang (1993) where they apply it to a very similar type of equation. In the considered 1D case only few basis elements are needed which yields a fast approximation. However, there are situations where one would expect diffculties due to the (oscillating) characteristics of Legendre polynomials. Higher spatial dimension of the hyperbolic equation or large maturation velocities are examples for such situations. In the following, we first recall certain characteristics and further utilized features of Legendre polynomials. For more details we refer the reader to the book by Abramowitz and Stegun (1970).

**Legendre polynomials**Legendre polynomials \(L_{j},\, j\in {\mathbb {N}},\) are orthogonal polynomials on \([-1,1]\) with

**Approximation**Let us define the basis functions

*N*-dimensional subspace \(X_N \subset L^2_w(0,1)\) by

*N*ordinary differential equations

### Theorem 1

### Proof

- (1)We first show that (22) satisfies (21). Using \(E_\mathrm {min}>0\) it follows from Lemmas 1-3 that the mappings \(t\mapsto v_i(E(t;{{\varvec{u}}}))\), and \(t\mapsto \kappa _i^\varepsilon (x;E(t;{{\varvec{u}}}))\), \(1 \le i \le 5\), are continuously differentiable on [0,
*T*] for every \(x\in \varOmega \). Hence, the mapping \({\varvec{A}}^\varepsilon (E(\cdot ;{{\varvec{u}}})):[0,T]\rightarrow {\mathbb {R}}^{N\times N}\) is continuously differentiable for every \({{\varvec{u}}}\in {{\mathscr {U}}_\mathsf {ad}}\). We obtain for any \(t\in (0,T]\), where \(g(\cdot ;E)\) is continuousFurther, if we choose \(t=0\) in (22) we get \({\tilde{\varvec{y}}}(0)={\tilde{\varvec{y}}}_0\). Hence, \({\tilde{\varvec{y}}}\) satisfies (21) at every time instance, where \(g(\cdot ;E)\) is continuous.$$\begin{aligned} \frac{\mathrm {d}}{\mathrm {d}t}{\tilde{\varvec{y}}}(t)&={\varvec{A}}^\varepsilon (E(t;{{\varvec{u}}})) e^{\int _0^t{\varvec{A}}^\varepsilon (E(s;{{\varvec{u}}}))\,\mathrm {d}s}{\tilde{\varvec{y}}}_\circ +g(t;E(t;{{\varvec{u}}})){\varvec{d}} \\&\quad +{\varvec{A}}^\varepsilon (E(t;{{\varvec{u}}}))\bigg (\int _0^te^{\int _\tau ^t{\varvec{A}}^\varepsilon (E(s;{{\varvec{u}}}))\,\mathrm {d}s}g(\tau ;E(\tau ;{{\varvec{u}}}))\,\mathrm {d}\tau \bigg ){\varvec{d}}\\&={\varvec{A}}^\varepsilon (E(t;{{\varvec{u}}})){\tilde{\varvec{y}}}(t)+g(t;E(t;{{\varvec{u}}})) {\varvec{d}}. \end{aligned}$$ - (2)Uniqueness: Assume there exist two solutions \({\tilde{\varvec{y}}}^1\), \({\tilde{\varvec{y}}}^2\) to (21). We set \({\tilde{\varvec{z}}}={\tilde{\varvec{y}}}^1-{\tilde{\varvec{y}}}^2\). Then, it follows thatfor all \(t\in (0,T]\), where \(g(\cdot ;E)\) is continuous. Since \({\varvec{A}}^\varepsilon (E(\cdot ;{{\varvec{u}}}))\) and \({\tilde{\varvec{z}}}\) are continuous, we can extend the derivative of \({\tilde{\varvec{z}}}\) by \({\varvec{A}}^\varepsilon (E(t;{{\varvec{u}}})){\tilde{\varvec{z}}}(t) \) for all \(t\in (0,T]\). Furthermore, we have \({\tilde{\varvec{y}}}^1(0)={\tilde{\varvec{y}}}^2(0)\). From Gronwall’s inequality we infer that \({\tilde{\varvec{z}}}(0)=0\) in [0,$$\begin{aligned} \frac{\mathrm {d}}{\mathrm {d}t}\,{\tilde{\varvec{z}}}(t)={\varvec{A}}^\varepsilon (E(t;{{\varvec{u}}})){\tilde{\varvec{z}}}(t) \end{aligned}$$
*T*], which implies \({\tilde{\varvec{y}}}^1={\tilde{\varvec{y}}}^2\) in [0,*T*].

### Remark 1

- (1)Due to Theorem 1 the solutionto (19) belongs to \(H^1(0,T;L^2_w(0,1))\) provided \(g(\cdot ;E):[0,T]\rightarrow {\mathbb {R}}\) is piecewise continuous for every \(E\in {{\mathscr {E}}_\mathsf {ad}}\).$$\begin{aligned} {\tilde{y}}_N(t,\xi ) = \sum _{j=0}^{N-1}{{\tilde{\mathrm {y}}}}_j(t)e_j(\xi ),\quad (t,\xi )\in [0,T]\times [0,1], \end{aligned}$$
- (2)
To solve (21) a time integration method has to be applied. In our numerical experiments we apply the implicit Euler method.\(\Diamond \)

### 3.3 The control-to-state operator

*i*. Due to Theorem 1 we define the non-linear solution operatorwhere Open image in new window is the generalized Cartesian product. Hence, \({\tilde{\varvec{y}}}=({\tilde{\varvec{y}}}_i)_{1 \le i\le 5}={\mathcal {S}}_N({{\varvec{u}}})\) satisfies (\(\varvec{{\widetilde{S}}_1}\))–(\(\varvec{{\widetilde{S}}_5}\)). Then, the discretized total RBC population is given by (cf. (7))

*N*.

## 4 The optimal EPO dosing

*T*]. For the number \(n_u\in {\mathbb {N}}\) let the days \(\{t_u^j\}_{j=1}^{n_u+1}\) with a constant EPO rate in \([t_u^j,t_u^{j+1})\), \(j=1,\ldots ,n_u\), be given as

### 4.1 The optimal control problem

The goal is to stabilize the Hgb around a desired value of 10.5 g/dl in order to bring and keep the Hgb into the target window of 10–12 g/dl. Values in this range are considered as safe. We choose this lower value within the range because in case of an overshoot the Hgb can not be pulled down actively but one has to wait till it decreases of itself. The optimal control problem is formulated for the number of RBCs and not for the Hgb. This means that, for each patient, prior to optimization a desired total amount \(P^d\) of RBCs is calculated using formula (8).

*T*]

### Remark 2

- (1)
Note that (\({{\hat{\mathbf{P}}}}\)) is a non-linear optimization problem. Hence, (\({{\hat{\mathbf{P}}}}\)) is non-convex, so that several local minima might exist.\(\Diamond \)

- (2)
It can be shown that (\({{\hat{\mathbf{P}}}}\)) possesses at least one optimal control in \({{\mathscr {U}}_\mathsf {ad}}\). For the sake of brevity, we do not present the proof here.

### 4.2 The NMPC method

Problem (\({{\hat{\mathbf{P}}}}\)) can not be treated as an open-loop problem since unforeseen events and disturbances can occur. In reality, predicted and measured Hgb values will differ which has to be taken into account by means of a closed-loop controller. Moreover, patient parameters will not be constant over time which even makes readaptations of the model necessary in actual application.

### 4.3 Numerical solution of the open-loop problem (\({{\hat{\mathbf{P}}}}({t_\circ })\))

- (1)
Solve the state equations (\(\varvec{{\widetilde{S}}_1}\))–(\(\varvec{{\widetilde{S}}_5}\)) on the time horizon \([{t_\circ },{t_{\mathsf {f}}}]\) forward in time.

- (2)
Solve the adjoint state equations (\(\varvec{{\widetilde{A}}_5}\))–(\(\varvec{{\widetilde{A}}_1}\)) on the time horizon \([{t_\circ },{t_{\mathsf {f}}}]\) backward in time.

- (3)
Compute the gradient \(\nabla {\hat{J}}_N\).

## 5 Numerical results

For our numerical experiments we have chosen the data sets from five patients which capture the main occurring characteristics. The constant endogenous erythropoietin concentration \(E^\mathrm {end}\) for example is once far above the threshold for neocytolysis (\(\tau _E = 80\)), twice just slightly below and twice clearly smaller. The aim was to find a general optimization setting that works for diverse data sets. The parameters can be looked up in Table 8.

For computation of the population densities we scale the hyperbolic equations by \(10^8\) which is legitimate since the equations are linear with respect to the state variables \(y_i\), \(i = 1, \ldots , 5\). For discretization we use \(N=15\) Legendre polynomials and the time step size \(\varDelta t = 0.01\).

We first have a look at the predicted Hgb curves when no EPO is administered. Then, we show how the NMPC algorithm is able to correct anemia under the assumption of knowing the true model and patient parameters and being able to continuously and perfectly measure Hgb.

### 5.1 Uncontrolled Hgb concentration

We begin by taking a look at the predicted Hgb concentrations without EPO administration. As can be seen exemplarily in Fig. 3 the Hgb levels are running in a steady state far below the target range. This state of anemia would be critical for patients since it increases cardiovascular disease and death risk (Strippoli et al. 2004).

### 5.2 NMPC

#### 5.2.1 Settings

Total EPO doses for different values \(c_\gamma \) for patients 1 and 2

Patient 1 | Patient 2 | |||||
---|---|---|---|---|---|---|

\(c_\gamma \) | 0.1 | 10 | 100 | 0.1 | 1 | 10 |

Total dose (U) | 48,323 | 42,683 | 30,616 | 121,354 | 119,114 | 71,706 |

Our primary goal is *Hgb in target*. Thus, we choose \(c_\gamma \) such that the penalization of control costs does not block the NMPC algorithm from controlling the Hgb levels into the target range. The above test we have also done for the other patients. For patient 5 the value \(c_\gamma = 0.1\) is fine as well while patients 3 and 4 require a weaker penalization of \(c_\gamma = 0.01\). Concluding, with \(c_\gamma = 0.01\) the Hgb levels of all patients can be brought into target only that for patients 1,2 and 5 this can as well be achieved with \(c_\gamma = 0.1\) and consequently a lower total EPO dose.

#### 5.2.2 Bleeding

Bleeding patients 3 and 4: total EPO doses for different values \(u_\mathrm {max}\) after the first bleeding

Patient 4 | Patient 3 | |||||
---|---|---|---|---|---|---|

\(u_\mathrm {max}\) | 1000 | 2000 | 25,000 | 1000 | 2000 | 25,000 |

Total dose (U) | 79,124 | 91,853 | 187,660 | 83,980 | 94,948 | 199,077 |

#### 5.2.3 Missed administrations/dosing errors

#### 5.2.4 Constant EPO rates

Patient 4: total EPO doses for different constant periods

Constant period | 1 day | 1 week | 3 weeks | 4 weeks |
---|---|---|---|---|

total dose (U) | 55,598 | 56,964 | 56,721 | 57,695 |

## 6 Conclusion

The presented NMPC algorithm to correct anemia in HD patients was tested in various in-silico experiments and showed excellent performance in stabilizing simulated Hgb levels. The introduced framework uses a system of non-linear hyperbolic PDEs, which previously has been adapted to individual patients using clinically measured Hgb levels to predict individual patient response to EPO treatment (Fuertinger et al. 2018). The proposed NMPC would allow to optimize anemia treatment based on single patient data sets only. It would further allow to set individual Hgb targets for certain patient groups as proposed by the KDIGO work group. Thus, the presented work is a first step towards fully individualized anemia therapy.

The conducted in-silico experiments show that a fixed optimization setting for the controller scheme is sufficient to correct the anemia of patients with very different characteristics in the underlying prediction model. However, the penalization of the control costs needs to be tuned on a subgroup or even per-patient level to minimize the amount of administered EPO. The NMPC method requires a comparatively long horizon to account for the system’s large time delay. We have determined the required horizon length experimentally. Given this horizon length, the controller can handle the delay of the system response to treatment and achieves to stabilize Hgb levels even when presented with simulated events such as bleedings, missed administrations and EPO dosing errors. The presented controller has been tested under the assumption that treatment with EPO can be provided continuously. While this is an interesting in-silico experiment, such a therapy is currently clinically not possible as there are no “EPO pumps”, similar to the insulin pumps used for diabetes treatment, available. As a first restriction on the control we have investigated to allow changes in the EPO rate less frequently resulting in constant EPO rates over several weeks. These lead to slight oscillations of the total RBC population around the target state. However, for periods of up to four weeks the oscillations are negligible.

Different controller schemes to correct anemia in HD patients have been proposed over the last years by various groups (Barbieri et al. 2015, 2016; Brier and Gaweda 2011; Brier et al. 2010; Martínez-Martínez et al. 2014; McAllister 2017; Nichols et al. 2011). Some of them have been successfully tested in clinical trials (Brier and Gaweda 2011; Brier et al. 2010) or even been implemented in the clinical routine (Barbieri et al. 2015, 2016). Most of the proposed solutions to correct anemia are based on MPC techniques. In general, the time-varying nature of the process, long time delays in the system, high inter-individual variability in the specifics or the response of the system to treatment and the need to react to unforeseen events such as bleedings and missed doses renders the problem of correcting and stabilizing anemia in HD patients more suitable to MPC based controller approaches than PID (proportional-integral-derivative) ones. Previously presented and clinically validated MPC approaches have been based on neural networks as the underlying prediction model (Barbieri et al. 2015, 2016; Brier and Gaweda 2011; Brier et al. 2010). Such models require large training and validation sets to find the optimal weights for the neural network. The presented approach, where the underlying model is a system of coupled PDEs, although one of the more complex ones currently proposed, allows to optimize anemia treatment based on single patient data sets only. The required data from a single patient is routinely measured in clinics (as presented in Fuertinger et al. 2018). Thus, the proposed NMPC approach would provide a fully personalized anemia treatment strategy that even allows the setting of individual Hgb goals as suggested by the KDIGO guidelines. Further, estimated model parameters and their longitudinal development in individual patients potentially allow to gain further insights in the specifics of renal anemia. It might allow to better understand why some patients do not respond to treatment (e.g. short red blood cell life span versus insufficient bone marrow reaction to treatment).

In summary, the presented NMPC algorithm has the potential to bring more patients in the Hgb target range while decreasing Hgb variability and EPO utilization. However, we are still two major steps away from clinically testing the proposed NMPC approach: First, the control structure needs to be changed such that EPO is only administered during dialysis treatments (in general three times per week). With the chosen approach we will be able to analyze the effect of reducing administration times on Hgb stability. Second, the patient-model mismatch and uncertainty in parameter estimates together with measurement noise need to be addressed. In order to deal with parameter uncertainty in the underlying model, for instance, so called “robust MPC” methods would need to be incorporated into the framework. In addition, model estimates will need to be updated on a regular basis using the measured Hgb data. This is currently under investigation by the group but is beyond the scope of this publication.

## Notes

## References

- Abramowitz M, Stegun I (1970) Handbook of mathematical functions. Dover Publications Inc., New YorkzbMATHGoogle Scholar
- Barbieri C, Mari F, Stopper A, Gatti E, Escandell-Montero P, Martínez-Martínez JM, Martín-Guerrero JD (2015) A new machine learning approach for predicting the response to anemia treatment in a large cohort of end stage renal disease patients undergoing dialysis. Comput Biol Med 61:56–61CrossRefGoogle Scholar
- Barbieri C, Bolzoni E, Mari F, Cattinelli I, Bellocchio F, Martin JD, Amato C, Stopper A, Gatti E, Macdougall IC, Stuard S, Canaud B (2016) Performance of a predictive model for long-term hemoglobin response to darbepoetin and iron administration in a large cohort of hemodialysis patients. PLoS ONE 11:1–18CrossRefGoogle Scholar
- Bequette BW (2013) Algorithms for a closed-loop artificial pancreas: the case for model predictive control. J Diabetes Sci Technol 7(6):1632–1643CrossRefGoogle Scholar
- Berns JS, Elzein H, Lynn RI, Fishbane S, Meisels IS, Deoreo PB (2003) Hemoglobin variability in epoetin-treated hemodialysis patients. Kidney Int 64(4):1514–21CrossRefGoogle Scholar
- Brier ME, Gaweda AE (2011) Predictive modeling for improved anemia management in dialysis patients. Curr Opin Nephrol Hypertens 20(6):573–6CrossRefGoogle Scholar
- Brier ME, Gaweda AE, Dailey A, Aronoff GR, Jacobs AA (2010) Randomized trial of model predictive control for improved anemia management. Clin J Am Soc Nephrol 5(5):814–20CrossRefGoogle Scholar
- DOPPS (2017) Arbor research collaborative for health: DOPPS practice monitor. http://www.dopps.org/DPM/. Accessed 04 Mar 2019
- Fishbane S, Berns JS (2005) Hemoglobin cycling in hemodialysis patients treated with recombinant human erythropoietin. Kidney Int 68(3):1337–43CrossRefGoogle Scholar
- Fishbane S, Berns JS (2007) Evidence and implications of haemoglobin cycling in anaemia management. Nephrol Dial Transplant 22(8):2129–32CrossRefGoogle Scholar
- Fuertinger DH (2012) A model of erythropoiesis. PhD thesis, Karl-Franzens University GrazGoogle Scholar
- Fuertinger DH, Kappel F, Thijssen S, Levin NW, Kotanko P (2013) A model of erythropoiesis in adults with sufficient iron availability. J Math Biol 66(6):1209–1240MathSciNetCrossRefGoogle Scholar
- Fuertinger DH, Kappel F, Zhang H, Thijssen S, Kotanko P (2018) Prediction of hemoglobin levels in individual hemodialysis patients by means of a mathematical model of erythropoiesis. PLoS ONE 13(4):1–14. https://doi.org/10.1371/journal.pone.0195918 CrossRefGoogle Scholar
- Grüne L, Pannek J (2011) Nonlinear model predictive control. Communications and control engineering. Springer, LondonzbMATHGoogle Scholar
- Hinze M, Pinnau R, Ulbrich M, Ulbrich S (2009) Optimization with PDE constraints. Springer, BerlinzbMATHGoogle Scholar
- Ito K, Kappel F (2002) Evolution equations and approximations. World Scientific, SingaporeCrossRefGoogle Scholar
- Kappel F, Zhang K (1993) Approximation of linear age-structured population model using legendre polynomials. J Math Anal Appl 180:518–549MathSciNetCrossRefGoogle Scholar
- KDIGO (2012) Clinical practice guideline for anemia in chronic kidney disease. Kidney Int Suppl 2:279–335CrossRefGoogle Scholar
- Kelley CT (1999) Iterative methods for optimization. Frontiers in applied mathematics. SIAM, PhiladelphiaCrossRefGoogle Scholar
- Ma J, Dou Y, Zhang H, Thijssen S, Williams S, Kuntsevich V, Ouellet G, Wong MMY, Persic V, Kruse A, Rosales L, Wang Y, Levin NW, Kotanko P (2017) Correlation between inflammatory biomarkers and red blood cell life span in chronic hemodialysis patients. Blood Purif 43:200–205CrossRefGoogle Scholar
- Mactier R, Davies S, Dudley C, Harden P, Jones C, Kanagasundaram S, Lewington A, Richardson D, Taal M, Andrews P, Baker R, Breen C, Duncan N, Farrington K, Fluck R, Geddes C, Goldsmith D, Hoenich N, Holt S, Jardine A, Jenkins S, Kumwenda M, Lindley E, McGregor M, Mikhail A, Sharples E, Shrestha B, Shrivastava R, Stedden S, Warwick G, Wilkie M, Woodrow G, Wright M (2011) Summary of the 5th edition of the renal association clinical practice guidelines (2009–2012). Nephron Clin Pract 118(Suppl 1):27–70CrossRefGoogle Scholar
- Magni L, Raimondo DM, Bossi L, Man CD, De Nicolao G, Kovatchev B, Cobelli C (2007) Model predictive control of type 1 diabetes: an in silico trial. J Diabetes Sci Technol 1(6):804–12CrossRefGoogle Scholar
- Martínez-Martínez JM, Escandell-Montero P, Barbieri C, Soria-Olivas E, Mari F, Martínez-Sober M, Amato C, Serrano López AJ, Bassi M, Magdalena-Benedito R, Stopper A, Martín-Guerrero JD, Gatti E (2014) Prediction of the hemoglobin level in hemodialysis patients using machine learning techniques. Comput Methods Programs Biomed 117(2):208–217CrossRefGoogle Scholar
- McAllister JCW (2017) Modeling and control of hemoglobin for anemia management in chronic kidney disease. Master’s thesis, University of AlbertaGoogle Scholar
- Nichols B, Shrestha RP, Horowitz J, Hollot CV, Germain MJ, Gaweda AE, Chait Y (2011) Simplification of an erythropoiesis model for design of anemia management protocols in end stage renal disease. pp 83–86Google Scholar
- Pottgiesser T, Specker W, Umhau M, Dickhuth HH, Roecker K, Schumacher YO (2008) Recovery of hemoglobin mass after blood donation. Transfusion 48:1390–1397CrossRefGoogle Scholar
- Rawlings JB, Mayne DQ (2009) Model predictive control: theory and design. Nob Hill Pub, MadisonGoogle Scholar
- Strikwerda JC (2004) Finite difference schemes and partial differential equations. Springer, PhiladelphiazbMATHGoogle Scholar
- Strippoli GF, Craig JC, Manno C, Schena FP (2004) Hemoglobin targets for the anemia of chronic kidney disease: a meta-analysis of randomized, controlled trials. J Am Soc Nephrol 15:3154–65CrossRefGoogle Scholar
- USRDS (2018) United States Renal Data System annual data report: epidemiology of kidney disease in the United States. National Institutes of Health, National Institute of Diabetes and Digestive and Kidney Diseases, Bethesda, MDGoogle Scholar
- Yang W, Israni RK, Brunelli SM, Joffe MM, Fishbane S, Feldman HI (2007) Hemoglobin variability and mortality in esrd. J Am Soc Nephrol 18(12):3164–70CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.