An experimental design framework for Markovian gene regulatory networks under stationary control policy
- 161 Downloads
- 1 Citations
Abstract
Background
A fundamental problem for translational genomics is to find optimal therapies based on gene regulatory intervention. Dynamic intervention involves a control policy that optimally reduces a cost function based on phenotype by externally altering the state of the network over time. When a gene regulatory network (GRN) model is fully known, the problem is addressed using classical dynamic programming based on the Markov chain associated with the network. When the network is uncertain, a Bayesian framework can be applied, where policy optimality is with respect to both the dynamical objective and the uncertainty, as characterized by a prior distribution. In the presence of uncertainty, it is of great practical interest to develop an experimental design strategy and thereby select experiments that optimally reduce a measure of uncertainty.
Results
In this paper, we employ mean objective cost of uncertainty (MOCU), which quantifies uncertainty based on the degree to which uncertainty degrades the operational objective, that being the cost owing to undesirable phenotypes. We assume that a number of conditional probabilities characterizing regulatory relationships among genes are unknown in the Markovian GRN. In sum, there is a prior distribution which can be updated to a posterior distribution by observing a regulatory trajectory, and an optimal control policy, known as an “intrinsically Bayesian robust” (IBR) policy. To obtain a better IBR policy, we select an experiment that minimizes the MOCU remaining after applying its output to the network. At this point, we can either stop and find the resulting IBR policy or proceed to determine more unknown conditional probabilities via regulatory observation and find the IBR policy from the resulting posterior distribution. For sequential experimental design this entire process is iterated. Owing to the computational complexity of experimental design, which requires computation of many potential IBR policies, we implement an approximate method utilizing mean first passage times (MFPTs) – but only in experimental design, the final policy being an IBR policy.
Conclusions
Comprehensive performance analysis based on extensive simulations on synthetic and real GRNs demonstrate the efficacy of the proposed method, including the accuracy and computational advantage of the approximate MFPT-based design.
Keywords
Experimental design Gene regulatory networks Mean objective cost of uncertainty (MOCU) Network intervention Markov chains Dynamical interventionAbbreviations
- BNp
Boolean network with perturbation
- CPM
Conditional probability matrix
- ETPM
Expected transition probability matrix
- GRN
Gene regulatory network
- IBR
Intrinsically Bayesian robust
- MDP
Markov decision process
- MFPT
Mean first passage time
- MOCU
Mean objective cost of uncertainty
- TPM
Transition probability matrix
Background
A salient aim of translational genomics is to develop new drugs via constructing gene regulatory network (GRN) models characterizing the interactions among genes and then use these models to design therapeutic interventions. Most intervention strategies in the literature assume perfect knowledge regarding the network model. However, unfortunately this is not a realistic assumption in many real-world biomedical applications as uncertainty is inherent in genomics due to the complexity of biological systems, experimental limitations, noise, etc. Presence of model uncertainty degrades the performance of interventions.
Markovian genetic networks, an example of which are probabilistic Boolean networks (PBNs), have received great attention in recent years [1, 2, 3, 4, 5]. These networks have been shown to be effective in mimicking the behavior of biological systems, particularly as they are able to capture the randomness of biological phenomena by means of a transition probability matrix (TPM). The long-run behavior of a Markovian network is determined by a steady-state distribution over network states. Designing therapeutic interventions for these networks, often studied in the context of Markov decision processes (MDPs), has been extensively studied over the past two decades [6]. The basic assumption behind many intervention algorithms is that the TPM is perfectly known.
When dealing with network models possessing uncertainty, it is prudent to design a robust intervention that provides acceptable performance across an uncertainty class of possible models compatible with the current state of knowledge. In general, the problem of designing robust operators (or interventions in this paper) is typically viewed from two different perspectives: minimax robustness and Bayesian robustness. Under a minimax criterion, the robust operator has the best worst-case performance across the uncertainty class. The main problem with minimax robustness is that it is very conservative and gives too much attention to outlier models in the uncertainty class that may possess negligible likelihood.
Bayesian robustness addresses this issue by assigning a prior probability distribution reflecting existing knowledge about the model. Under this criterion, the aim is to find a robust operator possessing the optimal performance on average relative to this prior distribution. In the context of Bayesian robustness, when optimality is relative to the prior distribution, the resulting operator is called an intrinsically Bayesian robust (IBR) operator, examples being IBR Kalman filter in signal estimation [7], IBR signal compression [8], and IBR network structural intervention for gene regulatory networks [9, 10]. When optimality is relative to the posterior distribution obtained by incorporating observations into the prior distribution, the robust operator is called an optimal Bayesian operator [11, 12, 13, 14].
It is of prime interest to reduce model uncertainty via additional experiments and thereby improve the performance of the intervention. Since conducting all potential experiments is not feasible in many biomedical applications owing to operational constraints such as budget, time, and equipment limitations, it is imperative to utilize an experimental design strategy to rank experiments and then conduct only those experiments with high priority [15, 16, 17, 18].
As experiments are aimed at reducing uncertainty, a crucial step in experimental design is uncertainty quantification. From a translational perspective, we are not concerned with overall uncertainty, but rather with the degradation induced by the uncertainty in the intervention performance. Taking this into account, we employ an objective-based uncertainty quantification scheme called the mean objective cost of uncertainty (MOCU) [10]. MOCU has been successfully used for developing experimental design in gene regulatory networks when structural interventions are concerned [15, 19, 20, 21].
In this paper, we extend the application of the objective-based experimental design for GRNs to the realm of dynamical interventions. The interactions among genes are characterized by a set of conditional probability matrices where the conditional probabilities in each matrix correspond to the regulatory relationship between a gene and its regulating genes. We address the experimental design problem involving a GRN model in which a number of probabilities across conditional probability matrices are missing. Unknown conditional probabilities are represented by conjugate prior distributions which are closed under consecutive observations. In this paper, we show how the uncertainty in the conditional probabilities can be translated into the uncertainty in an unknown transition probability matrix. Furthermore, we show how additional information in terms of a trajectory of consecutive state transitions from the true system, if available, can be integrated to update prior distributions to posterior distributions containing lesser uncertainty. Deriving IBR control policies, which involves minimizing the average cost relative to the prior distribution among all stationary control policies, is at the very core of our experimental design calculations. In this regard, we take advantage of the fact that an IBR control policy can be derived by using an effective transition probability matrix that represents the uncertainty class of transition probability matrices. We should emphasize the optimality of the IBR control policy, which is selected from all possible stationary policies as opposed to the model-constrained Bayesian robust (MCBR) control policy [22], which is selected from among only the policies that are optimal for networks belonging to the uncertainty class.
It is worth mentioning that due to the computational complexity limitation, we are only concerned with stationary control policies in this paper. Another approach for designing a Bayesian robust control policy is to design a non-stationary policy, referred to as the optimal Bayesian robust (OBR) control policy. In addition to the expected immediate cost and different future costs obtained due to being in different states at the next time steps, an OBR policy also considers the effect of observations obtained by different actions on the sequence of different posterior distributions, which makes an OBR policy be non-stationary. In an OBR setting, the control problem is transformed into an equivalent problem in which each state, being referred to as a hyperstate, contains both the ordinary state of the system and the state of the knowledge reflecting the prior information and the history of observations from the system. Utilizing the concept of hyperstates for designing OBR control policies has roots in the classical works of Bellman and Kalaba [23], Silver [24], Gozzolino [25], and Martin [26]. The major obstacle of the OBR theory is its enormous computational complexity [27, 28, 29], such that it cannot be applied to networks of larger than 4 genes, even when only network control is concerned [29], let alone experimental design whose complexity is several-fold more than that of the control problem. Hence, taking into account complexity considerations with OBR, we focus on IBR stationary policies for our experimental design problem, which still requires massive computations but at a more tolerable cost compared to OBR policies.
To mitigate the computational complexity burden of experimental design, and considering the fact that computing the IBR control policy can be computationally demanding, we approximate it by using the method of mean first passage time (MFPT) [30]. The main motivation behind utilizing MFPT for controlling GRN networks is the desire to reach desirable states and leave undesirable states in the shortest time possible. Using this intuition, MFPT is used in [31] to derive a stationary control policy that can be used as an approximation for the optimal control policy and in [32] to find the best control gene. Using the concept of MFPT, we approximate the IBR control policy required for the experimental design and thereby lower the complexity of the experimental design. We emphasize that the MFPT approximation is only used for experimental design and that the implemented control policy will always be the optimal stationary control policy.
We summarize the main contributions of the paper. (1) Despite all the previous MOCU-based experimental design methods whose focus was on structural interventions [15, 19, 20], in this paper we consider the class of stationary interventions and derive a closed-form solution for the IBR stationary intervention when the TPM is unknown. (2) While in the previous works, uncertain parameters involve a number of regulatory edges between genes, in this paper we consider the case that a number of conditional probabilities characterizing regulatory relationships between genes are unknown. Given that conditional probabilities can be estimated using time series gene expression data generated through a biological experiment, it is more realistic to consider these probabilities, rather than regulatory edges, as the outcomes of biological experiments. This new uncertainty assumption requires us to define a new uncertainty class and prior probability model. (3) To address the complexity concerns of the proposed method, we propose an approximate experimental design method utilizing mean first passage times (MFPTs) in which we extend the application of MFPT-based controls to unknown TPMs.
Methods
Markovian regulatory networks
In a network with n genes, a set of binary variables V={X_{1},X_{2},…,X_{n}}, X_{i}∈{0,1}, determines the expression states of the genes. The vector of gene expression values at time t, X(t)=(X_{1}(t),…,X_{n}(t)), referred to as the gene activity profile (GAP), defines the network state at each time step. In a Markovian regulatory network, network dynamics involves a trajectory of states over time governed by the transition rule X(t+1)=f(X(t),w(t)), t≥0, where w(t)∈Ξ captures randomness in the system and \(f:\mathcal {S}\times \Xi \rightarrow \mathcal {S}\), \(\mathcal {S}=\left \{0,1,\dots,2^{n}-1\right \}\) being the set of corresponding decimal representations for the network states, is a mapping that characterizes the state transitions in the network. The sequence of states over time can be viewed as a Markov chain characterized by a transition probability matrix (TPM) \(\mathbf {P}=[P_{ij}]_{i,j=0}^{2^{n}-1}\), where P_{ij}=Pr[X(t+1)=j|X(t)=i], Pr[ ·] being the probability operator. An ergodic Markov chain is guaranteed to possess a unique steady-state distribution π, such that π^{T}=π^{T}P, T being the transpose operator.
where j_{k} is the binary value of the k-th gene in state j and \(\Gamma _{X_{k}}[i]\) is the vector of binary values of the regulating genes for X_{k} extracted from the representation of state i. For example, consider a 3-gene network, n=3, in which gene X_{1} (k=1 in (2)) is regulated by genes X_{2} and X_{3}. For this network, when computing P_{14} (i=1 and j=4 in (2)), j_{k}=1 (as X_{1}=1 for j=4) and \(\Gamma _{X_{1}}[i]=(0,1)\) (as (X_{2},X_{3})=(0,1) for i=1).
From a translational perspective, the states of a network can be partitioned into two sets: desirable states \(\mathcal {D}\), being associated with healthy phenotypes, and undesirable states \(\mathcal {U}\), corresponding to pathological cell functions such as cancer. The goal of therapeutic interventions is to alter the dynamical behavior of the network in such a way as to reduce the steady-state probability \(\pi _{\mathcal {U}}={\sum \nolimits }_{i\in \mathcal {U}}\pi _{i}\) of the network entering the undesirable states. There are two different approaches for network interventions: structural interventions and dynamical interventions. In a structural intervention, the goal is to modify the dynamical behavior of the network via a one-time change in its underlying regulatory structure [9, 33, 34, 35]. Dynamical interventions are typically studied in the framework of Markov decision processes and are characterized by control policies. These interventions usually involve the change in the expression of one or more genes, being called control genes, and can be applied either over a finite-time [36, 37, 38] or an infinite-time horizon [31, 39].
Optimal dynamical control
where states \(\tilde {i}\) and i differ only in the value of gene g.
The optimal cost J^{∗} =(J^{∗}(0),…,J^{∗}(2^{n}−1)) is the unique solution of (7) among all bounded functions and the control policy μ^{∗} that attains the minimum in (7) is stationary, i.e., μ^{∗}=(μ^{∗},μ^{∗},..) [40]. In order to find the fixed point in the Bellman’s optimality equation and thereby find the optimal control policy, dynamic programming algorithms, including the value iteration algorithm that iteratively estimates the cost function, can be used.
MOCU-based optimal experimental design framework
In this section, we review the general framework of the experimental design method in [15], which is based on the concept of the mean objective cost of uncertainty (MOCU) [10]. Let θ=(θ_{1},θ_{2},…,θ_{T}) be composed of T uncertain parameters in a network model. The set of all possible realizations for θ is denoted by Θ and is called an uncertainty class. A prior distribution f(θ) is assigned to θ, which reflects the likelihood of each realization of θ being the true value.
where the expectation is taken relative to the prior distribution f(θ).
MOCU measures the model uncertainty in terms of the expected increased error due to applying an IBR intervention (the chosen intervention in the presence of uncertainty) instead of an optimal intervention (the chosen intervention in the absence of uncertainty). Uncertainty quantification based on MOCU can lay the groundwork for objective-based experimental design.
A number of experimental design methods based on the MOCU framework have been proposed in the literature [15, 19, 20]. In all of these cases, the MOCU-based experimental design can reduce the number of needed experiments significantly in comparison to other selection policies such as entropy-based experimental design, pure exploitation, or random selection policy.
Uncertainty in transition probability matrix
Assume that regulatory information between a gene and its regulating genes is missing for a number of genes in the network. In other words, a number of rows in the n conditional probability matrices are unknown. We represent unknown conditional probabilities by a set of random variables θ=(θ_{1},θ_{2},…,θ_{T}). Since each row of the CPM adds up to one, i.e., C_{j,1}(X_{i})+C_{j,2}(X_{i})=1, there is only one degree of freedom. The uncertainty in the CPMs will eventually show up in the corresponding TPM and thereby can affect the performance of the control policy. Therefore, it is of interest to reduce the uncertainty in the CPMs. We seek an experimental design method that efficiently guides us on which unknown conditional probability to determine first.
where B(α_{i},β_{i}) is the beta function. The expected value of θ_{i} ∼ Beta(α_{i},β_{i}) is \(\mathrm {E}[\theta _{i}]=\frac {\alpha _{i}}{\alpha _{i}+\beta _{i}}\). When α_{i} = β_{i}=1, the beta distribution becomes a uniform distribution over interval [0,1].
Since the uncertainty of an unknown conditional probability is governed by the corresponding terms α^{′}, β^{′}, and given the fact that observation(s) can potentially increase α’s and β’s according to (17) and (18), availability of a state trajectory \(\mathcal {X}_{L}\) is equivalent to a lesser initial uncertainty, and hence a simpler experimental design problem.
Optimal experimental design for determining unknown conditional probabilities
and the minimum is attained by the IBR control policy \(\mu ^{\Theta }(X_{0})=\left (\mu ^{\Theta }_{1}(X_{0}),\mu _{2}^{\Theta }(X_{0}),\dots \right)\).
where Pr_{θ}[ ·] is the probability operator relative to θ and E_{θ}[ ·] is taken relative to the updated prior (posterior) distribution \(f(\boldsymbol {\theta }|\mathcal {X}_{L})\). The ETPM P^{Θ} represents the uncertainty class of TPMs and enables finding the IBR control policy μ^{Θ} for an uncertainty class of TPMs in the same way that the optimal control policy for a known TPM is found. Since the θ_{i}’s are independent, the expectation can be brought inside the product. Each conditional probability term in (23) is either known, whose known value is used for multiplication, or is unknown and corresponds to an unknown parameter θ_{i} whose expected value obtained according to (19) is used in the multiplication.
The following three theorems, whose proofs are similar to those in [40] relative to a known TPM, lay out the theoretical foundation for finding the IBR control policy.
Theorem 1
(Convergence of the algorithm) Letting \(J:\mathcal {S}\rightarrow \mathcal {R}\) be a bounded function, for any \(i\in \mathcal {S}\), the optimal average cost function J^{Θ}(i) satisfies \(J^{\Theta }(i)={\lim }_{M\rightarrow \infty }T^{M}J(i)\phantom {\dot {i}\!}\).
Theorem 2
Theorem 3
(Necessary and sufficient condition) A stationary policy μ^{Θ} is an IBR control policy if and only if for each \(i\in \mathcal {S}\), μ^{Θ}(i) attains the minimum in Bellman’s optimality equation.
Based on Theorem 1, J^{Θ} can be computed recursively using the value iteration algorithm in the same way that this algorithm is used to find the optimal control policy for a known TPM. The converged cost satisfies Bellman’s optimality equation (Theorem 2). Also, the corresponding policy is a stationary IBR control policy (Theorem 3). μ^{Θ} attains the minimum in the Bellman’s optimality equation, where the ordinary TPM is replaced by the ETPM.
The concept of effective quantities has also been used for deriving IBR operators in other problems: for example in [7], effective noise statistics are used to derive the IBR Kalman filter; or in [8], effective covariance matrix is used for achieving IBR signal compression.
where the IBR control policy for the reduced uncertainty class \(\phantom {\dot {i}\!}\Theta |\left (\theta _{i}=\theta ^{\prime }_{i}\right)\) is found using the ETPM \(\mathbf {P}^{\Theta |\theta _{i}=\theta ^{\prime }_{i}}\phantom {\dot {i}\!}\) obtained relative to the conditional probability distribution \(f\left (\boldsymbol {\theta }|\theta _{i}=\theta ^{\prime }_{i}\right)\phantom {\dot {i}\!}\).
According to (28), to evaluate the determination of each unknown parameter θ_{i}, for each realization \(\theta ^{\prime }_{i}\) of θ_{i}, we need to obtain the average cost of the IBR control policy \(\mu ^{\Theta |\theta _{i}=\theta ^{\prime }_{i}}\phantom {\dot {i}\!}\) across the reduced uncertainty class \(\Theta |\left (\theta _{i}=\theta ^{\prime }_{i}\right)\phantom {\dot {i}\!}\) and then take the average of all these average costs relative to the marginal distribution of parameter θ_{i}. In practice, the expression in (28) is approximated via Monte-Carlo simulations. We draw a number of samples from the marginal distribution of θ_{i} and then approximate the expression being minimized in (28) as the average of all inner expectations computed for each generated sample. The steps required for obtaining the primary parameter \(\phantom {\dot {i}\!}\theta _{i^{\ast }}\) are summarized in Algorithm 1. The inputs to this algorithm are n CPMs characterizing the GRN, T unknown parameters θ_{i} corresponding to unknown conditional probabilities, hyperparameters (α_{i},β_{i}) for the prior beta distributions, the state trajectory \(\mathcal {X}_{L}\), and ζ, r, and I, which determine the discount factor, cost function, and the number of iterations for value iteration, respectively.
Finding IBR control policies for an uncertainty class (like finding the optimal control policy for a known TPM [41]) is computationally expensive and the complexity grows exponentially with the number of genes. Therefore, most of the computational burden of the experimental design is in finding IBR control policies. To mitigate the complexity of the proposed method, in the next section we propose an approximate method for computing IBR control policies.
Approximate experimental design based on MFPT
The mean first passage time (MFPT) from state i to j measures how long it would take on average that the network transitions from state i to state j.
where Δ in (32) and (33) is a tuning parameter that should be adjusted based on the definition for the cost function r(i,j,u).
In the spirit of the MFPT-based approximation for optimal control, we approximate the IBR control policy needed in experimental design via MFPT. Taking into account that the IBR control policy μ^{Θ} is in fact the optimal control policy relative to the ETPM and that MFPT can be used as an approximation for the optimal control policy, we approximate the IBR control policy by finding the MFPT-based control policy relative to the ETPM and denote it by \(\mu ^{\text {MFPT}}_{\Theta }\), i.e.,
\(\mu ^{\text {MFPT}}_{\mathbf {P}^{\Theta }}\) is obtained by solving (29)-(33) for the effective transition probability matrix P^{Θ}. When we approximate the IBR control policy via MFPT in experimental design, the average cost needed in (28) is computed via Monte-Carlo simulations (over different initial network states) as
where N is the total number of simulations and r^{(n)}(·) is the accrued total discounted cost in the n-th simulation. The pseudo-code for the approximate method is the same as Algorithm 1 except for steps 19 and 20. For the approximate method, in step 19, we find \(\mu ^{\text {MFPT}}_{\Theta |\hat {\theta }_{i}}\) via the MFPT approach. Then in step 20, we plug \(\mu ^{\text {MFPT}}_{\Theta |\hat {\theta }_{i}}\), obtained from step 19, in (35) to compute \(\eta (\hat {\theta }_{i})\).
Having used the MFPT approach for the sole purpose of reducing the uncertainty class, the IBR control policy is obtained by solving Bellman’s equation using value iteration.
Computational complexity analysis
The computationally demanding step in the proposed experimental design method is to find IBR control policies. If there are T different unknown parameters and we generate M different samples for Monte-Carlo simulations for each one, then we need to find IBR control policies in our experimental design calculations T×M times. Since we use value iterations to solve Bellman’s equation, if we assume that the value iteration converges in I iterations, then we need to compute \((|\mathcal {S}|\times |\mathcal {C}|)^{I}\) terminal costs to obtain the control policy, \(|\mathcal {S}|\) being the number of network states and \(|\mathcal {C}|\) being the number of control actions. In this paper, we focus on binary networks and binary control actions, i.e., \(|\mathcal {S}|=2^{n}\), n being the number of genes and \(\mathcal {C}=2\). Therefore, the order of complexity when experimental design based on the IBR control policy is implemented is \(\mathcal {O}\left (T\times M\times (2^{n+1})^{I}\right)\). The complexity grows exponentially with the number of genes and polynomially with the number of unknown parameters.
The complexity of the approximate experimental design is much lower because there is no iterative calculation in MFPT. It is enough to solve the two linear equations in (30) and (31), which involves two matrix inversions. Although applying MFPT for experimental design requires us to find the average cost of the MFPT-based IBR control policy via Monte-Carlo simulations, this overhead complexity for MFPT is not concerning and still the complexity of the MFPT-based approach is much smaller in comparison to that of the optimal experimental design. Since the calculations for each unknown parameter and each realization of that parameter can be done independently, a parallel implementation can be used for the proposed experimental design methods.
Results
In this section, we study the performance of the proposed methods based on synthetic and real gene networks. As a class of Markovian regulatory network, we consider Boolean networks with perturbation for the simulations.
Boolean networks with perturbation
An n-gene Boolean network is defined by a set of binary variables V={X_{1},X_{2},..,X_{n}}, and a set of Boolean functions F={f_{1},f_{2},…,f_{n}}, where \(f_{i}\!:\{0,1\}^{k_{i}}\rightarrow \{0,1\}\) determines the value of gene X_{i} when it has k_{i} regulating genes. The transition rule X(t+1)=F(X(t)) governs the evolution of states over time. In a Boolean network with perturbation (BNp), each gene may flip its value with a small perturbation probability p. In this network, the next state at time t+1 is F(X(t)) with probability (1−p)^{n} or F(X(t))⊕γ with probability 1−(1−p)^{n}, where γ is a binary vector of size n and ⊕ is the component-wise addition modulo 2 operator. The underlying state evolution of a BNp over time can be viewed as a Markov chain with a transition probability matrix P. The TPM can be derived using the regulatory structure of the network and the perturbation probability [22]. When p > 0 the Markov chain is guaranteed to possess a unique steady-state distribution π.
Synthetic networks
We first randomly generate a number of BNps and then from the corresponding TPMs we extract the set of conditional probabilities for each gene in the network. We consider BNps with 6 genes. The number of regulating genes for each gene is set to 2 and they are randomly selected from the set of genes. Therefore, the size of the CPM for each gene is 4×2. The bias (probability) that a Boolean function takes on the value 1 is randomly selected from a beta distribution with variance 0.0001 and mean 0.5. The perturbation probability p is set to 0.01. We use this protocol to generate 100 random BNps from which we generate 100 different sets of CPMs.
where \(\Gamma _{X_{i}}[\!S]=j\) and \(S^{\prime }_{i}=1\). Since more than one row in a TPM might correspond to \(\Gamma _{X_{i}}=j\), in order to have a consistent procedure for extracting conditional probabilities, we take the average of all the values found for the rows corresponding to \(\Gamma _{X_{i}}=j\).
This cost function reflects penalties assigned to undesirable states and also to the transitions to which the control action is applied. The discount factor ζ is set to 0.2. The tuning parameter Δ for the MFPT method is set to Δ=0.3. We use value iteration with 4 iterations to find the control policies in the optimal design method and for evaluating chosen experiments. All initial beta distributions for unknown conditional probabilities θ_{i} in the network are Beta(1,1), a uniform distribution. We run simulations for different numbers L of initial data used for updating priors.
In the first set of simulations, we generate 100 synthetic BNps. After extracting corresponding conditional probabilities for each network, we randomly select 5 conditional probabilities in each network and assume they are unknown. The aim is to decide which unknown conditional probability should be determined first. For each network, we generate a state trajectory \(\mathcal {X}_{L}=\{\mathbf {X}(0),\dots,\mathbf {X}(L)\}\), used for updating initial hyperparameters, by simulating the underlying true network.
In the simulations, when we want to evaluate the determination of an unknown probability θ_{i}, we put back its true value ϕ_{i}, which was discarded during experimental design calculations, in the network, thereby resulting in a new uncertainty class Θ|(θ_{i}=ϕ_{i}) of remaining unknown probabilities. We find the IBR control policy \(\mu ^{\Theta |(\theta _{i}=\phi _{i})}\phantom {\dot {i}\!}\) for this new uncertainty class by solving Bellman’s equation relative to \(\mathbf {P}^{\Theta |(\theta _{i}=\phi _{i})}\phantom {\dot {i}\!}\). We then apply \(\phantom {\dot {i}\!}\mu ^{\Theta |(\theta _{i}=\phi _{i})}\) to the underlying true network and run the controlled network until the horizon length 6 according to the underlying true TPM and \(\mu ^{\Theta |(\theta _{i}=\phi _{i})}\phantom {\dot {i}\!}\), and record the cost at each time based on the network state at that time and the cost function r(i,j,c) in (38). Then we compute the total discounted cost over the horizon by accumulating the costs incurred over the horizon length according to the discount factor ζ. We repeat this process of calculating the total discounted cost for 10,000 iterations over different network initial states X_{0} and state transition paths. Note that although the underlying controlled TPM is fixed, there are still different state transition paths over the horizon due to the randomness characterized by the TPM. We represent the cost corresponding to determining parameter θ_{i} as the average of all 10,000 total discounted costs and denote it by J(θ_{i}). For comparing different experimental design approaches, we report the average of J(θ_{i}) over 100 generated synthetic networks and 100 different sets of assumed true values for the unknown probabilities in each network drawn from the beta prior distributions.
Comparison of the ranked experiments according to the optimal and approximate methods
\(\mathcal {E}_{1^{\prime }}\) | \(\mathcal {E}_{2^{\prime }}\) | \(\mathcal {E}_{3^{\prime }}\) | \(\mathcal {E}_{4^{\prime }}\) | \(\mathcal {E}_{5^{\prime }}\) | |
---|---|---|---|---|---|
(a) L=0 (no initial data) | |||||
Optimal | 1.2215 | 1.2305 | 1.2399 | 1.2433 | 1.2427 |
Approximate | 1.2246 | 1.2340 | 1.2388 | 1.2390 | 1.2416 |
(b) L=10 | |||||
Optimal | 1.1615 | 1.1736 | 1.1780 | 1.1790 | 1.1792 |
Approximate | 1.1646 | 1.1747 | 1.1767 | 1.1775 | 1.1779 |
(c) L=20 | |||||
Optimal | 1.1573 | 1.1660 | 1.1706 | 1.1720 | 1.1705 |
Approximate | 1.1598 | 1.1665 | 1.1701 | 1.1704 | 1.1695 |
(d) L=50 | |||||
Optimal | 1.1463 | 1.1534 | 1.1558 | 1.1560 | 1.1561 |
Approximate | 1.1487 | 1.1529 | 1.1557 | 1.1547 | 1.1557 |
The comparison of the average costs obtained after choosing the experiment via different selection policies
L=0 | L=10 | L=20 | L=50 | |
---|---|---|---|---|
J(θ_{rnd}) | 1.2350 | 1.1743 | 1.1673 | 1.1535 |
J(θ_{approx}) | 1.2246 | 1.1646 | 1.1598 | 1.1487 |
J(θ_{opt}) | 1.2215 | 1.1615 | 1.1573 | 1.1463 |
The percentage of success, failure, and tie for performing the chosen experiment rather than the suboptimal experiments
\(\mathcal {E}_{1^{\prime }}\sim \mathcal {E}_{2^{\prime }} \) | \(\mathcal {E}_{1^{\prime }}\sim \mathcal {E}_{3^{\prime }} \) | \(\mathcal {E}_{1^{\prime }}\sim \mathcal {E}_{4^{\prime }} \) | \(\mathcal {E}_{1^{\prime }}\sim \mathcal {E}_{5^{\prime }} \) | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
S | F | T | S | F | T | S | F | T | S | F | T | |
(a) L=0 (no initial data) | ||||||||||||
Optimal | 49.8 | 45.3 | 4.9 | 54.9 | 39.8 | 5.3 | 55.2 | 39.6 | 5.1 | 55.9 | 38.7 | 5.4 |
Approximate | 50.0 | 44.6 | 5.4 | 51.0 | 43.4 | 5.6 | 51.8 | 42.4 | 5.8 | 52.1 | 42.5 | 5.4 |
(b) L=10 | ||||||||||||
Optimal | 54.0 | 40.9 | 5.1 | 55.3 | 38.6 | 6.0 | 56.4 | 37.8 | 5.8 | 56.8 | 37.7 | 5.5 |
Approximate | 50.5 | 43.2 | 6.3 | 52.0 | 42.5 | 5.5 | 53 | 41.0 | 6.0 | 52.8 | 41.0 | 6.2 |
(c) L=20 | ||||||||||||
Optimal | 50.4 | 43.8 | 5.8 | 52.1 | 41.5 | 6.4 | 52.8 | 41.4 | 5.8 | 53.9 | 40.3 | 5.7 |
Approximate | 50.0 | 44.2 | 5.8 | 50.8 | 42.4 | 6.8 | 51.2 | 42.8 | 6.0 | 51.4 | 42.3 | 6.3 |
(d) L=50 | ||||||||||||
Optimal | 50.1 | 43.2 | 6.7 | 52.2 | 41.2 | 6.6 | 52.9 | 40.0 | 7.1 | 52.3 | 41.8 | 5.9 |
Approximate | 48.7 | 44.5 | 6.8 | 51.0 | 41.7 | 7.3 | 50.0 | 43.6 | 6.4 | 50.8 | 42.9 | 6.3 |
Real network example: TP53 pathways
Real network example: mammalian cell cycle
The set of Boolean functions for a mutated cell cycle [46]
Gene | Node | Boolean function |
---|---|---|
CycD | X _{1} | Extracellular signal |
Rb | X _{2} | \((\overline {X_{1}}\wedge \overline {X_{4}}\wedge \overline {X_{9}}\wedge \overline {X_{8}})\) |
E2F | X _{3} | \((\overline {X_{2}}\wedge \overline {X_{9}}\wedge \overline {X_{8}})\) |
CycE | X _{4} | \((X_{3}\wedge \overline {X_{2}})\) |
Cdc20 | X _{5} | X _{8} |
Cdh1 | X _{6} | \((\overline {X_{9}}\wedge \overline {X_{8}})\vee X_{5}\) |
UbcH10 | X _{7} | \(\overline {X_{6}}\vee (X_{6}\wedge X_{7} \wedge (X_{5}\vee X_{9}\vee X_{8}))\) |
CycB | X _{8} | \((\overline {X_{5}}\wedge \overline {X_{6}})\) |
CycA | X _{9} | \((X_{3}\wedge \overline {X_{2}}\wedge \overline {X_{5}}\wedge (\overline {X_{6}}\wedge \overline {X_{7}}))\) |
\(\vee (X_{9}\wedge \overline {X_{2}}\wedge \overline {X_{5}}\wedge (\overline {X_{6}\wedge X_{7}}))\) |
Discussion
An inherent problem of dealing with stationary control policies in a Markovian network is computational complexity, which is due to the exponential increase of the number of states with the network size. We have been able to mitigate the complexity of the experimental design and thereby push the size limit (as demonstrated in Fig. 2) by proposing an approximate experimental design approach based on mean first passage time. However, we believe that more complexity reduction should be achieved for addressing experimental design for extremely large gene networks. Considering these intrinsic computational issues, we plan to find more efficient approximations and investigate accelerated implementation of the method via efficient computer architecture platforms, such as Graphic Processing Unit (GPU).
Another consideration is the accuracy of the prior distributions used in the experimental design calculation. The performance of the experimental design depends on the degree to which the prior probabilities can describe the existing knowledge regarding uncertain parameters accurately. The problem of finding optimal prior probabilities in this context can be solved under a prior construction optimization framework, which involves constructing a mapping that transforms signaling relations to constraints on the prior distribution. Constructing optimal priors has been done for genomic classification [14, 47]. In future work, we aim to develop an optimization framework to address prior construction for experimental design in gene regulatory networks.
Conclusions
Given the complexity of biological systems and the cost of experiments, experimental design is of great practical significance in translational genomics. In this paper, we address the problem of optimal experimental design for gene regulatory networks controlled with stationary control policies. The proposed experimental design framework is based on the notion of mean objective cost of uncertainty, which views model uncertainty in terms of the increased cost it induces. Future work includes further reducing the computational cost of the method and also designing optimization frameworks for constructing optimal prior distributions. Also, another avenue of research is to implement an integrative experimental design method that can utilize the RNA-Seq data for the genes on the same pathway [48] for optimal uncertainty reduction.
Notes
Acknowledgments
The authors would like to acknowledge Texas A&M High Performance Research Computing for providing computational resources to perform simulations in this paper.
Funding
Publication costs for this article were funded by the senior author’s institution.
Availability of data and materials
Data and MATLAB source code are available from the corresponding author upon request.
About this supplement
This article has been published as part of BMC Systems Biology Volume 12 Supplement 8, 2018: Selected articles from the International Conference on Intelligent Biology and Medicine (ICIBM) 2018: systems biology. The full contents of the supplement are available online at https://bmcsystbiol.biomedcentral.com/articles/supplements/volume-12-supplement-8.
Authors’ contributions
RD conceived the method, developed the algorithm, performed the simulations, analyzed the results, and wrote the first draft. MSE helped with the prior update step, analyzed the results, and commented on the manuscript. ERD conceived the method, oversaw the project, analyzed the results, and edited the manuscript. All authors have read and approved the final manuscript.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Kim S, Li H, Dougherty ER, Cao N, Chen Y, Bittner M, Suh EB. Can Markov chain models mimic biological regulation?J Biol Syst. 2002; 10(04):337–57.CrossRefGoogle Scholar
- 2.Zhao W, Serpedin E, Dougherty ER. Inferring gene regulatory networks from time series data using the minimum description length principle. Bioinformatics. 2006; 22(17):2129–35.CrossRefGoogle Scholar
- 3.Friedman N, Murphy K, Russell S. Learning the structure of dynamic probabilistic networks. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers Inc.: 1998. p. 139–147.Google Scholar
- 4.Chen PC, Chen JW. A Markovian approach to the control of genetic regulatory networks. Biosystems. 2007; 90(2):535–45.CrossRefGoogle Scholar
- 5.Liang J, Han J. Stochastic Boolean networks: An efficient approach to modeling gene regulatory networks. BMC Syst Biol. 2012; 6(1):113.CrossRefGoogle Scholar
- 6.Datta A, Pal R, Choudhary A, Dougherty ER. Control approaches for probabilistic gene regulatory networks-what approaches have been developed for addreassinig the issue of intervention?IEEE Signal Process Mag. 2007; 24(1):54–63.CrossRefGoogle Scholar
- 7.Dehghannasiri R, Esfahani MS, Dougherty ER. Intrinsically bayesian robust kalman filter: An innovation process approach. IEEE Trans Signal Process. 2017; 65(10):2531–46.CrossRefGoogle Scholar
- 8.Dehghannasiri R, Qian X, Dougherty ER. Intrinsically Bayesian robust Karhunen-Loeve compression. Signal Process. 2018; 144:311–22.CrossRefGoogle Scholar
- 9.Xiao Y, Dougherty ER. The impact of function perturbations in boolean networks. Bioinformatics. 2007; 23(10):1265–73.CrossRefGoogle Scholar
- 10.Yoon BJ, Qian X, Dougherty ER. Quantifying the objective cost of uncertainty in complex dynamical systems. IEEE Trans Signal Process. 2013; 61(9):2256–66.CrossRefGoogle Scholar
- 11.Qian X, Dougherty ER. Bayesian regression with network prior: Optimal bayesian filtering perspective. IEEE Trans Signal Process. 2016; 64(23):6243–53.CrossRefGoogle Scholar
- 12.Broumand A, Esfahani MS, Yoon B-J, Dougherty ER. Discrete optimal Bayesian classification with error-conditioned sequential sampling. Pattern Recog. 2015; 48(11):3766–82.CrossRefGoogle Scholar
- 13.Dehghannasiri R, Qian X, Dougherty ER. A Bayesian robust Kalman smoothing framework for state-space models with uncertain noise statistics. EURASIP J Adv Signal Process. 2018; 2018(1):55.CrossRefGoogle Scholar
- 14.Boluki S, Esfahani MS, Qian X, Dougherty ER. Constructing pathway-based priors within a gaussian mixture model for Bayesian regression and classification. IEEE/ACM Trans Comput Biol Bioinf. 2017. https://doi.org/10.1109/TCBB.2017.2778715.
- 15.Dehghannasiri R, Yoon B-J, Dougherty ER. Optimal experimental design for gene regulatory networks in the presence of uncertainty. IEEE/ACM Trans Comput Biol Bioinf. 2015; 12(4):938–50.CrossRefGoogle Scholar
- 16.Sverchkov Y, Craven M. A review of active learning approaches to experimental design for uncovering biological networks. PLoS Comput Biol. 2017; 13(6):1005466.CrossRefGoogle Scholar
- 17.Kim M, Tagkopoulos I. Data integration and predictive modeling methods for multi-omics datasets. Mol Omics. 2018; 14(1):8–25.CrossRefGoogle Scholar
- 18.Steiert B, Raue A, Timmer J, Kreutz C. Experimental design for parameter estimation of gene regulatory networks. PloS ONE. 2012; 7(7):40052.CrossRefGoogle Scholar
- 19.Dehghannasiri R, Yoon B-J, Dougherty ER. Efficient experimental design for uncertainty reduction in gene regulatory networks. BMC Bioinformatics. 2015; 16(13):2.CrossRefGoogle Scholar
- 20.Mohsenizadeh D, Dehghannasiri R, Dougherty E. Optimal objective-based experimental design for uncertain dynamical gene networks with experimental error. IEEE/ACM Trans Comput Biol Bioinf. 2018; 15(1):218–30.CrossRefGoogle Scholar
- 21.Imani M, Dehghannasiri R, Braga-Neto UM, Dougherty ER. Sequential experimental design for optimal structural intervention in gene regulatory networks based on the mean objective cost of uncertainty. Cancer Informat. 2018; 17:1–10.CrossRefGoogle Scholar
- 22.Pal R, Datta A, Dougherty ER. Robust intervention in probabilistic boolean networks. IEEE Trans Signal Process. 2008; 56(3):1280–94.CrossRefGoogle Scholar
- 23.Bellman R, Kalaba R. Dynamic programming and adaptive processes: Mathematical foundation. IRE Trans Autom Control. 1960; AC-5(1):5–10.CrossRefGoogle Scholar
- 24.Silver EA. Markovian decision processes with uncertain transition probabilities or rewards: MIT; 1963.Google Scholar
- 25.Gozzolino JM, Gonzalez-Zubieta R, Miller RL. Markovian decision processes with uncertain transition probabilities. Technical report. 1965.Google Scholar
- 26.Martin JJ. Bayesian Decision Problems and Markov Chains. New York: Wiley; 1967.Google Scholar
- 27.Kumar P. A survey of some results in stochastic adaptive control. SIAM J Control Optim. 1985; 23(3):329–80.CrossRefGoogle Scholar
- 28.Van Hee KM, Hee K. Bayesian Control of Markov Chains vol. 95. Amsterdam: Mathematisch Centrum; 1978.Google Scholar
- 29.Yousefi MR, Dougherty ER. A comparison study of optimal and suboptimal intervention policies for gene regulatory networks in the presence of uncertainty. EURASIP J Bioinf Syst Biol. 2014; 2014(1):6.CrossRefGoogle Scholar
- 30.Norris JR. Markov Chains vol. 2. Cambridge: Cambridge university press; 1998.Google Scholar
- 31.Vahedi G, Faryabi B, Chamberland J, Datta A, Dougherty ER. Intervention in gene regulatory networks via a stationary mean-first-passage-time control policy. IEEE Trans Biomed Eng. 2008; 55(10):2319–31.CrossRefGoogle Scholar
- 32.Shmulevich I, Dougherty ER, Zhang W. Gene perturbation and intervention in probabilistic boolean networks. Bioinformatics. 2002; 18(10):1319–31.CrossRefGoogle Scholar
- 33.Shmulevich I, Dougherty ER, Zhang W. Control of stationary behavior in probabilistic boolean networks by means of structural intervention. Biol Syst. 2002; 10(4):431–46.CrossRefGoogle Scholar
- 34.Qian X, Dougherty ER. Effect of function perturbation on the steady-state distribution of genetic regulatory networks: Optimal structural intervention. IEEE Trans Signal Process. 2008; 56(10):4966–76.CrossRefGoogle Scholar
- 35.Hu M, Shen L, Zan X, Shang X, Liu W. An efficient algorithm to identify the optimal one-bit perturbation based on the basin-of-state size of boolean networks. Sci Rep. 2016; 6(26247):1–11.Google Scholar
- 36.Datta A, Choudhary A, Bittner ML, Dougherty ER. External control in markovian genetic regulatory networks. Mach Learn. 2003; 52(1-2):169–91.CrossRefGoogle Scholar
- 37.Yang C, Wai-Ki C, Nam-Kiu T, Ho-Yin L. On finite-horizon control of genetic regulatory networks with multiple hard-constraints. BMC Syst Biol. 2010; 4(2):14.CrossRefGoogle Scholar
- 38.Ching W-K, Zhang S-Q, Jiao Y, Akutsu T, Tsing N-K, Wong A. Optimal control policy for probabilistic boolean networks with hard constraints. IET Syst Biol. 2009; 3(2):90–9.CrossRefGoogle Scholar
- 39.Pal R, Datta A, Dougherty ER. Optimal infinite-horizon control for probabilistic boolean networks. IEEE Trans Signal Process. 2006; 54(6):2375–87.CrossRefGoogle Scholar
- 40.Bertsekas DP. Dynamic Programming and Optimal Control vol. 1. Belmont: Athena Scientific; 1995.Google Scholar
- 41.Akutsu T, Hayashida M, Ching W-K, Ng MK. Control of Boolean networks: Hardness results and algorithms for tree structured networks. J Theor Biol. 2007; 244(4):670–9.CrossRefGoogle Scholar
- 42.Batchelor E, Loewer A, Lahav G. The ups and downs of p53: understanding protein dynamics in single cells. Nat Rev Cancer. 2009; 9(5):371.CrossRefGoogle Scholar
- 43.Weinberg R. The Biology of Cancer. Princeton: Garland Science; 2007.Google Scholar
- 44.Leach FS, Tokino T, Meltzer P, Burrell M, Oliner JD, Smith S, Hill DE, Sidransky D, Kinzler KW, Vogelstein B. p53 mutation and MDM2 amplification in human soft tissue sarcomas. Cancer Res. 1993; 53(10):2231–4.Google Scholar
- 45.Reis RM, Könü-Lebleblicioglu D, Lopes JM, Kleihues P, Ohgaki H. Genetic profile of gliosarcomas. Am J Pathol. 2000; 156(2):425–32.CrossRefGoogle Scholar
- 46.Fauré A, Naldi A, Chaouiya C, Thieffry D. Dynamical analysis of a generic boolean model for the control of the mammalian cell cycle. Bioinformatics. 2006; 22(14):124–31.CrossRefGoogle Scholar
- 47.Boluki S, Esfahani MS, Qian X, Dougherty ER. Incorporating biological prior knowledge for bayesian learning via maximal knowledge-driven information priors. BMC Bioinformatics. 2017; 18(14):552.CrossRefGoogle Scholar
- 48.Broumand A, Hu T. A length bias corrected likelihood ratio test for the detection of differentially expressed pathways in RNA-Seq data. In: IEEE Global Conference on Signal and Information Processing. IEEE: 2015. p. 1145–9.Google Scholar
Copyright information
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.