A whitening approach to probabilistic canonical correlation analysis for omics data integration
Abstract
Background
Canonical correlation analysis (CCA) is a classic statistical tool for investigating complex multivariate data. Correspondingly, it has found many diverse applications, ranging from molecular biology and medicine to social science and finance. Intriguingly, despite the importance and pervasiveness of CCA, only recently a probabilistic understanding of CCA is developing, moving from an algorithmic to a modelbased perspective and enabling its application to largescale settings.
Results
Here, we revisit CCA from the perspective of statistical whitening of random variables and propose a simple yet flexible probabilistic model for CCA in the form of a twolayer latent variable generative model. The advantages of this variant of probabilistic CCA include nonambiguity of the latent variables, provisions for negative canonical correlations, possibility of nonnormal generative variables, as well as ease of interpretation on all levels of the model. In addition, we show that it lends itself to computationally efficient estimation in highdimensional settings using regularized inference. We test our approach to CCA analysis in simulations and apply it to two omics data sets illustrating the integration of gene expression data, lipid concentrations and methylation levels.
Conclusions
Our whitening approach to CCA provides a unifying perspective on CCA, linking together sphering procedures, multivariate regression and corresponding probabilistic generative models. Furthermore, we offer an efficient computer implementation in the “whitening” R package available at https://CRAN.Rproject.org/package=whitening.
Keywords
Multivariate analysis Probabilistic canonical correlation analysis Data integrationBackground
Canonical correlation analysis (CCA) is a classic and highly versatile statistical approach to investigate the linear relationship between two sets of variables [1, 2]. CCA helps to decode complex dependency structures in multivariate data and to identify groups of interacting variables. Consequently, it has numerous practical applications in molecular biology, for example omics data integration [3] and network analysis [4], but also in many other areas such as econometrics or social science.
In its original formulation CCA is viewed as an algorithmic procedure optimizing a set of objective functions, rather than as a probablistic model for the data. Only relatively recently this perspective has changed. Bach and Jordan [5] proposed a latent variable model for CCA building on earlier work on probabilistic principal component analysis (PCA) by [6]. The probabilistic approach to CCA not only allows to derive the classic CCA algorithm but also provide an avenue for Bayesian variants [7, 8].
In parallel to establishing probabilistic CCA the classic CCA approach has also been further developed in the last decade by introducing variants of the CCA algorithm that are more pertinent for highdimensional data sets now routinely collected in the life and physical sciences. In particular, the problem of singularity in the original CCA algorithm is resolved by introducing sparsity and regularization [9, 10, 11, 12, 13] and, similarly, largescale computation is addressed by new algorithms [14, 15].
In this note, we revisit both classic and probabilistic CCA from the perspective of whitening of random variables [16]. As a result, we propose a simple yet flexible probabilistic model for CCA linking together multivariate regression, latent variable models, and highdimensional estimation. Crucially, this model for CCA not only facilitates comprehensive understanding of both classic and probabilistic CCA via the process of whitening but also extends CCA by allowing for negative canonical correlations and providing the flexibility to include nonnormal latent variables.
The remainder of this paper is as follows. First, we present our main results. After reviewing classical CCA we demonstrate that the classic CCA algorithm is special form of whitening. Next, we show that the link of CCA with multivariate regression leads to a probabilistic twolevel latent variable model for CCA that directly reproduces classic CCA without any rotational ambiguity. Subsequently, we discuss our approach by applying it to both synthetic data as well as to multiple integrated omics data sets. Finally, we describe our implementation in R and highlight computational and algorithmic aspects.
Much of our discussion is framed in terms of random vectors and their properties rather than in terms of data matrices. This allows us to study the probabilistic model underlying CCA separate from associated statistical procedures for estimation.
Multivariate notation
We consider two random vectors X=(X_{1},…,X_{p})^{T} and Y=(Y_{1},…,Y_{q})^{T} of dimension p and q. Their respective multivariate distributions F_{X} and F_{Y} have expectation E(X)=μ_{X} and E(Y)=μ_{Y} and covariance var(X)=Σ_{X} and var(Y)=Σ_{Y}. The crosscovariance between X and Y is given by cov(X,Y)=Σ_{XY}. The corresponding correlation matrices are denoted by P_{X}, P_{Y}, and P_{XY}. By V_{X}=diag(Σ_{X}) and V_{Y}=diag(Σ_{Y}) we refer to the diagonal matrices containing the variances only, allowing to decompose covariances as Σ=V^{1/2}PV^{1/2}. The composite vector (X^{T},Y^{T})^{T} has therefore mean \(\left (\boldsymbol {\mu }_{\boldsymbol {X}}^{T}, \boldsymbol {\mu }_{\boldsymbol {Y}}^{T}\right)^{T}\) and covariance \(\left (\begin {array}{cc} \boldsymbol {\Sigma }_{\boldsymbol {X}} & \boldsymbol {\Sigma }_{\boldsymbol {X} \boldsymbol {Y}}\\ \boldsymbol {\Sigma }_{\boldsymbol {X} \boldsymbol {Y}}^{T} & \boldsymbol {\Sigma }_{\boldsymbol {Y}} \end {array}\right)\).
Vectorvalued samples of the random vectors X and Y are denoted by x_{i} and y_{i} so that (x_{1},…,x_{i},…,x_{n})^{T} is the n×p data matrix for X containing n observed samples (one in each row). Correspondingly, the empirical mean for X is given by \(\hat {\boldsymbol {\mu }}_{\boldsymbol {X}} = \bar {\boldsymbol {x}} =\frac {1}{n} {\sum \nolimits }_{i=1}^{n} \boldsymbol {x}_{i}\), the unbiased covariance estimate is \(\widehat {\boldsymbol {\Sigma }}_{\boldsymbol {X}} = \boldsymbol {S}_{\boldsymbol {X}} = \frac {1}{n1} {\sum \nolimits }^{n}_{i=1} (\boldsymbol {x}_{i} \bar {\boldsymbol {x}})\left (\boldsymbol {x}_{i} \bar {\boldsymbol {x}}\right)^{T}\), and the corresponding correlation estimate is denoted by \(\widehat {\boldsymbol {P}}_{\boldsymbol {X}} = \boldsymbol {R}_{\boldsymbol {X}}\).
Results
We first introduce CCA from a classical perspective, then we demonstrate that CCA is best understood as a special and uniquely defined type of whitening transformation. Next, we investigate the close link of CCA with multivariate regression. This not only allows to interpret CCA as regression model and to better understand canonical correlations, but also provides the basis for a probabilistic generative latent variable model of CCA based on whitening. This model is introduced in the last subsection.
Classical CCA
In matrix notation, with A=(α_{1},…,α_{p})^{T}, B=(β_{1},…,β_{q})^{T}, and Λ=diag(λ_{i}), the above can be written as cor(AX,BY)=Λ as well as cor(AX)=I and cor(BY)=I. The projected vectors AX and BY are also called the CCA scores or the canonical variables.
Hotelling (1936) [1] showed that there are, assuming full rank covariance matrices Σ_{X} and Σ_{Y}, exactly m= min(p,q) canonical correlations and pairs of canonical directions α_{i} and β_{i}, and that these can be computed analytically from a generalized eigenvalue problem (e.g., [2]). Further below we will see how canonical directions and correlations follow almost effortlessly from a whitening perspective of CCA.
Since correlations are invariant against rescaling, optimizing Eq. 1 determines the canonical directions α_{i} and β_{i} only up to their respective lengths, and we can thus arbitrarily fix the magnitude of the vectors α_{i} and β_{i}. A common choice is to simply normalize them to unit length so that \(\boldsymbol {\alpha }_{i}^{T} \boldsymbol {\alpha }_{i}= 1\) and \(\boldsymbol {\beta }_{i}^{T} \boldsymbol {\beta }_{i}= 1\).
Similarly, the overall sign of the canonical directions α_{i} and β_{j} is also undetermined. As a result, different implementations of CCA may yield canonical directions with different signs, and depending on the adopted convention this can be used either to enforce positive or to allow negative canonical correlations, see below for further discussion in the light of CCA as a regression model.
Because it optimizes correlation, CCA is invariant against location translation of the original vectors X and Y, yielding identical canonical directions and correlations in this case. However, under scale transformation of X and Y only the canonical correlations λ_{i} remain invariant whereas the directions will differ as they depend on the variances V_{X} and V_{Y}. Therefore, to facilitate comparative analysis and interpretation the canonical directions the random vectors X and Y (and associated data) are often standardized.
Classical CCA uses the empirical covariance matrix S to obtain canonical correlations and directions. However, S can only be safely employed if the number of observations is much larger than the dimensions of either of the two random vectors X and Y, since otherwise S constitutes only a poor estimate of the underlying covariance structure and in addition may also become singular. Therefore, to render CCA applicable to small sample highdimensional data two main strategies are common: one is to directly employ regularization on the level of the covariance and correlation matrices to stabilize and improve their estimation; the other is to devise probabilistic models for CCA to facilitate application of Bayesian inference and other regularized statistical procedures.
Whitening transformations and CCA
Background on whitening
Here, Q_{X} is an orthogonal matrix; therefore the whitening matrix W_{X} itself is not orthogonal unless P_{X}=V_{X}=I_{p}. The choice of Q_{X} determines the type of whitening [16]. For example, using Q_{X}=I_{p} leads to ZCAcor whitening, also known as Mahalanobis whitening based on the correlation matrix. PCAcor whitening, another widely used sphering technique, is obtained by setting Q_{X}=G^{T}, where G is the eigensystem resulting from the spectral decomposition of the correlation matrix P_{X}=GΘG^{T}. Since there is a sign ambiguity in the eigenvectors G we adopt the convention of [16] to adjust columns signs of G, or equivalently row signs of Q_{x}, so that the rotation matrix Q_{X} has a positive diagonal.
Note that the sum of squared correlations in each column of Ψ_{X} sum up to 1, as \(\text {diag}\left (\boldsymbol {\Psi }_{\boldsymbol {X}}^{T}\boldsymbol {\Psi }_{\boldsymbol {X}}\right) = \text {diag}(\boldsymbol {P}_{\boldsymbol {X}}) = \boldsymbol {I}_{p}\).
CCA whitening
We will show now that CCA has a very close relationship to whitening. In particular, the objective of CCA can be seen to be equivalent to simultaneous whitening of both X and Y, with a diagonality constraint on the crosscorrelation matrix between the whitened \(\widetilde {\boldsymbol {X}}\) and \(\widetilde {\boldsymbol {Y}}\).
First, we make the choice to standardize the canonical directions α_{i} and β_{i} according to \(\text {var}\left (\boldsymbol {\alpha }_{i}^{T} \boldsymbol {X}\right) = \boldsymbol {\alpha }_{i}^{T} \boldsymbol {\Sigma }_{\boldsymbol {X}} \boldsymbol {\alpha }_{i}= 1\) and \(\text {var}\left (\boldsymbol {\beta }_{i}^{T} \boldsymbol {Y}\right) = \boldsymbol {\beta }_{i}^{T} \boldsymbol {\Sigma }_{\boldsymbol {Y}} \boldsymbol {\beta }_{i}= 1\). As a result α_{i} and β_{i} form the basis of two whitening matrices, W_{X}=(α_{1},…,α_{p})^{T}=A and W_{Y}=(β_{1},…,β_{q})^{T}=B, with rows containing the canonical directions. The length constraint \(\boldsymbol {\alpha }_{i}^{T} \boldsymbol {\Sigma }_{\boldsymbol {X}} \boldsymbol {\alpha }_{i}= 1\) thus becomes \(\boldsymbol {W}_{\boldsymbol {X}} \boldsymbol {\Sigma }_{\boldsymbol {X}}\boldsymbol {W}_{\boldsymbol {X}}^{T} = \boldsymbol {I}_{p}\) meaning that W_{X} (and W_{Y}) is indeed a valid whitening matrix.
Following the terminology in [17] we may call K the correlationadjusted crosscorrelation matrix between X and Y.
This provides the rotation matrices \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}} \) and the \(\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}} \) of dimensions m×p and m×q, respectively, and the m×m matrix Λ=diag(λ_{i}) containing the singular values of K, which are also the singular values of \(\boldsymbol {P}_{\widetilde {\boldsymbol {X}} \widetilde {\boldsymbol {Y}}}\). Since m= min(p,q) the larger of the two rotation matrices will not be a square matrix but it can nonetheless be used for whitening via Eqs. 4 and 5 since it still is semiorthogonal with \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}} \left (\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}}\right)^{T} = \boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}} \left (\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}}\right)^{T} = \boldsymbol {I}_{m}\). As a result, we obtain \(\text {cor}\left (\widetilde {X}^{\text {CCA}}_{i}, \widetilde {Y}^{\text {CCA}}_{i}\right) = \lambda _{i}\) for i=1…m, i.e. the canonical correlations are identical to the singular values of K.
Hence, CCA may be viewed as the outcome of a uniquely determined whitening transformation with underlying sphering matrices \(\boldsymbol {W}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {W}_{\boldsymbol {Y}}^{\text {CCA}}\) induced by the rotation matrices \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}}\). Thus, the distinctive feature of CCA whitening, in contrast to other common forms of whitening described in [16], is that by construction it is not only informed by P_{X} and P_{Y} but also by P_{XY}, which fixes all remaining rotational freedom.
CCA and multivariate regression
Optimal linear multivariate predictor
with minimum achieved \(\text {MSE}\left (\boldsymbol {a}^{\text {all}}, \boldsymbol {b}^{\text {all}}\right) = \text {Tr} \left (\boldsymbol {\Sigma }_{\boldsymbol {Y}} \right)  \text {Tr} \left (\boldsymbol {\Sigma }_{\boldsymbol {Y} \boldsymbol {X}} \boldsymbol {\Sigma }_{\boldsymbol {X}}^{1} \boldsymbol {\Sigma }_{\boldsymbol {X} \boldsymbol {Y}}\right)\).
If the response Y is univariate (q=1) then Δ reduces to the variancescaled coefficient of determination \(\sigma ^{2}_{Y} \boldsymbol {P}_{Y \boldsymbol {X}} \boldsymbol {P}_{\boldsymbol {X}}^{1} \boldsymbol {P}_{\boldsymbol {X} Y}\). Note that in the above no distributional assumptions are made other than specification of means and covariances.
Regression view of CCA
In the special case of CCAwhitening the regression coefficients further simplify to \(b^{\text {all}}_{ii} = \lambda _{i}\), i.e. the canonical correlations λ_{i} act as the regression coefficients linking CCAwhitened \(\widetilde {\boldsymbol {Y}}\) and \(\widetilde {\boldsymbol {X}}\). Furthermore, as the decrease in predictive MSE Δ is the sum of the squared canonical correlations (cf. Eq. 17), each \(\lambda _{i}^{2}\) can be interpreted as the variable importance of the corresponding variable in \(\widetilde {\boldsymbol {X}}^{\text {CCA}}\) to predict the outcome \(\widetilde {\boldsymbol {Y}}^{\text {CCA}}\). Thus, CCA directly results from multivariate regression between CCAwhitened random vectors, where the canonical correlations λ_{i} assume the role of regression coefficients and \(\lambda _{i}^{2}\) provides a natural measure to rank the canonical components in order of their respective predictive capability.
A key difference between classical CCA and regression is that in the latter both positive and negative coefficients are allowed to account for the directionality of the influence of the predictors. In contrast, in classical CCA only positive canonical correlations are permitted by convention. To reflect that CCA analysis is inherently a regression model we advocate here that canonical correlations should indeed be allowed to assume both positive and negative values, as fundamentally they are regression coefficients. This can be implemented by exploiting the sign ambiguity in the singular value decomposition of K (Eq. 10). In particular, the rows signs of \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}}\) and the signs of λ_{i} can be revised simultaneously without affecting K. We propose to choose \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}}\) such that both rotation matrices have a positive diagonal, and then to adjust the signs of the λ_{i} accordingly. Note that orthogonal matrices with positive diagonals are closest to the identity matrix (e.g. in terms of the Frobenius norm) and thus constitute minimal rotations.
Generative latent variable model for CCA
To clarify the workings behind Eq. 19 assume there are three uncorrelated random variables Z_{1}, Z_{2}, and Z_{3} with mean 0 and variance 1. We construct X_{1} as a mixture of Z_{1} and Z_{3} according to \(X_{1} = \sqrt {1\alpha } Z_{1} + \sqrt {\alpha } Z_{3}\) where α∈[0,1], and, correspondingly, X_{2} as a mixture of Z_{2} and Z_{3} via \(X_{2} = \sqrt {1\alpha } Z_{2} + \sqrt {\alpha } Z_{3}\). If α=0 then X_{1}=Z_{1} and X_{2}=Z_{2}, and if α=1 then X_{1}=X_{2}=Z_{3}. By design, the new variables have mean zero (E(X_{1})=E(X_{2})=0) and unit variance (var(X_{1})=var(X_{2})=1). Crucially, the weight α of the latent variable Z_{3} common to both mixtures induces a correlation between X_{1} and X_{2}. The covariance between X_{1} and X_{2} is \(\text {cov}(X_{1}, X_{2}) = \text {cov}\left (\sqrt {\alpha } Z_{3},\sqrt {\alpha } Z_{3} \right) = \alpha \), and since X_{1} and X_{2} have variance 1 we have cor(X_{1},X_{2})=α. In Eq. 19 this is further extended to allow a signed α and hence negative correlations.
Note that the above probabilistic model for CCA is in fact not a single model but a family of models, since we do not completely specify the underlying distributions, only their means and (co)variances. While in practice we will typically assume normally distributed generative latent variables, and hence normally distributed observations, it is equally possible to employ other distributions for the first level latent variables. For example, a rescaled tdistribution with a wider tail than the normal distribution may be employed to obtain a robustified version of CCA [18].
Discussion
Synthetic data
In order to test whether our algorithm allows to correctly identify negative canonical correlations we conducted simulations using simulated data. Specifically, we generated data X_{i} and y_{i} from a p+q dimensional multivariate normal distribution with zero mean and covariance matrix \(\left (\begin {array}{cc} \boldsymbol {\Sigma }_{\boldsymbol {X}} & \boldsymbol {\Sigma }_{\boldsymbol {X} \boldsymbol {Y}}\\ \boldsymbol {\Sigma }_{\boldsymbol {X} \boldsymbol {Y}}^{T} & \boldsymbol {\Sigma }_{\boldsymbol {Y}} \end {array}\right)\)where Σ_{X}=I_{p}, Σ_{Y}=I_{q} and Σ_{XY}=diag(λ_{i}). The canonical correlations where set to have alternating positive and negative signs λ_{1}=λ_{3}=λ_{5}=λ_{7}=λ_{9}=λ and λ_{2}=λ_{4}=λ_{6}=λ_{8}=λ_{10}=−λ with varying strength λ∈{0.3,0.4,0.5,0.6,0.7,0.8,0.9}. A similar setup was used in [14]. The dimensions were fixed at p=60 and q=10 and the sample size was n∈{20,30,50,100,200,500} so that both the small and large sample regime was covered. For each combination of n and λ the simulations were repeated 500 times, and our algorithm using shrinkage estimation of the underlying covariance matrices was employed to each of the 500 data sets to fit the CCA model. The resulting estimated canonical correlations were then compared with the corresponding true canonical correlations, and the proportion of correctly estimated signs was recorded.
Nutrimouse data
We now analyze two experimental omics data sets to illustrate our approach. Specifically, we demonstrate the capability of our variant of CCA to identify negative canonical correlations among canonical variates as well its application to highdimensional data where the number of samples n is smaller than the number of variables p and q.
The first data set is due to [19] and results from a nutrigenomic study in the mouse studying n=40 animals. The X variable collects the measurements of the gene expression of p=120 genes in liver cells. These were selected a priori considering the biological relevance for the study. The Y variable contains lipid concentrations of q=21 hepatic fatty acids, measured on the same animals. Before further analysis we standardized both X and Y.
The Cancer Genome Atlas LUSC data
As a further illustrative example we studied genomic data from The Cancer Genome Atlas (TCGA), a public resource that catalogues clinical data and molecular characterizations of many cancer types [20]. We used the TCGA2STAT tool to access the TCGA database from within R [21].
Specifically, we retrieved gene expression (RNASeq2) and methylation data for lung squamous cell carcinoma (LUSC) which is one of the most common types of lung cancer. After download, calibration and filtering as well as matching the two data types to 130 common patients following the guidelines in [21] we obtained two data matrices, one (X) measuring gene expression of p=206 genes and one (Y) containing methylation levels corresponding to q=234 probes. As clinical covariates the sex of each of the 130 patients (97 males, 33 females) was downloaded as well as the vital status (46 events in males, and 11 in females) and cancer end points, i.e. the number of days to last followup or the days to death. In addition, since smoking cigarettes is a key risk factor for lung cancer, the number of packs per year smoked was also recorded. The number of packs ranged from 7 to 240, so all of the patients for which this information was available were smokers.
Conclusions

first, we show that CCA is procedurally equivalent to a special whitening transformation, that unlike other general whitening procedures, is uniquely defined and without any rotational ambiguity;

second, we demonstrate the direct connection of CCA with multivariate regression and demonstrate that CCA is effectively a linear model between whitened variables, and that correspondingly canonical correlations are best understood as regression coefficients;

third, the regression perspective advocates for permitting both positive and negative canonical correlations and we show that this also allows to resolve the sign ambiguity present in the canonical directions;

fourth, we propose an easily interpretable probabilistic generative model for CCA as a twolayer latent variable framework that not only admits canonical correlations of both signs but also allows nonnormal latent variables;

and fifth, we provide a computationally effective computer implementation in the “whitening” R package based on highdimensional shrinkage estimation of the underlying covariance and correlation matrices and show that this approach performs well both for simulated data as well as in application to the analysis of various types of omics data.
In short, this work provides a unifying perspective on CCA, linking together sphering procedures, multivariate regression and corresponding probabilistic generative models, and also offers a practical tool for highdimensional CCA for practitioners in applied statistical data analysis.
Methods
Implementation in R
We have implemented our method for highdimensional CCA allowing for potential negative canonical correlations in the R package “whitening” that is freely available from https://CRAN.Rproject.org/package=whitening. The functions provided in this package incorporate the computational efficiencies described below. The R package also includes example scripts. The “whitening” package has been used to conduct the data analysis described in this paper. Further information and R code to reproduce the analyses in this paper is available at http://strimmerlab.org/software/whitening/.
Highdimensional estimation
Practical application of CCA, in both the classical and probabilistic variants, requires estimation of the joint covariance of X and Y from data, as well as the computation of the corresponding underlying whitening matrices \(\boldsymbol {W}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {W}_{\boldsymbol {Y}}^{\text {CCA}}\) (i.e. canonical directions) and canonical correlations λ_{i}.
In moderate dimensions and large sample size n, i.e. when both p and q are not excessively big and n is larger than both p and q the classic CCA algorithm is applicable and empirical or maximum likelhood estimates may be used. Conversely, if the sample size n is small compared to p and q then there exist numerous effective Bayesian, penalized likelihood and other related regularized estimators to obtain statistically efficient estimates of the required covariance matrices (e.g., [22, 23, 24, 25]). In our implementation in R and in the analysis below we use the shrinkage covariance estimation approach developed in [22] and also employed for CCA analysis in [14]. However, in principle any other preferred covariance estimator may be applied.
Algorithmic efficiencies
In addition to statistical issues concerning accurate estimation, high dimensionality also poses substantial challenges in algorithmic terms, with regard both to memory requirements as well as to computing time. Specifically, for large values of p and q directly performing the matrix operations necessary for CCA, such as computing the matrix square root or even simple matrix multiplication, will be prohibitive since these procedures typically scale in cubic order of p and q.
In particular, in a CCA analysis this affects i) the computation and estimation of the matrix K (Eq. 9) containing the adjusted crosscorrelations, and ii) the calculation of the whitening matrices \(\boldsymbol {W}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {W}_{\boldsymbol {Y}}^{\text {CCA}}\) with the canonical directions α_{i} and β_{i} from the rotation matrices \(\boldsymbol {Q}_{\boldsymbol {X}}^{\text {CCA}}\) and \(\boldsymbol {Q}_{\boldsymbol {Y}}^{\text {CCA}}\) (Eq. 5). These computational steps involve multiplication and squareroot calculations involving possibly very large matrices of dimension p×p and q×q.
Fortunately, in the small sample domain with n≤p,q there exist computational tricks to perform these matrix operations in a very effective and both time and memorysaving manner that avoids to directly compute and handle the largescale covariance matrices and their derived quantities [e.g. [26]]. Note this requires the use of regularized estimators, e.g. shrinkage or ridgetype estimation. Specifically, in our implementation of CCA we capitalize on an algorithm described in [27] (see “Zuber et al. algorithm” section for details) that allows to compute the matrix product of the inverse matrix square root of the shrinkage estimate of the correlation matrix R with a matrix M without the need to store or compute the full estimated correlation matrices. The computationals savings due to effective matrix operations for n<p and n<q can be substantial, going from O(p^{3}) and O(q^{3}) down to O(n^{3}) in terms of algorithmic complexity. Correspondingly, for example for p/n = 3 this implies time savings of factor 27 compared to “naive” direct computation.
Zuber et al. algorithm
Note that on the righthand side of these two equations no matrix of dimension p×p appears; instead all matrices are of much smaller size.
In the CCA context we apply this procedure to Eq. 5 in order to obtain the whitening matrix and the canonical directions and also to Eq. 9 to efficiently compute the matrix K.
Notes
Acknowledgements
KS thanks Martin Lotz for discussion. We also thank the anonymous referees for their many useful suggestions.
Funding
TJ was funded by a Wellcome Trust ISSF Ph.D. studentship. The funding body did not play any role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript
Availability of data and materials
The proposed method has been implemented in the R package “whitening” that is distributed from https://CRAN.Rproject.org/package=whitening. R code to reproduce the analysis is available from http://strimmerlab.org/software/whitening/.
Authors’ contributions
TJ and KS jointly conducted the research and wrote the paper. Both authors read and approved the manuscript.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
 1.Hotelling H. Relations between two sets of variates. Biometrika. 1936; 28:321–77.CrossRefGoogle Scholar
 2.Härdle WK, Simar L. Canonical correlation analysis. In: Applied Multivariate Statistical Analysis. Chap. 16. Berlin: Springer: 2015. p. 443–54.Google Scholar
 3.Cao DS, Liu S, Zeng WB, Liang YZ. Sparse canonical correlation analysis applied to omics studies for integrative analysis and biomarker discovery. J Chemometrics. 2015; 29:371–8.CrossRefGoogle Scholar
 4.Hong S, Chen X, Jin L, Xiong M. Canonical correlation analysis for RNAseq coexpression networks. Nucleic Acids Res. 2013; 41:95.CrossRefGoogle Scholar
 5.Bach FR, Jordan MI. A probabilistic interpretation of canonical correlation analysis. Technical Report No. 688, Department of Statistics. Berkeley: University of California; 2005.Google Scholar
 6.Tipping ME, Bishop CM. Probabilistic principal component analysis. J R Statist Soc B. 1999; 61(3):611–22. https://doi.org/10.1111/14679868.00196.CrossRefGoogle Scholar
 7.Wang C. Variational Bayesian approach to canonical correlation analysis. IEEE T Neural Net. 2007; 18:905–10.CrossRefGoogle Scholar
 8.Klami A, Kaski S. Local dependent components. Proceedings of the 24th International Conference on Machine Learning (ICML 2007). 2007; 24:425–32.Google Scholar
 9.Waaijenborg S, de Witt Hamer PCV, Zwinderman AH. Quantifying the association between gene expressions and DNAmarkers by penalized canonical correlation analysis. Stat Appl Genet Molec Biol. 2008;7(1). Article 3. https://doi.org/10.2202/15446115.1329.
 10.Parkhomenko E, Tritchler D, Beyene J. Sparse canonical correlation analysis with application to genomic data integration. Stat Appl Genet Molec Biol. 2009; 8:1.CrossRefGoogle Scholar
 11.Witten D, Tibshirani R, Hastie T. A penalized matrix decomposition, with applications to sparse principal components and canonical correlation analysis. Biostatistics. 2009; 10(3):515–34. https://doi.org/10.1093/biostatistics/kxp008.CrossRefGoogle Scholar
 12.Hardoon DR, ShaweTaylor J. Sparse canonical correlation analysis. Mach Learn. 2011; 83:331–53.CrossRefGoogle Scholar
 13.Wilms I, Croux C. Sparse canonical correlation analysis from a predictive point of view. Biomet J. 2015; 57:834–51.CrossRefGoogle Scholar
 14.CruzCano R, Lee MLT. Fast regularized canonical correlation analysis. Comp Stat Data Anal. 2014; 70:88–100.CrossRefGoogle Scholar
 15.Ma Z, Lu Y, Foster D. Finding linear structure in large datasets with scalable canonical correlation analysis. Proceedings of the 32th International Conference on Machine Learning (ICML 2015), PLMR. 2015; 37:169–78.Google Scholar
 16.Kessy A, Lewin A, Strimmer K. Optimal whitening and decorrelation. Am Stat. 2018; 72:309–14. https://doi.org/10.1080/00031305.2016.1277159.CrossRefGoogle Scholar
 17.Zuber V, Strimmer K. Highdimensional regression and variable selection using CAR scores. Stat Appl Genet Molec Biol. 2011; 10:34.CrossRefGoogle Scholar
 18.Adrover JG, Donato SM. A robust predictive approach for canonical correlation analysis. J Multiv Anal. 2015; 133:356–76.CrossRefGoogle Scholar
 19.Martin PGP, Guillou H, Lasserre F, Déjean S, Lan A, Pascussi JM, Cristobal MS, Legrand P, Besse P, Pineau T. Novel aspects of PPAR αmediated regulation of lipid and xenobiotic metabolism revealed through a multigenomic study. Hepatology. 2007; 54:767–77.CrossRefGoogle Scholar
 20.Kandoth C, McLellan MD, Vandin F, Ye K, Niu B, Lu C, Xie M, JF McMichael QZ, Wyczalkowski MA, Leiserson MDM, Miller CA, Welch JS, Walter MJ, Wendl MC, Ley TJ, Wilson RK, Raphael BJ, Ding L. Mutational landscape and significance across 12 major cancer types. Nature. 2013; 502:333–9.CrossRefGoogle Scholar
 21.Wan YW, Allen GI, Liu Z. TCGA2STAT: simple TCGA data access for integrated statistical analysis in R. Bioinformatics. 2016; 32:952–4.CrossRefGoogle Scholar
 22.Schäfer J, Strimmer K. A shrinkage approach to largescale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Molec Biol. 2005; 4:32.CrossRefGoogle Scholar
 23.Bickel PJ, Levina E. Regularized estimation of large covariance matrices. Ann Stat. 2008; 36:199–227.CrossRefGoogle Scholar
 24.Hannart A, Naveau P. Estimating high dimensional covariance matrices: A new look at the Gaussian conjugate framework. J Multiv Anal. 2014; 131:149–62.CrossRefGoogle Scholar
 25.Touloumis A. Nonparametric Steintype shrinkage covariance matrix estimators in highdimensional settings. Comp Stat Data Anal. 2015; 83:251–61.CrossRefGoogle Scholar
 26.Hastie T, Tibshirani T. Efficient quadratic regularization for expression arrays. Biostatistics. 2004; 5:329–40.CrossRefGoogle Scholar
 27.Zuber V, Duarte Silva AP, Strimmer K. A novel algorithm for simultaneous SNP selection in highdimensional genomewide association studies. BMC Bioinformatics. 2012; 13:284.CrossRefGoogle Scholar
Copyright information
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License(http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.