Total Least Squares

Abstract

The chapter treats total least squares (TLS), which in statistics corresponds to orthogonal regression. Some different extensions are discussed, including ways to show how uncertainties in different matrix elements may be related or correlated. The application of TLS to identification of dynamic systems is also treated.

Appendices

Appendix

11.A Further Details

11.1.1 11.A.1 The Eckart–Young–Mirsky Lemma

There is a neat result, due to Eckart and Young (1936) and Mirsky (1960), on the optimal low-rank approximation of a given matrix. It will be useful when deriving the solution to the TLS problem.

Lemma 11.3

Consider an $m \times n$ matrix $\mathbf {C}$ with $m \ge n$. Let $\mathbf {C}$ have a singular value decomposition

$$\begin{aligned} \mathbf {C}= \left( \begin{array}{cc} \mathbf U_1&\mathbf U_2 \end{array}\right) \left( \begin{array}{cc} \varvec{\varSigma }_1 &{} \mathbf {0}\\ \mathbf {0}&{} \varvec{\varSigma }_2 \end{array}\right) \left( \begin{array}{c} \mathbf {V}_1^T \\ \mathbf {V}_2^T \end{array}\right) \; , \end{aligned}$$

(11.52)

where $\varvec{\varSigma }_1$ is an $r \times r$ matrix containing the r largest singular values, and the other matrices have compatible dimensions.

The matrix $\hat{\mathbf {C}}$ defined as

$$\begin{aligned} \hat{\mathbf {C}} = \mathrm{arg} \min _{ \mathrm{rank} \hat{\mathbf {C}} = r} \parallel \mathbf {C}- \hat{\mathbf {C}} \parallel _\mathrm{F}^2 \; , \end{aligned}$$

(11.53)

is given by

$$\begin{aligned} \hat{\mathbf {C}} = \mathbf U_1 \varvec{\varSigma }_1 \mathbf {V}_1^T \; . \end{aligned}$$

(11.54)

Further,

$$\begin{aligned} \parallel \mathbf {C}- \hat{\mathbf {C}} \parallel _\mathrm{F}^2 = \parallel \varvec{\varSigma }_2 \parallel _\mathrm{F}^2 = \mathrm{tr} ( \varvec{\varSigma }_2^T \varvec{\varSigma }_2 ) \; . \end{aligned}$$

(11.55)

Proof

See Eckart and Young (1936). $\blacksquare$

11.1.2 11.A.2 Characterization of the TLS Solution

11.1.2.1 11.A.2.1 Proof of Lemma 11.1

Set

$$\begin{aligned} \mathbf {u}= \left( \begin{array}{c} 1 \\ \mathbf {x}_\mathrm{TLS} \end{array}\right) \; . \end{aligned}$$

One needs to find $\varDelta \mathbf {C}$ with minimal norm, such that

$$\begin{aligned} ( \mathbf {C}+ \varDelta \mathbf {C}) \mathbf {u}= \mathbf {0}\; . \end{aligned}$$

By applying Lemma 11.3, one gets $r = n-1$ and

$$\begin{aligned} \varDelta \mathbf {C}= - \mathbf U_2 \varvec{\varSigma }_2 \mathbf {V}_2^T, \ \ \mathbf {u}= \mathbf {V}_2, \ \ \sigma = \varvec{\varSigma }_2 \; , \end{aligned}$$

and the lemma follows directly.

11.1.2.2 11.A.2.2 Proof of Remark 11.3

First establish $\mathbf {V}_1^T \mathbf U= \mathbf {0}$ and

$$\begin{aligned} \mathbf {C}^T \mathbf {C}\mathbf {u}= & {} \left( \mathbf {V}_1 \varvec{\varSigma }_1 \mathbf U_1^T + \mathbf {V}_2 \varvec{\varSigma }_2 \mathbf U_2^T \right) \left( \mathbf U_1 \varvec{\varSigma }_1 \mathbf {V}_1^T + \mathbf U_2 \varvec{\varSigma }_2 \mathbf {V}_2^T \right) \mathbf {u}\nonumber \\= & {} \mathbf {V}_2 \varvec{\varSigma }_2^2 = \sigma ^2 \mathbf {u}\; . \end{aligned}$$

Using the definition (11.5) of $\mathbf {C}$ it thus holds

$$\begin{aligned} \left( \begin{array}{c} - \mathbf {b}^T \\ \mathbf {A}^T \end{array}\right) \left( \begin{array}{cc} - \mathbf {b}&\mathbf {A}\end{array}\right) \left( \begin{array}{c} 1 \\ \mathbf {x}_\mathrm{TLS} \end{array}\right) = \sigma ^2 \left( \begin{array}{c} 1 \\ \mathbf {x}_\mathrm{TLS} \end{array}\right) \; . \end{aligned}$$

Spelling out the lower part of this equation gives

$$\begin{aligned} -\mathbf {A}^T \mathbf {b}+ \mathbf {A}^T \mathbf {A}\mathbf {x}_\mathrm{TLS} = \sigma ^2 \mathbf {x}_\mathrm{TLS} \; , \end{aligned}$$

which proves (11.7).

11.1.3 11.A.3 Proof of Lemma 11.2

To examine the optimization problem, introduce the Lagrange multiplier vector $\varvec{\lambda }$ and the Lagrange function

$$\begin{aligned} L (\mathbf {x}, \varvec{\eta }) = \varvec{\eta }^T \mathbf {Q}\varvec{\eta }+ \varvec{\lambda }^T \left( \mathbf {f}(\mathbf {x}) - \mathbf {F}(\mathbf {x}) \varvec{\eta }\right) \; . \end{aligned}$$

(11.56)

Setting the gradient of L with respect to $\varvec{\eta }$ to zero leads to

$$\begin{aligned} 2 \mathbf {Q}\varvec{\eta }- \mathbf {F}^T (\mathbf {x}) \varvec{\lambda }= \mathbf {0}\Rightarrow \varvec{\eta }= \frac{1}{2} \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \varvec{\lambda }\; . \end{aligned}$$

(11.57)

Considering the constraint (11.21) (or setting the gradient of L with respect to $\varvec{\lambda }$ to zero) leads to

$$\begin{aligned} \mathbf {f}(\mathbf {x}) = \mathbf {F}(\mathbf {x}) \varvec{\eta }= \frac{1}{2} \mathbf {F}(\mathbf {x}) \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \varvec{\lambda }\Rightarrow \varvec{\lambda }= 2 \left( \mathbf {F}(\mathbf {x}) \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \right) ^{-1} \mathbf {f}(\mathbf {x}) \; . \end{aligned}$$

(11.58)

Therefore

$$\begin{aligned} \varvec{\eta }= \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \left( \mathbf {F}(\mathbf {x}) \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \right) ^{-1} \mathbf {f}(\mathbf {x}) \; , \end{aligned}$$

(11.59)

which always satisfies the constraint (11.21). Furthermore, simple algebra shows that the minimal value of the loss function (with respect $\varvec{\eta }$) becomes

$$\begin{aligned} \overline{V}(\mathbf {x}) = \min _{\varvec{\eta }} V(\varvec{\eta }) = \mathbf {f}^T (\mathbf {x}) \left( \mathbf {F}(\mathbf {x}) \mathbf {Q}^{-1} \mathbf {F}^T (\mathbf {x}) \right) ^{-1} \mathbf {f}(\mathbf {x}) \; , \end{aligned}$$

(11.60)

which shows (11.23) and completes the proof.