1 Introduction

The covariance matrices for observations are a basis for constructing the stochastic models for satellite and ground positioning systems. The covariances have an effect upon the elements of reliability matrices. It was already found in Wang and Chen (1994) and Schaffrin (1997) that at some level of observation correlations there may appear on a main diagonal of a reliability matrix the negative elements as well as the elements greater than 1. It was also noticed that the diagonal elements being one-dimensional quantities do not fully describe the responses of a model to gross errors in the observations. So, the proposal of a generalized reliability number appeared. Further study of this problem led to the so-called response-based reliability measures (Prószyński 2010) being a pair of quantities for each observation, i.e. a reliability number and an asymmetry index. They make ranking of observations with respect to internal reliability more correct than when using the reliability numbers alone. The measures will here be subject to further analyses.

Prior to formulating the objectives of the present research, it seems necessary to present the problems encountered already in the authors’ earlier research concerning the effect of observation correlations, i.e.

  • confirmed by publications on applied mathematics and numerical methods (Davies and Higham 2000; Budden et al. 2008; Numpacharoen and Atsawarungruangkit 2012), the difficulty in generating positive definite correlation matrices in a way that ensures a purely random character of the matrix elements throughout all the levels of correlation;

  • the lack of injectivity of the function f: \( {\mathbf{C}}_{\text{s}} \to { \det }{\mathbf{C}}_{\text{s}} \) (\( {\mathbf{C}}_{\text{s}} \)—a positive definite correlation matrix), where the degree of departure from injectivity increases with the number of observations in a system. Due to the above drawback, in order to make a graph presenting a correlation-dependent quantity as a function of the correlation level expressed in terms of \( { \det }{\mathbf{C}}_{\text{s}} \), we have a dispersion band instead of getting an unambiguous curve.

The above problems appear also in the present paper but are approached in a way that raises effectiveness of a matrix generating method and minimizes the effect of the lack of injectivity. The approach is explained briefly in “Appendix A”.

The main objectives of the paper are the following:

  • to formulate the properties of an oblique projection operator that are connected with the effect of observation correlations;

  • to provide a more detailed knowledge on variability of the response-based reliability measures with respect to the increase in observation correlations;

  • to compare the behaviour of the response-based reliability measures and of the so-called testing-based measures (i.e. generalized reliability numbers and MDBs) with the increase in observation correlations;

  • to create a theoretical basis for operating with reliability matrices in designing the positioning systems.

2 Recalling basic formulas and properties

We shall refer to the modified (i.e. standardized) form of the Gauss–Markov model (GMM) that exposes the correlation matrix \( {\mathbf{C}}_{\text{s}} \) (Prószyński 2010), the matrix being a dependent variable in the present analysis of correlation effect, i.e.

$$ {\mathbf{A}}_{\text{s}} {\mathbf{x}} + {\mathbf{e}}_{\text{s}} = {\mathbf{y}}_{\text{s}} \quad {\mathbf{e}}_{\text{s}} \sim ({\mathbf{0}}, \, {\mathbf{C}}_{\text{s}} ) $$
(1)

where x(u×1), ys(n×1), As(n×u), \( {\mathbf{e}}_{\text{s}} \) = \( - {\mathbf{v}}_{\text{s}} \), rank \( {\mathbf{A}}_{\text{s}} \) ≤ u, \( {\mathbf{C}}_{\text{s}} \)(n × n) (positive definite).

Let us remind that the form above is obtained by premultiplying both sides of the original GMM by \( {\mathbf{S}}^{ - 1} \), where \( {\mathbf{S}} = ( {\text{diag }}{\mathbf{C}} )^{1/2} \), and transforming the covariance matrix C (positive definite) accordingly.

The level of observation correlations in a model (1) will be measured with a global correlation index (Prószynski and Kwasniak 2018), defined by

$$ \rho_{\text{G}} = \sqrt {1 - { \det }{\mathbf{C}}_{\text{s}} } \quad \rho_{\text{G}} \in \,<\, 0, 1) $$
(2)

It should be noted that there is a straightforward relationship between \( \rho_{\text{G}} \) and the already introduced (Teunissen 1994) ambiguity decorrelation number \( r_{{\hat{a}}} \) based on the correlation matrix, i.e.

$$ \rho_{\text{G}} = \sqrt {1 - r_{{\hat{a}}}^{2} } $$

The function f: \( {\mathbf{C}}_{\text{s}} \to { \det }{\mathbf{C}}_{\text{s}} \) (or f:\( {\mathbf{C}}_{\text{s}} \to \rho_{\text{G}} \)) is not injective. This implies some ambiguity of the graphs for correlation-dependent quantities with \( \rho_{\text{G}} \) being a variable. That is why the use will also be made of constant correlation matrices (Tiit and Helemae 1997; Wied 2017), i.e. the positive definite matrices with all off-diagonal elements being set to one value of correlation. They are denoted here as \( {\mathbf{C}}_{{{\text{s,}}a}} \), where the function f:\( {\mathbf{C}}_{{{\text{s,}}a}} \to \rho_{{{\text{G,}}a}} \) is injective, separately for \( a \in ( - \frac{1}{n - 1}, \, 0) \) and \( a \in (0, 1) \). We can readily come to this conclusion by analysing the sign of derivative \( [\rho_{{{\text{G,}}a}} ]^{\prime } \).

A special representation of a positive definite correlation matrix (Prószyński and Kwaśniak 2018) will be applied in the analyses of correlation effect

$$ {\mathbf{C}}_{\text{s}} = q \cdot {\mathbf{P}}_{\text{s}}^{ - 1} $$
(3)

where q—scale factor; \( q = \left( {1 - \rho_{\text{G}}^{2} } \right)^{1/n} \); \( {\mathbf{P}}_{\text{s}} \)—internal weight matrix; det\( {\mathbf{P}}_{\text{s}} \) = 1.

The following quantities and formulas pertaining to internal reliability for systems with correlated observations will be analysed:

  1. (a)

    H—the reliability matrix, being an oblique projection operator (HH = H, \( {\mathbf{H}}^{\text{T}} \ne {\mathbf{H}} \))

$$ {\mathbf{H}} = {\mathbf{I}} - {\mathbf{A}}_{\text{s}} ({\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} )^{ + } {\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} $$
(4)

where (+) denotes a pseudoinverse.

The operator maps the disturbances in observations to increments in the least squares residuals. For uncorrelated observations, i.e. \( {\mathbf{C}}_{\text{s}} = {\mathbf{I}} \), formula (4) takes the form

$$ {\mathbf{H}}_{{\mathbf{I}}} = {\mathbf{I}} - {\mathbf{A}}_{\text{s}} ({\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{A}}_{\text{s}} )^{ + } {\mathbf{A}}_{\text{s}}^{\text{T}} $$
(5)

being an orthogonal projection operator (\( {\mathbf{H}}_{{\mathbf{I}}} {\mathbf{H}}_{{\mathbf{I}}} = {\mathbf{H}}_{{\mathbf{I}}} , \, {\mathbf{H}}_{{\mathbf{I}}}^{\text{T}} = {\mathbf{H}}_{{\mathbf{I}}} \)) and termed the redundancy matrix;

  1. (b)

    hi, wi (i =1,…, n)—the response-based measures of internal reliability for the ith observation; hi is a reliability number, wi is an asymmetry index. They are basic characteristics of the operator H and are defined as

$$ h_{i} = \{ {\mathbf{H}}\}_{ii} ;\quad w_{i} = \{ {\mathbf{W}}\}_{ii} ,\quad {\text{where}}\,{\mathbf{W}} = {\mathbf{H}} - {\mathbf{H}}^{\text{T}} {\mathbf{H}} $$
(6)

The measures hi, wi are bound by the following relationships

$$ \sum\limits_{1}^{n} {h_{i} } = r_{{\mathbf{H}}} ;\quad w_{i} \le h_{i} - \, h_{i}^{2} $$
(7)

where rH denotes \( {\text{rank }}{\mathbf{H}} \), being equal to \( {\text{Tr }}{\mathbf{H}} \).

There exists an interrelation between the asymmetry indices wi, i =1,…, n (see “Appendix B”).

  1. (c)

    G(i)—a global response to gross error in the ith observation (Prószyński 2010), defined by

$$ G_{{ (i\text{)}}} = \sqrt {\{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} } $$
(8)

and bound with hi and wi by the relationship \( G_{{\text{(}i )}} = \sqrt {h_{i} - w_{i} } \) (see formula (6)).

  1. (d)

    ri (i =1,…, n)—the generalized reliability numbers, given by

$$ r_{i} = \{ {\mathbf{H}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{H}}\}_{ii} $$
(9)

The quantities ri and \( \sqrt {r_{i} } \) are non-centrality parameters for unit gross error, in a global model test statistic and the local outlier test statistics, respectively. For \( {\mathbf{C}}_{\text{s}} = {\mathbf{I}} \), we have \( r_{i} = h_{{{\mathbf{I}}_{i} }} \).

Instead of the measures hi, wi, we can equivalently use a pair hi, ki . The parameter k is a ratio of the quasi-global response and a local response to a gross error in a particular observation (Prószyński 2010). It characterizes the relative strength of the local response (i.e. in that observation). The reliability criteria as expressed in terms of hi, ki are met in the region \( 0.5 < h_{i} < 1 \) and \( 0 < k_{i} < 1 \), termed in the above-mentioned reference the outlier-exposing area.

3 Methodology of research

The effect of observation correlations upon the reliability matrix always depends on the design matrix used. Also, the relationships linking the investigated indices hi, wi and G(i) of the reliability matrix and the observation correlations contained in the matrices \( {\mathbf{C}}_{\text{s}} \) are fairly complex. These two reasons call for the use of the specially planned methodology of research. So, for investigating the correlation effect numerical testing based on simulated networks will have to be the main research tool. For better determining of the effect itself, several networks with different levels of internal reliability will be used in the tests. Obviously, wherever feasible the purely theoretical derivations will be carried out to support the analyses of the test results.

Let us consider a correlation-dependent quantity x given as f(\( {\mathbf{A}}_{\text{s}} ,{\mathbf{C}}_{\text{s}} \)), where \( {\mathbf{A}}_{\text{s}} \), \( {\mathbf{C}}_{\text{s}} \) define the model (1). The correlation effect will be contained in a graph characterizing, for a given \( {\mathbf{A}}_{\text{s}} \), the variability of x due to the increase in observation correlations. Assuming the use of the global correlation index \( \rho_{\text{G}} \), the function x = f(\( {\mathbf{A}}_{\text{s}} ,{\mathbf{C}}_{\text{s}} \)) will have a form \( x = f(\rho_{\text{G}} |{\mathbf{A}}_{\text{s}} ) \), as expressed in a notation borrowed from conditional distributions. Since \( {\mathbf{A}}_{\text{s}} \) can be obtained with different values of observational standard deviations, we shall reduce the task to a single case of unit standard deviations (\( \sigma_{1} = \sigma_{2} = \cdots = \sigma_{n} = 1 \)) where \( {\mathbf{A}}_{\text{s}} = {\mathbf{A}} \), and hence, we can use the notation \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \). The effect of the increase in observation correlations will be determined on the basis of the values of x computed for different design matrices using specially created auxiliary indices.

Due to numerical problems with generating matrices of greater sizes (n ≥ 5), we decided to use also the simplest possible representation of correlation matrix \( {\mathbf{C}}_{\text{s}} \), i.e. the already mentioned constant correlation matrix, denoted as \( {\mathbf{C}}_{{{\text{s,}}a}} \). Although it yields specific matrix configurations, it represents injective function (see Sect. 2) and is readily created.

4 Auxiliary indices and their properties supporting analyses

For analysis of the effect of observation correlations on a reliability matrix H, contained in the graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) (see Sect. 3), the following auxiliary indices will be created (with n being the number of observations in a network), i.e.

  1. (a)

    dh, \( d_{{h_{{\mathbf{I}}} }} \)—the indices of mutual differentiation of hi values and \( h_{{{\mathbf{I}}_{i} }} \) values

They are defined in analogy to sample variance, i.e. as

$$ {\text{d}}_{h} = \tfrac{1}{n}\sum\limits_{1}^{n} {(h_{i} - \bar{h})^{2} } = \tfrac{1}{n}\sum\limits_{1}^{n} {h_{i}^{2} - \tfrac{2}{{n^{2} }}r_{{\mathbf{H}}} \sum\limits_{1}^{n} {h_{i} } + \tfrac{1}{{n^{2} }}r_{{\mathbf{H}}}^{2} = } \tfrac{1}{n}\sum\limits_{1}^{n} {h_{i}^{2} } - \tfrac{1}{{n^{2} }}r_{{\mathbf{H}}}^{2} $$
(10)
$$ {\text{d}}_{{h_{{\mathbf{I}}} }} = \tfrac{1}{n}\sum\limits_{1}^{n} {(h_{{{\mathbf{I}}_{i} }} - \bar{h}_{{\mathbf{I}}} )^{2} } = \tfrac{1}{n}\sum\limits_{1}^{n} {h_{{{\mathbf{I}}_{i} }}^{2} } - \tfrac{1}{{n^{2} }}r_{{\mathbf{H}}}^{2} $$
(11)

\( \Delta d_{h} \), \( [h_{{\rm max} } \, , \, h_{{\rm min} } ] \)—difference of indices as given above and the interval of hi values.

Due to \( r_{{\mathbf{H}}} \) being constant for a particular network,\( \, \tfrac{1}{n}\sum\nolimits_{n} {h_{i}^{2} } \) determines the shape of the graph \( d_{h} = f(\rho_{\text{G}} |{\mathbf{A}}) \).

  1. (b)

    \( \bar{w} \), \( [w_{{\rm max} } \, , \, w_{{\rm min} } ] \)—average asymmetry index and the interval of wi values.

Applying (6), we can write

$$ w_{i} = h_{i} - \{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} $$
(12)

and hence

$$ \bar{w} = \tfrac{1}{n}r_{{\mathbf{H}}} - \, \tfrac{1}{n}\sum\limits_{1}^{n} {\{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} } $$
(13)

Applying inequality (7), we may also get

$$ {\text{sup(}}\bar{w} )= \tfrac{1}{n}r_{{\mathbf{H}}} - \, \tfrac{1}{n}\sum\limits_{1}^{n} {h_{i}^{2} } $$
(14)

(c) \( \bar{r} \), \( {\text{d}}_{r} \)—average value of ri and the index of mutual differentiation of ri values.

As in the case of the index \( d_{h} \) we define \( {\text{d}}_{r} \) by

$$ {\text{d}}_{r} = \tfrac{1}{n}\sum\limits_{1}^{n} {(r_{i} - \bar{r})^{2} } $$
(15)

(d) \( \bar{g} \),\( {\text{d}}_{g} \)—average value of \( G_{ (i )}^{2} \) (see formula (8)) denoted as gi, and the index of mutual differentiation of \( g_{i} \) values.

Similarly to the case (c), we define \( {\text{d}}_{g} \) by

$$ {\text{d}}_{g} = \tfrac{1}{n}\sum\limits_{1}^{n} {(g_{i} - \bar{g})^{2} } $$
(16)

In all the above formulas, i = 1,…, n, n being the number of observations in a network.

5 Non-orthogonality versus asymmetry of projection H

To operate with one common term when referring to projection H and to the matrix H itself, we shall use in the former case the term nullspace instead of a more appropriate term the kernel.

The angles between spaces and their different definitions have been widely investigated in the mathematical literature (e.g. Golub and Van Loan 1989). A specific concept of minimal angle between complementary subspaces was introduced in (Ipsen and Meyer 1995). In the present paper, using basic definition of the angle between a pair of vectors, we start from standard methods of computing the angles between the range and the nullspace of projection operator H (see formula (4)) and on this basis derive a relationship between the angles of non-orthogonality and the asymmetry indices w.

Let U and V denote the range and the nullspace of projection H. Since H is an oblique projection, the subspaces U and V are not orthogonal. To determine the angles between U and V, we first find the vectors \( {\mathbf{u}} \in {\text{U}} \) such that Hu = u and \( {\mathbf{v}} \in {\text{V}} \) such that Hv = 0. The vectors u and v can be expressed in terms of \( {\mathbf{A}}_{\text{s}} \),\( {\mathbf{C}}_{\text{s}} \) (case 1) or can be determined directly on the basis of SVD (Singular Value Decomposition) of the matrix H (case 2). Both cases are presented below, i.e.

Case 1

From Hu = u, or equivalently, from (I − H)u = 0 where u belongs to the nullspace of complementary projection I − H, we get the relationship

$$ {\mathbf{A}}_{\text{s}} ({\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} )^{ + } {\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} \cdot {\mathbf{u}} = {\mathbf{0}} $$

and hence

$$ {\mathbf{u}} \bot \{ {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \to {\mathbf{u}} = \{ {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} \}_{ \bullet i}^{ \bot } \quad i = 1, \ldots ,n $$
(17)

or, equivalently

$$ {\mathbf{u}} = [{\mathbf{C}}_{\text{s}}^{ - 1} \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} ]^{ \bot } \quad i = 1, \ldots ,n $$
(18)

where \( \{ \, \cdot \, \}_{ \bullet i} \) denotes the ith column of the matrix in parentheses.

Applying the special representation of \( {\mathbf{C}}_{\text{s}} \), i.e. \( {\mathbf{C}}_{\text{s}} = q \cdot {\mathbf{P}}_{\text{s}}^{ - 1} \) (see formula (3)) and omitting the proportionality factor q, we get Eqs. (17), (18) as expressed in terms of the internal weight matrix \( {\mathbf{P}}_{\text{s}} \)

$$ {\mathbf{u}} = \{ {\mathbf{P}}_{\text{s}} {\mathbf{A}}_{\text{s}} \}_{ \bullet i}^{ \bot } \quad {\mathbf{u}} = [{\mathbf{P}}_{\text{s}} \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} ]^{ \bot } \quad i = 1, \ldots ,n $$
(19)

From Hv = 0, or equivalently from (I − H)v = v where v belongs to the range of complementary projection I − H, we get the relationship

$$ {\mathbf{A}}_{\text{s}} ({\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} )^{ + } {\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} \cdot {\mathbf{v}} = {\mathbf{v}} $$

Since (Rao and Mitra 1971)

$$ {\mathbf{A}}_{\text{s}} ({\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} )^{ + } {\mathbf{A}}_{\text{s}}^{\text{T}} {\mathbf{C}}_{\text{s}}^{ - 1} {\mathbf{A}}_{\text{s}} = {\mathbf{A}}_{\text{s}} $$

we obtain finally

$$ {\mathbf{v}} = \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \quad i = 1, \ldots ,n $$
(20)

The dimensions of the subspaces are dimU = n − (u − d) and dimV = u − d, what yields dimU + dimV = n.

The analogous analysis carried out for projection \( {\mathbf{H}}_{{\mathbf{I}}} \) as in formula (5) results in

$$ {\mathbf{u}}_{{\mathbf{I}}} = \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i}^{ \bot } \quad {\mathbf{v}}_{{\mathbf{I}}} = \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \quad i = 1, \ldots ,n $$

i.e. as expected, the range and the nullspace are orthogonal subspaces.

Case 2

On applying SVD to the matrix H, we get H = UΣVT, and in more detail

$$ {\mathbf{H}} = \left[ {\begin{array}{*{20}c} {{\mathbf{U}}_{r} } & {{\mathbf{U}}_{ * } } \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {{\varvec{\Sigma}}_{r} } & \quad {\mathbf{0}} \\ {\mathbf{0}} & \quad {\mathbf{0}} \\ \end{array} } \right]\left[ {\begin{array}{*{20}c} {{\mathbf{V}}_{ * } } & {{\mathbf{V}}_{n - r} } \\ \end{array} } \right]^{\text{T}} \quad {\text{or}}\quad {\mathbf{H}} = {\mathbf{U}}_{r} {\varvec{\Sigma}}_{r} {\mathbf{V}}_{ * }^{\text{T}} $$
(21)

where \( {\mathbf{U}}_{r} , \, {\mathbf{V}}_{n - r} \) are the orthonormal bases of the range space and the nullspace, respectively, \( {\varvec{\Sigma}}_{r} \) is a diagonal matrix of singular values, dimU = r, dimV = n − r, r is a rank of the matrix H, i.e. r = n − u + d. From the requirement that \( {\mathbf{HU}}_{r} = {\mathbf{U}}_{r} \), we get \( {\mathbf{U}}_{r} = {\mathbf{V}}_{ * } {\varvec{\Sigma}}_{r}^{ - 1} \).

The angles between the range and the nullspace of the projection H, denoted by \( \beta_{j,k} \), can be determined directly from the bases \( {\mathbf{U}}_{r} , \, {\mathbf{V}}_{n - r} \), according to the formula

$$ \left[ {{ \cos }\beta_{j,k} } \right]_{r \times (n - r )} = {\mathbf{U}}_{r}^{\text{T}} {\mathbf{V}}_{n - r} $$

Using the column vectors \( {\mathbf{u}}^{ \bot } = \{ {\mathbf{P}}_{\text{s}} {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \) and \( {\mathbf{v}} = \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \) as in (19) and (20), we can find the angles of non-orthogonality \( \alpha \), defined as \( \alpha = \pi /2 - \beta \). The angles \( \alpha \) obtained on the basis of these vectors will obviously not be consistent (both in number and value) with those that result from \( \beta_{j,k} \), i.e. \( \alpha_{j,k} =\uppi/2 - \beta_{j,k} \).

To find the relationship between the angles of non-orthogonality \( \alpha \) and the asymmetry indices w, we must also consider the space spanned by the rows of H. So, on the basis of (21) we get

$$ {\mathbf{H}}^{\text{T}} = {\mathbf{V}}_{ * } {\varvec{\Sigma}}_{r} {\mathbf{U}}_{r}^{\text{T}} $$

Since, according to (21) \( {\mathbf{V}}_{ * }^{\text{T}} {\mathbf{V}}_{n - r} = {\mathbf{0}} \), the range of \( {\mathbf{H}}^{\text{T}} \) is orthogonal to the nullspace of H. Hence, the angles between the column vectors of H, i.e. \( \{ {\mathbf{H}}\}_{ \bullet i} \), and the row vectors of H, i.e. \( \{ {\mathbf{H}}\}_{i \bullet } \), will be the angles of non-orthogonality of projection H

$$ { \cos }\alpha_{i} = \tfrac{{\left\langle {\{ {\mathbf{H}}\}_{i \bullet } \cdot \{ {\mathbf{H}}\}_{ \bullet i} } \right\rangle }}{{\left| {\{ {\mathbf{H}}\}_{i \bullet } } \right| \, \cdot \, \left| {\{ {\mathbf{H}}\} _{ \bullet i} } \right|}} = \tfrac{{h_{i} }}{{\sqrt {\{ {\mathbf{HH}}^{\text{T}} \}_{ii} } \cdot \sqrt {\{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} } \cdot \, }}\quad i = \, 1, \ldots ,n $$
(22)

Mutual orientation of the vectors \( \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} \), \( [{\mathbf{P}}_{\text{s}} \{ {\mathbf{A}}_{\text{s}} \}_{ \bullet i} ]^{ \bot } \) and \( \{ {\mathbf{H}}\}_{i \bullet } \),\( \{ {\mathbf{H}}\}_{ \bullet i} \) together with indication of the angles \( \alpha \) and β is presented schematically in Fig. 1. The angles are deliberately not indexed to show only their definitional relationship, i.e. \( \alpha = \pi /2 - \beta \).

Fig. 1
figure 1

Mutual orientation of the vectors forming the angles α and β

Full size image

On the basis of formulas (6) and (22), we get the relationship between the asymmetry indices \( w_{i} \) and the angles of non-orthogonality \( \alpha_{i} \)

$$ \begin{aligned} { \cos }\alpha_{i} & = \frac{{w_{i} + \{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} }}{{\sqrt {\{ {\mathbf{HH}}^{\text{T}} \}_{ii} } \cdot \sqrt {\{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} } }}\quad i = \, 1, \ldots ,n \\ w_{i} & = \sqrt {\{ {\mathbf{HH}}^{\text{T}} \}_{ii} } \cdot \sqrt {\{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} } { \cos }\alpha_{i} - \{ {\mathbf{H}}^{\text{T}} {\mathbf{H}}\}_{ii} \quad i = 1, \ldots ,n \\ \end{aligned} $$
(23)

As could be expected the increase in observation correlations exerts a distorting effect on the range of projection H. This can be explained by the character of variability of the ratio (condition number) \( k^{ * } = \frac{{\lambda_{\rm max}^{ * } }}{{\lambda_{\rm min}^{ * } }} \) for the matrix \( {\mathbf{P}}_{\text{s}} \) with the decrease in det \( {\mathbf{C}}_{\text{s}} \).

On the basis of spectral decomposition of \( {\mathbf{C}}_{\text{s}} \), i.e. \( {\mathbf{C}}_{\text{s}} \) = WΛWT, and the definition (3), i.e. \( {\mathbf{C}}_{\text{s}} = q{\mathbf{P}}_{\text{s}}^{ - 1} \), we get \( {\mathbf{P}}_{\text{s}} = {\mathbf{W}}(q{\varvec{\Lambda}}^{ - 1} ){\mathbf{W}}^{\text{T}} \) and \( \lambda_{i}^{ * } = q\lambda_{i}^{ - 1} \) (i = 1, …, n). We can check that \( \, \mathop \Pi \nolimits_{1}^{n} \lambda_{i}^{ * } = { \det }{\mathbf{P}}_{\text{s}} = 1 \).

Furthermore, we get \( \lambda_{\rm max}^{ * } = q\lambda_{\rm min}^{ - 1} \), \( \lambda_{\rm min}^{ * } = q\lambda_{\rm max}^{ - 1} \), and finally, \( k^{ * } = \frac{{\lambda_{\rm max}^{ * } }}{{\lambda_{\rm min}^{ * } }} = \frac{{\lambda_{\rm max} }}{{\lambda_{\rm min} }} = k \).

With the decrease in det \( {\mathbf{C}}_{\text{s}} \) (and thus, with the increase in \( \rho_{\text{G}} \)), the ratio k, and hence \( k^{ * } \), increases. With \( \mathop \Pi \nolimits_{1}^{n} \lambda_{i} \to 0 \), both these ratios tend to infinity. This shows that the increase in observation correlations distorts more and more the angles between the vectors of the range of the operator H.

6 Results of numerical tests and their discussion

6.1 Test networks

We have adopted the test networks used in our earlier study devoted to the effect of observation correlations. The levelling networks V1, V2 and a horizontal network H, each treated as free network, are shown in Fig. 2.

Fig. 2
figure 2

Test networks

Full size image

Their brief characteristics contain the following features:

  • the range of internal reliability indices for \( {\mathbf{C}}_{\text{s}} = {\mathbf{I}} \) [in brackets];

  • number of observations n and redundancy f.

The X and Y coordinates for a horizontal network are provided in Table 1.

Table 1 Point coordinates for a horizontal network
Full size table

6.2 Results of the tests

Figures 3, 4, 5, 6, 7 and 8 show graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) built for quantities x such as:\( \, \tfrac{1}{n}\sum\nolimits_{n} {h_{i}^{2} } \), \( \Delta d_{h} \), \( (h_{{\rm min} } ,h_{{\rm max} } ) \), \( \bar{w} \), \( (w_{{\rm min} } ,w_{{\rm max} } ) \), \( \bar{r} \), \( d_{r} \), \( \bar{g} \), \( d_{g} \). The graphs based on generated \( {\mathbf{C}}_{\text{s}} \) configurations are marked with the symbol \( {\mathbf{C}}_{\text{s}} \), whereas those obtained using constant correlation matrices are marked with the symbol \( {\mathbf{C}}_{{{\text{s,}}a}} \). To minimize the effect of the lack of injectivity of the function f :\( {\mathbf{C}}_{\text{s}} \to { \det }{\mathbf{C}}_{\text{s}} \), the former graphs, except for the minimal and maximal quantities, are the curves joining the mean values of x obtained for the generated \( {\mathbf{C}}_{\text{s}} \) configurations. The asymptotic sections of the \( {\mathbf{C}}_{{{\text{s,}}a}} \) graphs do not appear within the drawings since they start at the values of \( \rho_{\text{G}} \) very close to 1.

Fig. 3
figure 3

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( x = \, \tfrac{1}{n}\sum\nolimits_{n} {h_{i}^{2} } \) and \( x = \Delta d_{h} \) (larger scale is used)

Full size image
Fig. 4
figure 4

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( x = h_{\rm min} \) and \( x = h_{\rm max} \)

Full size image
Fig. 5
figure 5

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( x = \bar{w} \)

Full size image
Fig. 6
figure 6

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( {{x}} = w_{{\rm min} } \) and \( {{x}} = w_{{\rm max} } \)

Full size image
Fig. 7
figure 7

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( x = \, \bar{r} \) and \( x = \, \bar{g} \)

Full size image
Fig. 8
figure 8

Graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) for \( x = d_{r} \) and \( x = d_{g} \)

Full size image

The graphs represent a greater number of graphs that have been carried out in the research. Their reduction, aimed at saving space of the present paper, was possible thanks to the previously derived theoretical relationships linking some of the above quantities (Sect. 4).

6.3 Discussion of the results

The graphs in Figs. 3, 4, 5, 6, 7 and 8, supported by rigorous relationships (10)–(16), allow to formulate the following conclusions concerning the effect of increase in observation correlations:

  • as shown in Fig. 3 the values of \( \tfrac{1}{n}\sum\nolimits_{n} {h_{i}^{2} } \) and \( \Delta d_{h} = d_{h} - d_{{h_{I} }} \) (interrelated by (10) and (11)) are increasing. Their increase is considerably smaller for \( {\mathbf{C}}_{{{\text{s,}}a}} \) graphs. The differentiation of \( h_{i} \) values due to observation correlations rises clearly for \( \rho_{\text{G}} \) > 0.6;

  • in compliance with the increase in \( \Delta d_{h} \) (as in Fig. 3), we observe in Fig. 4 an increase in the width of the interval \( (h_{\rm min} ,h_{\rm max} ) \). This effect is much smaller for \( {\mathbf{C}}_{{{\text{s,}}a}} \) graphs;

  • for the network V1 of small redundancy (f = 2) \( h_{{\rm min} } \) comes down to 0 at \( \rho_{\text{G}} \) = 0.8 and assumes negative values for \( \rho_{\text{G}} \) > 0.8, while for V2 with f = 5 \( h_{{\rm min} } \) reaches 0 only at \( \rho_{\text{G}} \) = 0.95). For the network H with f = 18, the increase in the width of the interval \( (h_{\rm min} ,h_{\rm max} ) \) as computed for \( {\mathbf{C}}_{{{\text{s,}}a}} \) configurations is smaller than for networks with f = 2 and f = 5. We may then conclude that the higher the network redundancy, the smaller is its reaction to the increase in observation correlations;

  • the values of \( \bar{w} \) are decreasing (Fig. 5) and the width of the interval \( (w_{{\rm min} } ,w_{{\rm max} } ) \) is increasing (Fig. 6). The lower limit moves towards high negative values. The results are consistent with formulas (12), (13) and the growth of \( \, \tfrac{1}{n}\sum\nolimits_{n} {h_{i}^{ 2} } \) (Fig. 3) and \( \, \bar{g} \) (Fig. 7);

  • as shown in Fig. 7 the increase in \( \bar{r} \) is of similar character as that in \( \bar{g} \). However, for higher values of \( \rho_{\text{G}} \) and especially for \( \rho_{\text{G}} \) approaching 1 the increase in \( \bar{r} \) is greater than that in \( \bar{g} \). This can be due to the influence of the scale factor q (see formula (3)) being in \( \bar{r} \) a magnifying factor \( q^{ - 1} \), while \( \bar{g} \) is independent of q. The differentiation of the \( r_{i} \) values increases faster than that of \( g_{i} \) values (Fig. 8), which can be explained by the similar reason with that in this case a magnifying factor in \( d_{r} \) is \( q^{ - 2} \).

We can see that despite the lack of injectivity of the function f :\( {\mathbf{C}}_{\text{s}} \to { \det }{\mathbf{C}}_{\text{s}} \), the graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) are consistent between themselves and also with the graphs obtained on the basis of \( {\mathbf{C}}_{{{\text{s,}}a}} \) configurations. So, there are reasonable grounds to consider that they yield a general but correct description of the effects of increase in observation correlations upon the measures of internal reliability. Such a description is for use in designing networks with correlated observations, the networks with low redundancies in particular.

7 Analysis of suitability of response-based measures as compared to testing-based measures

It should be noted first of all that the response-based measures \( h_{i} ,w_{i} \) are applicable only for a priori analyses of networks, while the testing-based measures \( r_{i} \) and \( {\text{MDB}}_{{{\text{s,}}i}} \) can be used both for a priori analyses (\( r_{i} ,{\text{ MDB}}_{{{\text{s,}}i}} \)) and for outlier detection (\( r_{i} \)) and identification (\( \sqrt {r_{i} } \)).

Here are the features of the response-based measures important when comparing with the testing-based measures:

  • the pairs (\( h_{i} ,w_{i} \)) are the unambiguous measures of internal reliability of networks with correlated observations, as they are based on two basic characteristics of the oblique projection operator H, i.e. the elements on a main diagonal (\( h_{i} \)) and asymmetry indices (\( w_{i} \)) linked with the angles of non-orthogonality;

  • \( h_{i} \) is a direct response of a model in the ith observation to a single gross error residing in this observation. The \( r_{i} \) and \( {\text{MDB}}_{{{\text{s,}}i}} \) measures characterize the sensitivity of a global model test to a single gross error residing in the ith observation. You can see the difference between the two types of measures particularly clearly in the case when f = 1 (at any level of observation correlations), where the \( h_{i} \) values are mutually differentiated but the \( {\text{MDB}}_{{{\text{s,}}i}} \) ones are all identical. It is obviously an extreme and purely theoretical example, since such networks should be avoided in practice;

  • the criteria for (\( h_{i} ,w_{i} \)) are clearly interpretable since they are formulated in terms of reactions to a gross error (i.e. magnitude and sign of the error compensation and relative strength of the response);

  • the values of \( h_{i} \) are mutually related and, as shown in the present paper, so are the values of \( w_{i} \). Both types are interrelated by the inequality. Unlike \( r_{i} \), they do not depend on the scale factor q;

  • with the increase in observation correlations the differentiation of \( h_{i} \) values increases and the average value of \( w_{i} \) decreases. It means a gradual worsening of the model responses in individual observations. The smaller the redundancy f of the model, the worsening becomes greater. In the extreme case, i.e. when f = 1, at larger values of \( \rho_{\text{G}} \) (0.7 and more) the measures h for some observations may reach negative values as well as values greater than 1. The responses in such observations do not properly compensate the effect of a gross error in terms of both magnitude and sign. There may as well be the responses that are too small (compared to other responses), for the error to be detected and identified;

  • since the MDB only describes the sensitivity of a global model test to the presence of a single gross error in the observations, it should be supplemented with minimal identifiable error MIB (Teunissen 2017; Zaminpardaz and Teunissen 2018; Imparato et al. 2018) or the identifiability index ID (Prószyński 2015), to be aware of the magnitudes of the type III errors. Therefore, for the purposes of a priori analyses, MDBs cannot be considered as autonomous measures. The \( h_{i} ,w_{i} \) measures are only to a certain extent burdened with this disadvantage, since the pairs (\( h_{i} ,w_{i} \)) falling into the outlier-exposing area, except the pairs falling outside, show satisfactory values of ID indices. The conclusion can be that the direct response \( (h_{i} ,w_{i} ) \) in a particular observation contains also some information as to the level of detectability and identifiability of a gross error in that observation.

8 Concluding remarks

Basic characteristics of the reliability matrix, which is an oblique projection operator, describe the responses of a network (or more generally—an observation system) to a single gross error and are sensitive to the increase in observation correlations. This justifies their use as so-called response-based measures of internal reliability (\( h_{i} ,w_{i} \)). The measures can be a supplementation of the testing-based measures \( r_{i} ,{\text{ MDB}}_{{{\text{s,}}i}} \) for a priori analyses of networks carried out at the stage of their design. Although both types of measures show compliance in responding to the increase in observation correlations, they have specific features of their own. So, while taken together they can provide more complete information about the behaviour of networks in the presence of gross errors and about the chances for their detection and identification. For the systems with small redundancy, as may occur in engineering surveys, the response-based measures seem to be specially useful.

The graphs \( x = f(\rho_{\text{G}} |{\mathbf{A}}) \) based on generated \( {\mathbf{C}}_{\text{s}} \) configurations are limited to small network sizes (\( n \le 5 \)) and despite the applied minimizing procedure are affected by the lack of injectivity of the function f :\( {\mathbf{C}}_{\text{s}} \to \rho_{\text{G}} \). The graphs for networks of greater sizes and less affected by this drawback could be obtained by means of more powerful computers and more advanced software than those commonly available used in the present research. Having in mind the need to improve the design methods of systems with correlated observations, it seems purposeful to undertake such studies.

For further studies of correlation effect on the internal reliability of networks, it seems useful to take into account other relevant contributions as for instance those in Wang and Knight (2012) and Yang et al. (2013).