Aerosol Layer Height over Water via Oxygen A-Band Observations from Space: A Tutorial

Abstract

Aerosols are a highly problematic area in climate science for several reasons. On the one hand, they are at least partially anthropogenic, originating from industrial facilities spewing pollution as well as agricultural activity, seasonal biomass burning, land-use change, and even wood-stove cooking in densely populated regions. On the other hand, aerosols interact in very poorly understood ways with clouds and hence, indirectly, the climate system as a whole (Boucher and Randall 2014).

Appendix: IC Analysis in Optimal Estimation Theory

1.1 Least-Squares Fit of a Forward Model to Data

The standard approach (Bevington and Robinson 1992; Press et al. 1989) to fitting a generally nonlinear forward model ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ to data ${\mathbf {y}}$ is to equate them, as n-dimensional vectors in measurement space, with allowance for random error $\varvec{\epsilon }$:

$$\begin{aligned} {\mathbf {y}}= {\mathbf {F}}({\mathbf {x}};{\mathbf {b}}) + \varvec{\epsilon }. \end{aligned}$$

(31)

where ${\mathbf {x}}$ is the m-dimensional vector in a state space that contains all the parameters used to find the best fit to the data; ${\mathbf {b}}$ is another vector in the space of non- or otherwise-retrieved parameters that are imperfectly known to within a known uncertainty.

We will consider two sources of error in $\varvec{\epsilon }$: (i) instrumental error that affects ${\mathbf {y}}$, and (ii) forward model error that affects ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$. Our main concern in the latter case is uncertainty on ${\mathbf {b}}$, the parts of a larger state vector that have to be treated as given when retrieving ${\mathbf {x}}$. Instrumental error is characterized by its $n \times n$ variance/covariance matrix

$$\begin{aligned} {\mathbf {S}}_{\mathbf {y}}= \mathrm{E}[({\mathbf {y}}-\mathrm{E}[{\mathbf {y}}])({\mathbf {y}}-\mathrm{E}[{\mathbf {y}}])^\mathrm{T}] \end{aligned}$$

(32)

where $\mathrm{E}[\cdot ]$ denotes mathematical expectation. Uncertainty on the non/otherwise-retrieved parameters is defined by their variance/co-variance matrix ${\mathbf {S}}_{\mathbf {b}}$, which can be converted into the equivalent ${\mathbf {S}}_{\mathbf {F}}$ of the measurement error matrix in (32) by using the Jacobian matrix ${\mathbf {J}}_{\mathbf {b}}= \partial {\mathbf {F}}/\partial {\mathbf {b}}$ and its transpose ${\mathbf {J}}_{\mathbf {b}}^\mathrm{T}$:

$$\begin{aligned} {\mathbf {S}}_{\mathbf {F}}= {\mathbf {J}}_{\mathbf {b}}{\mathbf {S}}_{\mathbf {b}}{\mathbf {J}}_{\mathbf {b}}^\mathrm{T}. \end{aligned}$$

(33)

Since we are confident that measurement error in ${\mathbf {y}}$ and modeling error in ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ are independent random variables, their (co)variance matrices just add to form:

$$\begin{aligned} {\mathbf {S}}_{\varvec{\epsilon }} = {\mathbf {S}}_{\mathbf {y}}+ {\mathbf {S}}_{\mathbf {F}}. \end{aligned}$$

(34)

The classic least-squares minimization method determines the estimate

$$\begin{aligned} {\mathbf {x}}_\text {bp} = \mathop {\text {argmin}}\limits _{{\mathbf {x}}}\left[ \Psi _{\mathbf {y}}({\mathbf {x}}) \right] \end{aligned}$$

(35)

for the best possible fit of the data for model ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ by finding the minimum of the cost function

$$\begin{aligned} \Psi _{\mathbf {y}}({\mathbf {x}}) = \frac{1}{2} ({\mathbf {y}}-{\mathbf {F}}({\mathbf {x}};{\mathbf {b}}))^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} ({\mathbf {y}}-{\mathbf {F}}({\mathbf {x}};{\mathbf {b}})). \end{aligned}$$

(36)

In the frequent situation where ${\mathbf {S}}_{\varvec{\epsilon }}$ is diagonal, $\Psi _{\mathbf {y}}({\mathbf {x}})$ is the sum of squared “observed–predicted” residuals down-weighted by the uncertainty on each measurement. Assuming, for the moment, that $\Psi _{\mathbf {y}}({\mathbf {x}})$ is convex, one can use the iterative Gauss-Newton algorithm:

$$\begin{aligned} {\mathbf {x}}_{i+1} = {\mathbf {x}}_{i} + {\mathbf {S}}_{\mathbf {x}}{\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} ({\mathbf {y}}-{\mathbf {F}}({\mathbf {x}}_i;{\mathbf {b}})), \end{aligned}$$

(37)

where ${\mathbf {J}}= \partial {\mathbf {F}}/\partial {\mathbf {x}}$ is the usual $n \times m$ Jacobian matrix, and

$$\begin{aligned} {\mathbf {S}}_{\mathbf {x}}= \left( {\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} {\mathbf {J}}\right) ^{-1}, \end{aligned}$$

(38)

using an arbitrary starting position ${\mathbf {x}}_0$; this method converges in one step if ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ is linear in ${\mathbf {x}}$. Iteration is stopped when the non-dimensional weighted sum of residuals in (36), often denoted as $\chi ^2$/2, is $\approx n$/2.

Note that ${\mathbf {x}}_\text {bp}$ from (35) will not be identical to the true state vector in (31) for at least two basic reasons, starting with the least serious:

• First, the data ${\mathbf {y}}$ are collected with a specific realization of the random instrument noise in $\varvec{\epsilon }$, i.e., the component quantified by ${\mathbf {S}}_{\mathbf {y}}$ in (34). Averaging over different measurements of ${\mathbf {y}}$ will reduce the noise level (by the square-root of the number of independent measurements) but not eliminate it. However, the distance between the true and estimated state vectors is likely to be on the order of the square-root of the diagonal elements of ${\mathbf {S}}_{\mathbf {x}}$ in (38), i.e., the predicted retrieval error for each element of ${\mathbf {x}}$:

  $$\begin{aligned} \text {StDev}[x_j] = \sqrt{S_{{\mathbf {x}}jj}} (j = 1,\dots ,m). \end{aligned}$$

  (39)

• Moreover, most interesting problems, in remote sensing in particular, result in non-convex cost functions in (36), simply because ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ is non-linear. Hence there is no guarantee that it has a single global minimum to find by iterative Gauss-Newton steps. So, even assuming that the true state is near the global minimum of $\Psi _{\mathbf {y}}({\mathbf {x}})$, depending on the initial guess ${\mathbf {x}}_0$ at the state vector, the iterative search could end in a local minimum that is generally not within the distance in (39) of the true state. A well-known mitigation strategy for the non-convexity problem is to use the Levenberg-Marquardt minimization algorithm (Marquardt 1963; Bevington and Robinson 1992) where ${\mathbf {S}}_{\mathbf {x}}^{-1} = {\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} {\mathbf {J}}$ in (38) is replaced by a weighted sum of itself and its diagonal elements. The weight starts high on the diagonal matrix, which make the iteration behave like a straightforward gradient descent, which can work for non-convex function minimization (away from inflection points). As the weight shifts away from the diagonal, the method reverts to Gauss-Newton steps as the correct minimum is approached in a state-space region where $\Psi _{\mathbf {y}}({\mathbf {x}})$ is locally convex.

Finally, the whole inverse problem at hand can be ill-posed, that is, ${\mathbf {S}}_{\mathbf {x}}^{-1}$ can be nearly singular, thus making its inversion in (38) highly unstable. The trajectory of iterative minimization in (37) will then be very sensitive to small perturbations of ${\mathbf {y}}$, e.g., instrumental noise. We now introduce a solution to this last issue since in remote sensing there is often much redundancy in the observations with respect to the unknowns in the retrieval.

1.2 Optimal Estimation

Rodgers’ (2000) theory of optimal estimation (OE) revisits the above inverse problem of parameter estimation in the forward signal model from a Bayesian perspective, as a means for dealing with chronic ill-posedness by introducing regularization. Bayes theorem relates two conditional probabilities and two unconditional counterparts:

$$\begin{aligned} P({\mathbf {x}}|{\mathbf {y}}) = P({\mathbf {y}}|{\mathbf {x}}) P({\mathbf {x}}) / P({\mathbf {y}}), \end{aligned}$$

(40)

where the last one is just a normalization factor of no particular interest here. $P({\mathbf {x}})$ is the “prior” or“a priori” probability, while $P({\mathbf {x}}|{\mathbf {y}})$ is the “posterior” probability.

Letting $|\cdot |$ denote the determinant of a square matrix, and assuming gaussian distributions, we have

$$\begin{aligned} \log P({\mathbf {x}}|{\mathbf {y}}) = -\frac{1}{2}\left[ ({\mathbf {x}}-\hat{{\mathbf {x}}})^\mathrm{T}{\mathbf {S}}_{\mathbf {x}}^{-1} ({\mathbf {x}}-\hat{{\mathbf {x}}}) + \log |{\mathbf {S}}_{\mathbf {x}}| + m\log (2\pi )\right] \end{aligned}$$

(41)

for the posterior probability of atmospheric state properties ${\mathbf {x}}$, given data ${\mathbf {y}}$. Also, we note that $\hat{{\mathbf {x}}}$ is a new estimate of the prevailing state vector. Both $\hat{{\mathbf {x}}}$ and ${\mathbf {S}}_{\mathbf {x}}$ will depend on ${\mathbf {y}}$, ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$, and related quantities, as shown below. In addition, we have

$$\begin{aligned} \log P({\mathbf {x}}) = -\frac{1}{2}\left[ ({\mathbf {x}}-{\mathbf {x}}_\text {a})^\mathrm{T}{\mathbf {S}}_\text {a}^{-1}({\mathbf {x}}-{\mathbf {x}}_\text {a}) + \log |{\mathbf {S}}_\text {a}| + m\log (2\pi ) \right] \end{aligned}$$

(42)

for the a priori probability of atmospheric state, i.e., what we know about it before any observations are made. Finally, we have

$$\begin{aligned} \log P({\mathbf {y}}|{\mathbf {x}}) = -\Psi _{\mathbf {y}}({\mathbf {x}}) - \frac{1}{2} \log \left[ (2\pi )^n|{\mathbf {S}}_{\varvec{\epsilon }}|\right] \end{aligned}$$

(43)

for the probability of obtaining the specific observations ${\mathbf {y}}$, given the atmospheric state ${\mathbf {x}}$, i.e., probability of seeing the random residuals used in (36) to estimate the cost function $\Psi _{\mathbf {y}}({\mathbf {x}})$.

One then defines $\hat{{\mathbf {x}}}$ as the state with maximum likelihood, i.e., that maximizes $P({\mathbf {y}}|{\mathbf {x}})P({\mathbf {x}})$, a product of two gaussian PDFs. In other words,

$$\begin{aligned} \hat{{\mathbf {x}}} = \mathop {\text {argmin}}\limits _{{\mathbf {x}}}\left[ \Psi _{\mathbf {y}}({\mathbf {x}}) + \frac{1}{2}({\mathbf {x}}-{\mathbf {x}}_\text {a})^\mathrm{T}{\mathbf {S}}_\text {a}^{-1}({\mathbf {x}}-{\mathbf {x}}_\text {a}) \right] , \end{aligned}$$

(44)

instead of (35). The Gauss-Newton algorithm in (37) still applies but, rather than (38), we now have

$$\begin{aligned} {\mathbf {S}}_{\mathbf {x}}= \left( {\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} {\mathbf {J}}+ {\mathbf {S}}_\text {a}^{-1} \right) ^{-1}. \end{aligned}$$

(45)

Although there may be better choices, one can always start the iterations with ${\mathbf {x}}_0 = {\mathbf {x}}_\text {a}$. As in (39), the diagonal elements $S_{{\mathbf {x}}jj}$ of ${\mathbf {S}}_{\mathbf {x}}$ are the posterior estimates of variance on the retrieved properties in $\hat{{\mathbf {x}}}$.

1.3 Degrees of Freedom

At any rate, and contrary to (38), the matrix inversion problem in (45) is by design well-posed, thanks to the presence of ${\mathbf {S}}_\text {a}^{-1}$. However, convergence goes to $\hat{{\mathbf {x}}}$ rather than ${\mathbf {x}}_\text {bp}$ in (35).

It is therefore of interest to evaluate the $m \times m$ matrix

$$\begin{aligned} {\mathbf {A}}= \left( {\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} {\mathbf {J}}+ {\mathbf {S}}_\text {a}^{-1} \right) ^{-1} {\mathbf {J}}^\mathrm{T}{\mathbf {S}}_{\varvec{\epsilon }}^{-1} {\mathbf {J}}, \end{aligned}$$

(46)

i.e., the left-hand product of (45) with the potentially ill-conditioned matrix to be inverted in (38). Recall that these expressions are the predicted uncertainties on the retrieved state properties with and without regularization, respectively; in the latter case, however, the potentially unstable matrix inversion is not performed. A little algebra leads to the simpler expression

$$\begin{aligned} {\mathbf {A}}= {\mathbf {I}}_m - {\mathbf {S}}_{\mathbf {x}}{\mathbf {S}}_\text {a}^{-1}, \end{aligned}$$

(47)

where ${\mathbf {I}}_m$ is the $m \times m$ identity matrix. Now, in OE theory, ${\mathbf {A}}$ is known as the “averaging kernel,” and Rodgers (2000) shows that it can be defined simply as $\partial \hat{{\mathbf {x}}}/\partial {\mathbf {x}}$, where ${\mathbf {x}}$ is the true state parameter.

If ${\mathbf {S}}_\text {a}$ is diagonal (no prior covariances), then the diagonal terms of ${\mathbf {A}}$ are

$$\begin{aligned} A_{jj} = 1 - \frac{S_{{\mathbf {x}}jj}}{S_{\text {a}jj}} \end{aligned}$$

(48)

for $j = 1,\dots ,m$. Since we can anticipate that $0 \le S_{{\mathbf {x}}jj} \le S_{\text {a}jj}$, we have $A_{jj} \in [0,1]$. $A_{jj}$ is known as the partial “degree of freedom” (DOF) for the retrieved property $x_j$. It is directly related to the predicted retrieval error

$$\begin{aligned} \text {StDev}[x_j] = \sqrt{S_{\text {a}jj}\times (1-A_{jj})} = \text {StDev}_\text {a}[x_j] \sqrt{1-A_{jj}}, \end{aligned}$$

(49)

recalling that ${\mathbf {S}}_\text {a}$ is taken as diagonal.

Following the methodology of Merlin et al. (2016), we use $A_{jj}$ and StDev[$x_j$] from (48)–(49) extensively in the main body of this article. The former is an intuitive non-dimensional metric of the value added by the observations projected onto a specific state variable. If $S_{{\mathbf {x}}jj} \ll S_{\text {a}jj}$, then $A_{jj}$ approaches unity, which means that the observations have vastly improved our knowledge of the state variable $x_j$. If, on the contrary, $S_{{\mathbf {x}}jj} \lesssim S_{\text {a}jj}$, then $A_{jj}$ approaches zero, which means that the observations have not helped very much.

Figure 7 illustrates for $m = 2$ a typical concentration of probability measure (IC increase) in a familiar state space when going from the prior to posterior PDFs for ${\mathbf {x}}$. In both cases, we have traced, assuming gaussian PDFs, the lines of iso-probability value at $1/(2\pi )\sqrt{\mathrm{e}|{\mathbf {S}}|}$, equivalently, where $({\mathbf {x}}-\mathrm{E}[{\mathbf {x}}])^\mathrm{T}{\mathbf {S}}^{-1}({\mathbf {x}}-\mathrm{E}[{\mathbf {x}}]) = 1$ for easy visualization of the magnitudes of prior and posterior standard deviations. The area of each ellipsoid is $\propto$ $|{\mathbf {S}}|$. More specifically, the regions inside the ellipses both account for 68% of the prior and posterior events. Note that, while ${\mathbf {S}}_\text {a}$ is diagonal (no covariance), ${\mathbf {S}}_{\mathbf {x}}$ is generally not.

Fig. 7

Schematic representation of an OE for $m = 2$ atmospheric state parameters relevant to the investigation presented in the main body. Ellipses are iso-likelihood contours where we have $({\mathbf {x}}-\mathrm{E}[{\mathbf {x}}])^\mathrm{T}{\mathbf {S}}^{-1}({\mathbf {x}}-\mathrm{E}[{\mathbf {x}}]) = 1$, and these curves are traced respectively for the prior (grey) and posterior (black)

A better-known quantification of overall retrieval performance in OE theory is

$$\begin{aligned} \text {Tr}[{\mathbf {A}}] = \sum \limits _{j=1}^m A_{jj} \in [0,m], \end{aligned}$$

(50)

which is the (implicitly, total) number of Degrees of Freedom. It approximates the maximum number of state properties (among m) that can be retrieved from the n observations in ${\mathbf {y}}$ in view of: (i) the instrumental noise (${\mathbf {S}}_{\mathbf {y}}$); (ii) forward model error (${\mathbf {S}}_{\mathbf {F}}$), in this case, from uncertainty in non-retrieved parameters (${\mathbf {S}}_{\mathbf {b}}$); (iii) sensitivities of the forward model to the state variables to be retrieved (${\mathbf {J}}$), or not (${\mathbf {J}}_{\mathbf {b}}$); and (iv) prior information about the atmospheric state (${\mathbf {S}}_\text {a}$).

The off-diagonal terms in ${\mathbf {A}}$ are non-dimensional quantities of the type $\mathrm{E}\left[ (x_1-\hat{x}_1)(x_2-\hat{x}_2) \right]$ divided by a diagonal element of ${\mathbf {S}}_\text {a}$, again assuming it is diagonal. These normalized cross-correlations describe how different state variables interfere (statistically) with one-another, two-by-two. Keeping these numbers small results in more regular (hence, more easily inverted) matrix ${\mathbf {S}}_{\mathbf {x}}^{-1}$. That, in turn, will help the performance of the retrieval algorithm. In the design phase of the observation system, the forward model ${\mathbf {F}}({\mathbf {x}};{\mathbf {b}})$ can thus be used to optimize the sampling of ${\mathbf {y}}$ in a way that keeps cross-variable interference as small as possible.

1.4 Entropy and Information Content

Lastly, we relate DOFs to information theoretical concepts, and thus justify our claim all along that we are quantifying Information Content per se. Following Shannon (1948), Rodgers (1998) defines the increase in information (equivalently, decrease in entropy) associated with the acquisition of observations ${\mathbf {y}}$ and their processing—by, e.g., OE methods—as

$$\begin{aligned} -\Delta H = -\frac{\log _2(|{\mathbf {S}}_{\mathbf {x}}|)-\log _2(|{\mathbf {S}}_\text {a}|)}{2} = -\frac{1}{2}\log _2(|{\mathbf {S}}_{\mathbf {x}}{\mathbf {S}}_\text {a}^{-1}|) = -\frac{1}{2}\log _2(|{\mathbf {I}}_m - {\mathbf {A}}|), \end{aligned}$$

(51)

when expressed in bits. Information gain $-\Delta H$ ranges from 0$^+$ ($|{\mathbf {S}}_{\mathbf {x}}| \lesssim |{\mathbf {S}}_\text {a}|$) to $\infty$ ($|{\mathbf {S}}_{\mathbf {x}}| \ll |{\mathbf {S}}_\text {a}|$). This follows directly from the expression for the entropy of a generic m-dimensional gaussian probability density function, $P_m(\mathrm{E}[{\mathbf {x}}],{\mathbf {S}})$: from, e.g., (42), we have

$$\begin{aligned} H(m,|{\mathbf {S}}|) = -\mathrm{E}[\log P_m(\mathrm{E}[{\mathbf {x}}],{\mathbf {S}})] = \frac{1}{2}\log ((\mathrm{e}2\pi )^m|{\mathbf {S}}|), \end{aligned}$$

(52)

which is naturally independent of the mean $\mathrm{E}[{\mathbf {x}}]$. To visualize this IC increase (entropy decrease), $-\Delta H$ in (51) is $\nicefrac {1}{2}$ the log of the ratio of the areas of the two ellipsoids in Fig. 7.

Abbreviations & Acronyms

 

1D:

one-dimensional

AOT:

aerosol optical thickness

BC:

boundary condition

BRDF:

Bi-directional Reflection Distribution Function

BRF:

Bi-directional Reflection Factor

DOAS:

Differential Optical Absorption Spectroscopy

DOF:

Degree(s) Of Freedom

FWHM:

Full-Width Half-Max

GSFC:

Goddard Space Flight Center

IC:

Information Content

IR:

infra-red

JPL:

Jet Propulsion Laboratory

MAIA:

Multi-Angle Imager for Aerosols

MAS:

Multi-Angle Sensor

NASA:

National Aeronautics and Space Agency

OE:

Optimal Estimation

OCI:

Ocean Color Imager

PACE:

Plankton, Aerosol, Cloud, and ocean Ecosystem

RT:

radiative transfer

SNR:

signal-to-noise ratio

SSA:

single scattering albedo

StDev:

Standard Deviation

SZA:

solar zenith angle

TOA:

Top-of-Atmosphere

UV:

ultra-violet

VNIR:

visible/near-infrared

VIS:

visible

VZA:

viewing zenith angle