Feedforward Neural Networks

Abstract

This chapter provides a more in-depth description of supervised learning, deep learning, and neural networks—presenting the foundational mathematical and statistical learning concepts and explaining how they relate to real-world examples in trading, risk management, and investment management. These applications present challenges for forecasting and model design and are presented as a reoccurring theme throughout the book. This chapter moves towards a more engineering style exposition of neural networks, applying concepts in the previous chapters to elucidate various model design choices.

Appendix

1.1 Answers to Multiple Choice Questions

Question 1

Answer: 1, 2, 3, 4. All answers are found in the text.

Question 2

Answer: 1,2. A feedforward architecture is always convex w.r.t. each input variable if every activation function is convex and the weights are constrained to be either all positive or all negative. Simply using convex activation functions is not sufficient, since the composition of a convex function and the affine transformation of a convex function do not preserve the convexity. For example, if σ(x) = x ², w = −1, and b = 1, then σ(wσ(x) + b) = (−x ² + 1)² is not convex in x.

A feedforward architecture with positive weights is a monotonically increasing function of the input for any choice of monotonically increasing activation function.

The weights of a feedforward architecture need not be constrained for the output of a feedforward network to be bounded. For example, activating the output with a softmax function will bound the output. Only if the output is not activated, should the weights and bias in the final layer be bounded to ensure bounded output.

The bias terms in a network shift the output but also effect the derivatives of the output w.r.t. to the input when the layer is activated.

Question 3

Answer: 1,2,3,4. The training of a neural network involves minimizing a loss function w.r.t. the weights and biases over the training data. L ₁ regularization is used during model selection to penalize models with too many parameters. The loss function is augmented with a Lagrange penalty for the number of weights. In deep learning, regularization can be applied to each layer of the network. Therefore each layer has an associated regularization parameter. Back-propagation uses the chain rule to update the weights of the network but is not guaranteed to convergence to a unique minimum. This is because the loss function is not convex w.r.t. the weights. Stochastic gradient descent is a type of optimization method which is implemented with back-propagation. There are variants of SGD, however, such as adding Nestov’s momentum term , ADAM , or RMSProp .

1.2 Back-Propagation

Let us consider a feedforward architecture with an input layer, L − 1 hidden layers, and one output layer, with K units in the output layer for classification of K categories. As a result, we have L sets of weights and biases (W ^(ℓ), b ^(ℓ)) for ℓ = 1, …, L, corresponding to the layer inputs Z ^(ℓ−1) and outputs Z ^(ℓ) for ℓ = 1, …, L. Recall that each layer is an activation of a semi-affine transformation, I ^(ℓ)(Z ^(ℓ−1)) := W ^(L)Z ^(ℓ−1) + b ^(L). The corresponding activation functions are denoted as σ ^(ℓ). The activation function for the output layer is a softmax function, σ _s(x).

Here we use the cross-entropy as the loss function, which is defined as

$$\displaystyle \begin{aligned}\mathcal{L}:= -\sum_{k=1}^{K}Y_{k}\log \hat{Y}_{k}.\end{aligned} $$

The relationship between the layers, for ℓ ∈{1, …, L} are

$$\displaystyle \begin{aligned} \hat{Y} (X) & = Z^{(L)}=\sigma_s(I^{(L)}) \in [0,1]^{K},\\ Z^{(\ell)} & = \sigma^{(\ell)} \left ( I^{(\ell)} \right ), ~\ell=1,\dots,L-1,\\ Z^{(0)} & = X.\end{aligned} $$

The update rules for the weights and biases are

$$\displaystyle \begin{aligned} \Delta W^{(\ell)} &= - \gamma \nabla_{W^{(\ell)}}\mathcal{L},\\ \Delta {\mathbf{b}}^{(\ell)} &= - \gamma \nabla_{{\mathbf{b}}^{(\ell)}}\mathcal{L}. \end{aligned} $$

We now begin the back-propagation, tracking the intermediate calculations carefully using Einstein summation notation.

For the gradient of $\mathcal {L}$ w.r.t. W ^(L) we have

$$\displaystyle \begin{aligned} \frac{\partial \mathcal{L}}{\partial w_{ij}^{(L)}} &= \sum_{k=1}^{K}\frac{\partial \mathcal{L}}{\partial Z_{k}^{(L)}} \frac{\partial Z_{k}^{(L)}}{\partial w_{ij}^{(L)}}\\ &= \sum_{k=1}^{K}\frac{\partial \mathcal{L}}{\partial Z_{k}^{(L)}} \sum_{m=1}^{K}\frac{\partial Z_{k}^{(L)}}{\partial I_{m}^{(L)}} \frac{\partial I_{m}^{(L)}}{\partial w_{ij}^{(L)}} \end{aligned} $$

But

$$\displaystyle \begin{aligned} \frac{\partial \mathcal{L}}{\partial Z_{k}^{(L)}} &= -\frac{Y_{k}}{Z_{k}^{(L)}}\\ \frac{\partial Z_{k}^{(L)}}{\partial I_{m}^{(L)}} &= \frac{\partial}{\partial I_{m}^{(L)}}[\sigma(I^{(L)})]_{k}\\ &= \frac{\partial}{\partial I_{m}^{(L)}} \frac{\exp[I_{k}^{(L)}]}{\sum_{n=1}^{K}\exp[I_{n}^{(L)}]}\\ &= \begin{cases} -\frac{\exp[I_{k}^{(L)}]}{\sum_{n=1}^{K}\exp[I_{n}^{(L)}]} \frac{\exp[I_{m}^{(L)}]}{\sum_{n=1}^{K}\exp[I_{n}^{(L)}]} & \text{if } k \neq m \\ \frac{\exp[I_{k}^{(L)}]}{\sum_{n=1}^{K}\exp[I_{n}^{(L)}]} - \frac{\exp[I_{k}^{(L)}]} {\sum_{n=1}^{K}\exp[I_{n}^{(L)}]} \frac{\exp[I_{m}^{(L)}]}{\sum_{n=1}^{K}\exp[I_{n}^{(L)}]} & \text{otherwise} \end{cases}\\ &= \begin{cases} -\sigma_{k}\sigma_{m}& \text{if } k \neq m \\ \sigma_k(1 - \sigma_m) & \text{otherwise} \end{cases}\\ &= \sigma_k(\delta_{km} - \sigma_m) \quad \text{where} \, \delta_{km} \, \text{is the Kronecker's Delta}\\ \frac{\partial I_{m}^{(L)}}{\partial w_{ij}^{(L)}} &= \delta_{mi}Z_{j}^{(L-1)}\\ \implies \frac{\partial \mathcal{L}}{\partial w_{ij}^{(L)}} &= -\sum_{k=1}^{K}\frac{Y_{k}}{Z_{k}^{(L)}} \sum_{m=1}^{K} Z_{m}^{(L)}(\delta_{km} - Z_{m}^{(L)}) \delta_{mi}Z_{j}^{(L-1)}\\ &= -Z_{j}^{(L-1)} \sum_{k=1}^{K}Y_{k} (\delta_{ki} - Z_{i}^{(L)}) \\ &= Z_{j}^{(L-1)} (Z_{i}^{(L)}-Y_{i}), \end{aligned} $$

where we have used the fact that $\sum _{k=1}^{K}Y_{k}=1$ in the last equality. Similarly for b ^(L), we have

$$\displaystyle \begin{aligned} \frac{\partial \mathcal{L}}{\partial b_{i}^{(L)}} &= \sum_{k=1}^{K}\frac{\partial \mathcal{L}}{\partial Z_{k}^{(L)}} \sum_{m=1}^{K}\frac{\partial Z_{k}^{(L)}}{\partial I_{m}^{(L)}} \frac{\partial I_{m}^{(L)}}{\partial b_{i}^{(L)}}\\ &= Z_{i}^{(L)}-Y_{i} \end{aligned} $$

It follows that

$$\displaystyle \begin{aligned} \nabla_{{\mathbf{b}}^{(L)}}\mathcal{L} &= Z^{(L)}-Y\\ \nabla_{W^{(L)}}\mathcal{L} &= \nabla_{{\mathbf{b}}^{(L)}}\mathcal{L} \otimes {Z^{(L-1)}}, \end{aligned} $$

where ⊗ denotes the outer product.

Now for the gradient of $\mathcal {L}$ w.r.t. W ^(L−1) we have

$$\displaystyle \begin{aligned} \frac{\partial \mathcal{L}}{\partial w_{ij}^{(L-1)}} &= \sum_{k=1}^{K}\frac{\partial L}{\partial Z_{k}^{(L)}} \frac{\partial Z_{k}^{(L)}}{\partial w_{ij}^{(L-1)}}\\ &= \sum_{k=1}^{K}\frac{\partial \mathcal{L}}{\partial Z_{k}^{(L)}} \sum_{m=1}^{K}\frac{\partial Z_{k}^{(L)}}{\partial I_{m}^{(L)}} \sum_{n=1}^{n^{(L-1)}} \frac{\partial I_{m}^{(L)}}{\partial Z_{n}^{(L-1)}} \sum_{p=1}^{n^{(L-1)}} \frac{\partial Z_{n}^{(L-1)}}{\partial I_{p}^{(L-1)}} \frac{\partial I_{p}^{(L-1)}}{\partial w_{ij}^{(L-1)}} \end{aligned} $$

If we assume that σ ^(ℓ)(x) = sigmoid(x), ℓ ∈{1, …, L − 1}, then

$$\displaystyle \begin{aligned} \frac{\partial I_{m}^{(L)}}{\partial Z_{n}^{(L-1)}} &= w_{mn}^{(L)}\\ \frac{\partial Z_{n}^{(L-1)}}{\partial I_{p}^{(L-1)}} &= \frac{\partial}{\partial I_{p}^{(L-1)}}\bigg(\frac{1}{1+\exp(-I_{n}^{(L-1)})}\bigg)\\ &= \frac{1}{1+\exp(-I_{n}^{(L-1)})} \frac{\exp(-I_{n}^{(L-1)})}{1+\exp(-I_{n}^{(L-1)})} \, \delta_{np} \\ &= Z_{n}^{(L-1)} (1-Z_{n}^{(L-1)}) \, \delta_{np} = \sigma^{(L-1)}_n(1-\sigma^{(L-1)}_n)\delta_{np} \\ \frac{\partial I_{p}^{(L-1)}}{\partial w_{ij}^{(L-1)}} &= \delta_{pi} Z_{j}^{(L-2)} \\ \implies \frac{\partial L}{\partial w_{ij}^{(L)}} &= -\sum_{k=1}^{K}\frac{Y_{k}}{Z_{k}^{(L)}} \sum_{m=1}^{K}Z_{k}^{(L)}(\delta_{km} - Z_{m}^{(L)})\\ &\qquad \sum_{n=1}^{n^{(L-1)}} w_{mn}^{(L)} \sum_{p=1}^{n^{(L-1)}} Z_{n}^{(L-1)} (1-Z_{n}^{(L-1)}) \, \delta_{np} \delta_{pi} Z_{j}^{(L-2)} \\ &= -\sum_{k=1}^{K}Y_{k} \sum_{m=1}^{K}(\delta_{km} - Z_{m}^{(L)}) \sum_{n=1}^{n^{(L-1)}} w_{mn}^{(L)} Z_{n}^{(L-1)} (1-Z_{n}^{(L-1)}) \, \delta_{ni} Z_{j}^{(L-2)} \\ &= -\sum_{k=1}^{K}Y_{k} \sum_{m=1}^{K}(\delta_{km} - Z_{m}^{(L)}) w_{mi}^{(L)} Z_{i}^{(L-2)} (1-Z_{i}^{(L-1)}) Z_{j}^{(L-2)} \\ &= -Z_{j}^{(L-2)}Z_{i}^{(L-1)}(1-Z_{i}^{(L-1)}) \sum_{m=1}^{K} w_{mi}^{(L)} \sum_{k=1}^{K}(\delta_{km}Y_{k} - Z_{m}^{(L)}Y_{k}) \\ &= Z_{j}^{(L-2)}Z_{i}^{(L-1)} (1-Z_{i}^{(L-1)}) \sum_{m=1}^{K} w_{mi}^{(L)} (Z_{m}^{(L)} - Y_{m}) \\ &= Z_{j}^{(L-2)}Z_{i}^{(L-1)} (1-Z_{i}^{(L-1)}) (Z^{(L)} - Y)^{T} {\mathbf{w}}_{,i}^{(L)} \\ \end{aligned} $$

Similarly we have

$$\displaystyle \begin{aligned}\frac{\partial \mathcal{L}}{\partial b_{i}^{(L-1)}} = Z_{i}^{(L-1)} (1-Z_{i}^{(L-1)}) (Z^{(L)} - Y)^{T} {\mathbf{w}}_{,i}^{(L)}.\end{aligned}$$

It follows that we can define the following recursion relation for the loss gradient:

$$\displaystyle \begin{aligned} \nabla_{b^{(L-1)}}\mathcal{L} &= Z^{(L-1)} \circ (\mathbf{1}-Z^{(L-1)}) \circ ({W^{(L)}}^{T} \nabla_{b^{(L)}}\mathcal{L}) \\ \nabla_{W^{(L-1)}}\mathcal{L} &= \nabla_{b^{(L-1)}}\mathcal{L} \otimes Z^{(L-2)}\\ & = Z^{(L-1)} \circ (\mathbf{1}-Z^{(L-1)}) \circ ({W^{(L)}}^{T} \nabla_{W^{(L)}}\mathcal{L}), \end{aligned} $$

where ∘ denotes the Hadamard product (element-wise multiplication). This recursion relation can be generalized for all layers and choice of activation functions. To see this, let the back-propagation error $\delta ^{(\ell )}:=\nabla _{b^{(\ell )}}\mathcal {L}$, and since

$$\displaystyle \begin{aligned} \left[\frac{\partial \sigma^{(\ell)}}{\partial I^{(\ell)}}\right]_{ij}&=\frac{\partial \sigma_i^{(\ell)}}{\partial I_j^{(\ell)}}\\ &=\sigma_i^{(\ell)}(1-\sigma_i^{(\ell)})\delta_{ij} \end{aligned} $$

or equivalently in matrix–vector form

$$\displaystyle \begin{aligned}\nabla_{I^{(\ell)}} \sigma^{(\ell)}=\text{diag}(\sigma^{(\ell)} \circ (\mathbf{1}-\sigma^{(\ell)})),\end{aligned}$$

we can write, in general, for any choice of activation function for the hidden layer,

$$\displaystyle \begin{aligned}\delta^{(\ell)}=\nabla_{I^{(\ell)}} \sigma^{(\ell)}(W^{(\ell+1)})^T\delta^{(\ell+1)},\end{aligned}$$

and

$$\displaystyle \begin{aligned}\nabla_{W^{(\ell)}}\mathcal{L} = \delta^{(\ell)} \otimes Z^{(\ell-1)}.\end{aligned}$$

1.3 Proof of Theorem 4.2

Using the same deep structure shown in Fig. 4.9, Liang and Srikant (2016) find the binary expansion sequence {x ₀, …, x _n}. In this step, they used n binary steps units in total. Then they rewrite $g_{m+1}(\sum _{i=0}^{n}\frac {x_{i}}{2^{n}})$,

$$\displaystyle \begin{aligned} g_{m+1}\left(\sum_{i=0}^{n}\frac{x_{i}}{2^{i}}\right)&=\sum_{j=0}^{n}\left[x_{j}\cdot\frac{1}{2^{j}}g_{m}\left(\sum_{i=0}^{n}\frac{x_{i}}{2^{i}}\right)\right]\\ &=\sum_{j=0}^{n}\max\left[2(x_{j}-1)+\frac{1}{2^j}g_{m}\left(\sum_{i=0}^{n}\frac{x_{i}}{2^{i}}\right),0\right].{} \end{aligned} $$

(4.57)

Clearly Eq. 4.57 defines iterations between the outputs of neighboring layers. Defining the output of the multilayer neural network as $\hat {f}(x)=\sum _{i=0}^{p}a_{i}g_{i}\left (\sum _{j=0}^{n}\frac {x_{j}}{2^{j}}\right ).$ For this multilayer network, the approximation error is

$$\displaystyle \begin{aligned} |f(x)-\hat{f}(x)|&=\left|\sum_{i=0}^{p}a_{i}g_{i}\left(\sum_{j=0}^{n}\frac{x_{j}}{2^{j}}\right)-\sum_{i=0}^{p}a_{i}x^{i}\right|\\ &\le \sum_{i=0}^{p}\left[|a_{i}|\cdot\left|g_{i}\left(\sum_{j=0}^{n}\frac{x_{j}}{2^{j}}\right)-x^{i}\right|~\right]\le\frac{p}{2^{n-1}}. \end{aligned} $$

This indicates, to achieve ε-approximation error, one should choose $n=\left \lceil \log \frac {p}{\varepsilon }\right \rceil +1$. Besides, since $\mathcal {O}(n+p)$ layers with $\mathcal {O}(n)$ binary step units and $\mathcal {O}(pn)$ ReLU units are used in total, this multilayer neural network thus has $\mathcal {O}\left (p+\log \frac {p}{\varepsilon }\right )$ layers, $\mathcal {O}\left (\log \frac {p}{\varepsilon }\right )$ binary step units, and $\mathcal {O}\left (p\log \frac {p}{\varepsilon }\right )$ ReLU units.

1.4 Proof of Lemmas from Telgarsky (2016)

Proof (Proof of 4.1)

Let cI_f denote the partition of $\mathbb {R}$ corresponding to f, and cI_g denote the partition of $\mathbb {R}$ corresponding to g.

First consider f + g, and moreover any intervals U _f ∈cI_f and U _g ∈cI_g. Necessarily, f + g has a single slope along U _f ∩ U _g. Consequently, f + g is |cI|-sawtooth, where cI is the set of all intersections of intervals from cI_f and cI_g, meaning cI := {U _f ∩ U _g : U _f ∈cI_f, U _g ∈cI_g}. By sorting the left endpoints of elements of cI_f and cI_g, it follows that |cI|≤ k + l (the other intersections are empty).

For example, consider the example in Fig. 4.11 with partitions given in Table 4.2. The set of all intersections of intervals from cI_f and cI_g contains 3 elements:

$$\displaystyle \begin{aligned} \mbox{cI}=\{[0,\frac{1}{4}] \cap [0,\frac{1}{2}], (\frac{1}{4},1] \cap [0,\frac{1}{2}], (\frac{1}{4},1] \cap (\frac{1}{2},1]\} \end{aligned} $$

(4.58)

Table 4.2 Definitions of the functions f(x) and g(x)

Now consider f ∘ g, and in particular consider the image f(g(U _g)) for some interval U _g ∈cI_g. g is affine with a single slope along U _g; therefore, f is being considered along a single unbroken interval g(U _g). However, nothing prevents g(U _g) from hitting all the elements of cI_f; since U _g was arbitrary, it holds that f ∘ g is (|cI_f|⋅|cI_g|)-sawtooth. □

Proof

Recall the notation $\tilde f(x) := [f(x) \geq 1/2]$, whereby $\mathcal {E}(f) := \frac {1}{n}\sum _i [y_i\neq \tilde f(x_i)]$. Since f is piecewise monotonic with a corresponding partition $\mathbb {R}$ having at most t pieces, then f has at most 2t − 1 crossings of 1/2: at most one within each interval of the partition, and at most 1 at the right endpoint of all but the last interval. Consequently, $\tilde f$ is piecewise constant, where the corresponding partition of $\mathbb {R}$ is into at most 2t intervals. This means n points with alternating labels must land in 2t buckets, thus the total number of points landing in buckets with at least three points is at least n − 4t. □

1.5 Python Notebooks

The notebooks provided in the accompanying source code repository are designed to gain insight in toy classification datasets. They provide examples of deep feedforward classification, back-propagation, and Bayesian network classifiers. Further details of the notebooks are included in the README.md file.