SAOSA: Stable Adaptive Optimization for Stacked Auto-encoders

Abstract

The stacked auto-encoders are considered deep learning algorithms automatically extracting meaningful unsupervised features from the input data using a hierarcfhical learning process. The parameters are learnt layer-by-layer in each auto-encoder (AE). As optimization is one of the main components of the neural networks and auto-encoders, the learning rate is one of the crucial hyper-parameters of neural networks and AE. This issue on a large scale and especially sparse data sets is more important. In this paper, we adapt the learning rate for special AE corresponding to various components of AE networks in each stochastic gradient calculation and analyze the theoretical convergence of back-propagation learning for the proposed method. We also promote our methodology for online adaptive optimizations suitable for deep learning. We obtain promising results compared to constant learning rates on the (1) MNIST digit, (2) blogs-Gender-100 text, (3) smartphone based recognition of human activities and postural transitions time series, and (4) EEG brainwave feeling emotions time series classification tasks using a single machine.

Appendices

A Appendix: a solution for $c^o$

According to (21) and (25):

$$\begin{aligned} \mathbf{w} _j^o(k+1)= \mathbf{w} _j^o(k) +\frac{\mathbf{e _j(k)\mathbf{g} _j'(k)\mathbf{h} (k)}{(g_j'(k))^2\Vert \mathbf{h} (k)\Vert ^2+c^o(k)}}{\quad }for \ every \ j \end{aligned}$$

(48)

it is need $c^o (k)$ for each iteration k. Therefore, the adaptive algorithm based on gradient descent is:

$$\begin{aligned} \mathbf{c} ^o (k)= \mathbf{c} ^o (k-1)+\rho ^o \frac{\partial J(k)}{\partial \mathbf{c} ^o (k-1)} \end{aligned}$$

(49)

It is show, we should calculate gradient of cost function J(k) relative to gradient of $c^o (k-1)$. According to chain rule, we have:

$$\begin{aligned} \begin{aligned}&\frac{\partial J(k)}{\partial \mathbf{c} ^o(k-1)} \\&\quad =\frac{\partial J(k)}{\partial \mathbf{e} (k)} \frac{\partial \mathbf{e} (k)}{\partial \mathbf{r} (k)} \frac{\partial \mathbf{r} (k)}{\partial \mathbf{net}2 (k)} \frac{\partial \mathbf{net}2 (k)}{\partial W^o(k)} \frac{\partial W^o(k)}{\partial \varvec{\eta }^o(k-1)} \frac{\partial \varvec{\eta }^o(k-1)}{\partial \mathbf{c} ^o(k-1)} \\&\quad =-\mathbf{e} (k)\mathbf{g} '(k)\mathbf{h} (k)\frac{\partial W^o(k)}{\partial \varvec{\eta }^o(k-1)} \frac{\partial \varvec{\eta }^o(k-1)}{\partial \mathbf{c} ^o(k-1)} \end{aligned} \end{aligned}$$

(50)

If we change variable k to k-1 in (25) then,

$$\begin{aligned} \frac{\partial \eta _{j}^o(k-1)}{\partial c_{j}^o(k-1)} =\frac{-1}{((g_j'(k-1))^2\Vert \mathbf{h} (k-1)\Vert ^2+c_j^o(k-1))^2} \end{aligned}$$

(51)

and also, in (20) then,

$$\begin{aligned} \frac{\partial W^o(k)}{\partial \eta _{j}^o(k-1)} =e_j(k-1)g_j'(k-1)\mathbf{h} (k-1) \end{aligned}$$

(52)

Finally, according to (51), (52) and (50) we have:

$$\begin{aligned} \begin{aligned} \frac{\partial J(k)}{\partial c_j^o(k-1)}= \frac{e_j(k)g_j'(k)\mathbf{h} (k) \times e_j(k-1)g_j'(k-1)\mathbf{h} (k-1)}{((g_j'(k-1))^2\Vert \mathbf{h} (k-1)\Vert ^2+c_j^o(k-1))^2} \end{aligned} \end{aligned}$$

(53)

B Appendix: a solution for $c^h$

According to (30) and (32):

$$\begin{aligned} \mathbf{w} _i^h(k+1)= \mathbf{w} _i^h(k) +\frac{e_{j}(k)g_{j}'(k)\mathbf{w} _j^o(k)\mathbf{f} '(k)x_i(k)}{(g_j'(k))^2\Vert \mathbf{f} '(k)\Vert ^2 \Vert \mathbf{w} _j^o(k)\Vert ^2\Vert \mathbf{x} (k)\Vert ^2+c^h(k)} \end{aligned}$$

(54)

it is need $c^h (k)$ for each iteration k. Therefore, the adaptive algorithm based on gradient descent is:

$$\begin{aligned} \mathbf{c} ^h (k)= \mathbf{c} ^h (k-1)+\rho ^h \frac{\partial J(k)}{\partial \mathbf{c} ^h (k-1)} \end{aligned}$$

(55)

It is show, we should calculate gradient of cost function J(k) relative to gradient of $c^h (k-1)$. According to chain rule, we have:

$$\begin{aligned} \begin{aligned}&\frac{\partial J(k)}{\partial \mathbf{c} ^h(k-1)} \frac{\partial J(k)}{\partial \mathbf{e} (k)} \frac{\partial \mathbf{e} (k)}{\partial \mathbf{r} (k)} \frac{\partial \mathbf{r} (k)}{\partial \mathbf{net}2 (k)} \\&\quad =\frac{\partial \mathbf{net}2 (k)}{\partial \mathbf{h} (k)} \frac{\partial \mathbf{h} (k)}{\partial \mathbf{net}1 (k)} \frac{\partial \mathbf{net}1 (k)}{\partial W^h(k)} \frac{\partial W^h(k)}{\partial \varvec{\eta }^h(k-1)} \frac{\partial \varvec{\eta }^h(k-1)}{\partial \mathbf{c} ^h(k-1)} \\&\quad =-\mathbf{e} (k)\mathbf{g} '(k)W^o(k)\mathbf{f} '(k)\mathbf{x} (k)\frac{\partial W^h(k)}{\partial \varvec{\eta }^h(k-1)} \frac{\partial \varvec{\eta }^h(k-1)}{\partial \mathbf{c} ^h(k-1)} \end{aligned} \end{aligned}$$

(56)

If we change variable k to k-1 in (32) then,

$$\begin{aligned} \begin{aligned}&\frac{\partial \eta _{j}^h(k-1)}{\partial c_{j}^h(k-1)} \\&\quad =\frac{-1}{((g_j'(k-1))^2\Vert \mathbf{f} '(k-1)\Vert ^2 \Vert \mathbf{w} _j^o(k-1)\Vert ^2\Vert \mathbf{x} (k-1)\Vert ^2+c_j^h(k-1))^2} \end{aligned} \end{aligned}$$

(57)

and also, in (27) then,

$$\begin{aligned} \frac{\partial W^h(k)}{\partial \eta _{j}^h(k-1)} = e_j(k-1)g_j'(k-1)\mathbf{w} _j^o(k-1)\mathbf{f} '(k-1)x_i(k-1) \end{aligned}$$

(58)

Finally, according to (57), (58) and (56) we have:

$$\begin{aligned} \begin{aligned}&\frac{\partial J(k)}{\partial c_j^o(k-1)}= e_j(k)g_j'(k)\mathbf{w} _j^o(k)\mathbf{f} '(k)x_i(k) \\&\quad \times \frac{ e_j(k-1)g_j'(k-1)\mathbf{w} _j^o(k-1)\mathbf{f} '(k-1)x_i(k-1)}{((g_j'(k-1))^2\Vert \mathbf{f} '(k-1)\Vert ^2 \Vert \mathbf{w} _j^o(k-1)\Vert ^2\Vert \mathbf{x} (k-1)\Vert ^2+c_j^h(k-1))^2} \\ \end{aligned} \end{aligned}$$

(59)