A Parallel Forward-Backward Propagation Learning Scheme for Auto-Encoders

Abstract

Auto-encoders constitute one popular deep learning architecture for feature extraction. Since an auto-encoder has at least one bottle neck layer for feature representation and at least five layers for fitting nonlinear transformations, back-propagation learning (BPL) algorithms with saturated activation functions sometimes face the vanishing gradient problem, which slows convergence. Thus, several modified methods have been proposed to mitigate this problem. In this work, we propose the calculation of forward-propagated errors in parallel with back-propagated errors in the network, without modification of the activation functions or the network structure. Although this scheme for auto-encoder learning has a larger computational cost than that of BPL, processing time until convergence could be reduced by implementing parallel computing. In order to confirm the feasibility of this scheme, two simple problems were examined by training auto-encoders to acquire (1) identity mappings of two-dimensional points along the arc of a half-circle to extract the central angle and (2) hand-writing images to extract labeled digits. Both results indicate that the proposed scheme requires only about half of the iterations to reduce the cost value enough, compared to BPL.