Symmetric Rectified Linear Units for Fully Connected Deep Models

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11062)


Rectified Linear Units (ReLU) is one of the key aspects for the success of Deep Learning models. It has been shown that deep networks can be trained efficiently using ReLU without pre-training. In this paper, we compare and analyze various kinds of ReLU variants in fully-connected deep neural networks. We test ReLU, LReLU, ELU, SELU, mReLU and vReLU on two popular datasets: MNIST and Fashion-MNIST. We find vReLU, a symmetric ReLU variant, shows promising results in most experiments. Fully-connected networks (FCN) with vReLU activation are able to achieve a higher accuracy. It achieves relative improvement in test error rate of 39.9% compared to ReLU on MNIST dataset; and achieves relative improvement of 6.3% compared to ReLU on Fashion-MNIST dataset.


ReLU Deep models Symmetric ReLU 



This research was partially supported by NSFC under contract number 61472428 and U1711261.


  1. 1.
    Hahnloser, H.R., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Sebastian Seung, H.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789), 947–951 (2000)CrossRefGoogle Scholar
  2. 2.
    Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: AISTATS (2011)Google Scholar
  3. 3.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, vol. 60, pp. 1097–1105. Curran Associates Inc. (2012)Google Scholar
  4. 4.
    Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: ICML (2013)Google Scholar
  5. 5.
    Shah, A., Kadam, E., Shah, H., Shinde, S., Shingade, S.: Deep residual networks with exponential linear unit. In: International Symposium on Computer Vision and the Internet, pp. 59–65. ACM (2016)Google Scholar
  6. 6.
    Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S.: Self-normalizing neural networks. In: Advances in Neural Information Processing Systems (NIPS) (2017)Google Scholar
  7. 7.
    Zhao, Q., Griffin, L.D.: Suppressing the unusual: towards robust CNNs using symmetric activation functions. arXiv:1603.05145 [cs.CV] (2016)
  8. 8.
    Gens, R., Domingos, P.M.: Deep symmetry networks. In: Neural Information Processing Systems (NIPS 2014), pp. 2537–2545 (2014)Google Scholar
  9. 9.
    Nair, V., Hinton, G.: Rectified linear units improve restricted Boltzmann machines. In: ICML (1996)Google Scholar
  10. 10.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2344 (1998)CrossRefGoogle Scholar
  11. 11.
    Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv cs.LG/1708.07747 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Information and DEKE, MOERenmin University of ChinaBeijingChina

Personalised recommendations