CMOS Implementations of Rectified Linear Activation Function

  • P. PriyankaEmail author
  • G. K. Nisarga
  • S. Raghuram
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 892)


Deep Neural Networks have become an increasingly favourite choice for a variety of machine learning tasks. Two important components are largely responsible for this success, improved neural network functionalities, and availability of suitable hardware for training large complex networks. Using these types of novel networks and functions, Deep Neural Networks have been shown to be very highly efficient for various classification tasks. As the next level of optimization, dedicated ASIC and FPGA ICs are being developed, to realize Deep Neural Networks. This provides an additional level of performance optimization beyond traditional software-based implementations. Towards this direction, in this work, we have developed CMOS circuits for realizing the highly popular Rectified Linear (ReLu) activation function. The ReLu activation function has largely replaced the traditional sigmoid activation function due to better learning rates and reduced computational requirements. With dedicated CMOS implementations of such functions, we get better operating speed with lower power consumption, leading to improved real-time implementations of classification tasks.


Deep Neural Networks CMOS Rectified Linear Activation Function Neuromorphic circuits 


  1. 1.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 5, 1106–1114 (2012)Google Scholar
  2. 2.
    Chen, Y.-H., Emer, J., Sze, V.: Using dataflow to optimize energy efficiency of deep neural network accelerators. IEEE Micro 37(3), 12–21 (2017)CrossRefGoogle Scholar
  3. 3.
    Douglas, R., Mahowald, M., Mead, C.: Neuromorphic analogue VLSI. Annu. Rev. Neurosci. 18(1), 255–281 (1995)CrossRefGoogle Scholar
  4. 4.
    Vinod, N., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010)Google Scholar
  5. 5.
    He, K., et al.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision (2015)Google Scholar
  6. 6.
    Chen, Y.-H., Joel, E., Vivienne, S.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. In: 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). IEEE (2016)Google Scholar
  7. 7.
    Merolla, P.A., et al.: A million spiking-neuron integrated circuit with a scalable communication network and interface. Science 345(6197), 668–673 (2014)CrossRefGoogle Scholar
  8. 8.
    Yilmaz, M., Tunkar, B.A., Park, S., Elrayes, K., Mahmoud, M.A.E.: High-efficiency passive full wave rectification for electromagnetic harvesters. J. Appl. Phys. 116(13), 134902 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Department of ECERamaiah Institute of TechnologyBengaluruIndia

Personalised recommendations