CMOS Implementations of Rectified Linear Activation Function
Deep Neural Networks have become an increasingly favourite choice for a variety of machine learning tasks. Two important components are largely responsible for this success, improved neural network functionalities, and availability of suitable hardware for training large complex networks. Using these types of novel networks and functions, Deep Neural Networks have been shown to be very highly efficient for various classification tasks. As the next level of optimization, dedicated ASIC and FPGA ICs are being developed, to realize Deep Neural Networks. This provides an additional level of performance optimization beyond traditional software-based implementations. Towards this direction, in this work, we have developed CMOS circuits for realizing the highly popular Rectified Linear (ReLu) activation function. The ReLu activation function has largely replaced the traditional sigmoid activation function due to better learning rates and reduced computational requirements. With dedicated CMOS implementations of such functions, we get better operating speed with lower power consumption, leading to improved real-time implementations of classification tasks.
KeywordsDeep Neural Networks CMOS Rectified Linear Activation Function Neuromorphic circuits
- 1.Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 5, 1106–1114 (2012)Google Scholar
- 4.Vinod, N., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010)Google Scholar
- 5.He, K., et al.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision (2015)Google Scholar
- 6.Chen, Y.-H., Joel, E., Vivienne, S.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. In: 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA). IEEE (2016)Google Scholar