Multiple Binary OxRAMs as Synapses for Convolutional Neural Networks

  • E. Vianello
  • D. Garbin
  • O. Bichler
  • G. Piccolboni
  • G. Molas
  • B. De Salvo
  • L. Perniola
Part of the Cognitive Systems Monographs book series (COSMOS, volume 31)


Oxide-based resistive memory (OxRAM) devices find applications in memory, logic, and neuromorphic computing systems. Among the different dielectrics proposed in OxRAM stacks, hafnium oxide, HfO\(_{2}\), attracted growing interest because of its compatibility with typical BEOL advanced CMOS processing and promising performances in terms of endurance (higher than Flash) and switching speed (few tens of ns). This chapter describes an artificial synapse composed of multiple binary HfO\(_{2}\)-based OxRAM cells connected in parallel, thereby providing synaptic analog behavior. The VRRAM technology is presented as a possible solution to gain area with respect to planar approaches by realizing one VRRAM pillar per synapse. The HfO\(_{2}\)-based OxRAM synapse has been proposed for hardware implementation of power efficient Convolutional Neural Networks for visual pattern recognition applications. Finally, the synaptic weight resolution and the robustness to device variability of the network have been investigated. Statistical evaluation of device variability is obtained on a 16 kbit OxRAM memory array integrated into advanced 28 nm CMOS technology.


Convolutional Neural Network Convolution Operation Synaptic Conductance Kernel Feature Convolutional Layer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Mead, C.: Neuromorphic electronic systems. Proc. IEEE 78, 1629–1636 (1990)CrossRefGoogle Scholar
  2. 2.
    Indiveri, G.: Shih-Chii Liu: memory and information processing in neuromorphic systems. Proc. IEEE 103, 1379–1397 (2015)CrossRefGoogle Scholar
  3. 3.
    Prezioso, M., Merrikh-Bayat, F., Hoskins, B.D., Adam, G.C., Likharev, K.K., Strukov, D.B.: Training and operation of anintegrated neuromorphic network based on metal-oxide memristors. Nature 521, 61–64 (2015)CrossRefGoogle Scholar
  4. 4.
    Burr, G.W., Shelby, R.M., di Nolfo, C., Jang, J.W., Shenoy, R.S., Narayanan, P., Virwani, K., Giacometti, E.U., Kurdi, B., Hwang, H.: Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element. In: Electron Devices Meeting (IEDM) IEEE International, pp. 29.5.1–29.5.4 (2014)Google Scholar
  5. 5.
    Kuzum, D., Yu, S., Wong, H.-S.P.: Synaptic electronics: materials, devices and applications. Nanotechnology 24, 382001 (2013)CrossRefGoogle Scholar
  6. 6.
    Suri, M., Bichler, O., Querlioz, D., Palma, G., Vianello, E., Vuillaume, D., Gamrat, C.: DeSalvo: CBRAM devices as binary synapses for lowpower stochastic neuromorphic systems: auditory (Cochlea) and visual (Retina) cognitive processing applications. In: Electron Devices Meeting (IEDM) IEEE International, pp. 10.3.1–10.3.4 (2012)Google Scholar
  7. 7.
    Kuzum, D., Jeyasingh, R.G.D., Wong,H.-S.P.: Energy efficient programming of nanoelectronic synaptic devices for large-scale implementation of associative and temporal sequence learning. In: Electron Devices Meeting (IEDM) IEEE International, pp. 30.3.1–30.3.4 (2011)Google Scholar
  8. 8.
    Suri, M., Bichler, O., Querlioz, D., Cueto, O., Perniola, L., Sousa, V., Vuillaume, D., Gamrat, C., DeSalvo, B.: Phase change memory as synapse for ultra-dense neuromorphic systems: application to complex visual pattern extraction. In: Electron Devices Meeting (IEDM) IEEE International, pp. 4.4.1–4.4.4 (2011)Google Scholar
  9. 9.
    Wang, I.-T., Lin, Y.-C., Wang, Y.-F., Hsu, C.-W., Hou, T.-H.: 3D synaptic architecture with ultralow sub-10 fJ energy per spike for neuromorphic computation. In: Electron Devices Meeting (IEDM) IEEE International, pp. 28.5.1–28.5.4 (2014)Google Scholar
  10. 10.
    Park, S., Sheri, A., Kim, J., Noh, J., Jang, J., Jeon, M., Lee, B., Lee, B.R., Lee, B.H., Hwang, H.: Neuromorphic speech systems using advanced ReRAM-based synapse. In: Electron Devices Meeting (IEDM) IEEE International, pp. 25.6.1–25.6.4 (2013)Google Scholar
  11. 11.
    Vianello, E., Thomas, O., Molas, G., Turkyilmaz, O., Jovanovic, N., Garbin, D., Palma, G., Alayan, M., Nguyen, C., Coignus, J., Giraud, B., Benoist, T., Reyboz, M., Toffoli, A., Charpin, C., Clermidy, F., Perniola, L.: Resistive memories for ultra-low-power embedded computing design. In: Electron Devices Meeting (IEDM) IEEE International, pp. 6.3.1–6.3.4 (2014)Google Scholar
  12. 12.
    Benoist, A., Blonkowski, S., Jeannot, S., Denorme, S., Damiens, J., Berger, J., Candelier, P., Vianello, E., Grampeix, H., Nodin, J.E., Jalaguier, E., Perniola, L., Allard, B.: 28nm advanced CMOS resistive RAM solution as embedded non-volatile memory. In: Proceedings of the IEEE Reliability Physics Symposium, pp. 2E.6.1–2E.6.5 (2014)Google Scholar
  13. 13.
    Govoreanu, B., Kar, G.S., Chen, Y., Paraschiv, V., Kubicek, S., Fantini, A., Radu, I.P., Goux, L., Clima, S., Degraeve, R., Jossart, N., Richard, O., Vandeweyer, T., Seo, K., Hendrickx, P., Pourtois, G., Bender, H., Altimime, L., Wouters, D.J., Kittl, J.A., Jurczak, M.: 10x10nm\(^2\) Hf/HfOx crossbar resistive RAM with excellent performance, reliability and low-energy operation. In: Electron Devices Meeting (IEDM) IEEE International, pp. 31.6.1–31.6.4 (2011)Google Scholar
  14. 14.
    Garbin, D., Suri, M., Bichler, O., Querlioz, D., Gamrat, C., DeSalvo, B.: Probabilistic neuromorphic system using binary phase-change memory (PCM) synapses: detailed power consumption analysis. In: 3th IEEE Conference on Nanotechnology (IEEE-NANO), pp. 91–94 (2013)Google Scholar
  15. 15.
    Garbin, D., Vianello, E., Bichler, O., Rafhay, Q., Gamrat, C., Ghibaudo, G., DeSalvo, B., Perniola, L.: HfO\(_2\)-based OxRAM devices as synapses for convolutional neural networks. IEEE Trans. Electron Devices 62, 2494–2501 (2015)CrossRefGoogle Scholar
  16. 16.
    Bill, J., Legenstein, R.: A compound memristive synapse model for statistical learning through STDP in spiking neural networks. Front. Neurosci. 8, 412 (2014)Google Scholar
  17. 17.
    Simard, P.Y., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: Seventh IEEE International Conference on Document Analysis and Recognition, pp. 958–963 (2003)Google Scholar
  18. 18.
    Goldberg, D.H., Cauwenberghs, G., Andreou, A.G.: Probabilistic synaptic weighting in a reconfigurable network of VLSI integrate-and-fire neurons. Neural Netw. 14, 781–793 (2001)CrossRefGoogle Scholar
  19. 19.
    Piccolboni, G., Molas, G., Portal, J.M., Coquand, R., Bocquet, M., Garbin, D., Vianello, E., Carabasse, C., Delaye, V., Pellissier, C., Magis, T., Cagli, C., Gely, M., Cueto, O., Deleruyelle, O., Ghibaudo, G., DeSalvo, B., Perniola, L.: Investigation of the potentialities of Vertical Resistive RAM (VRRAM) for neuromorphic applications. In: lectron Devices Meeting (IEDM) IEEE International, pp. 17.2.1–17.2.4 (2015)Google Scholar
  20. 20.
    Garbin, D., Bichler, O., Vianello, E., Rafhay, Q., Gamrat, C., Perniola, L., Ghibaudo, G., DeSalvo, B.: Variability-tolerant convolutional neural network for pattern recognition applications based on OxRAM synapses. In: Electron Devices Meeting (IEDM) IEEE International, pp. 28.4.1–28.4.4 (2014)Google Scholar
  21. 21.
    Kim, S., Ishii, M., Lewis, S., Perri, T., BrightSky, M., Kim, W., Jordan, R., Burr, G.W., Sosa, N., Ray, A., Han, J.-P., Miller, C., Hosokawa, K., Lam, C.: NVM neuromorphic core with 64k-cell (256-by-256) phase change memory synaptic array with On-Chip neuron circuits for continuous in-situ learning. In: Electron Devices Meeting (IEDM) IEEE International, pp. 17.1.1–17.1.4 (2015)Google Scholar
  22. 22.
    Wang, Z., Ambrogio, S., Baletti, S., Ielemini, D.: A 2-transistor/1-resistor artificial synapse capable of communication and stochastic learning in neuromorphic systems. Front. Neurosci. 8, 438 (2015)CrossRefGoogle Scholar
  23. 23.
    Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cats visual cortex. J. Physiol. 160, 106–154 (1962)CrossRefGoogle Scholar
  24. 24.
    Felleman, D.J., Van Essen, D.C.: Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1, 1–47 (1991)CrossRefGoogle Scholar
  25. 25.
    Fukushima, K.: Artificial vision by multi-layered neural networks: neocognitron and its advances. Neural Netw. 37, 103–119 (2013)CrossRefGoogle Scholar
  26. 26.
    Ciresan, D., Meier, U., Masci, J., Schmidhuber, J.: Multi-column deep neural network for traffic sign classification. Neural Netw. 32, 333–338 (2012)CrossRefGoogle Scholar
  27. 27.
    Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., Barbano, P.E.: Toward automatic phenotyping of developing embryos from videos. IEEE Trans. Image Process. 14, 1360–1371 (2005)CrossRefGoogle Scholar
  28. 28.
    Sermanet, P., Kavukcuoglu, K., Chintala, S., LeCun, Y.: Pedestrian detection with unsupervised multi-stage feature learning. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 36263633 (2013)Google Scholar
  29. 29.
    Vaillant, R., Monrocq, C., LeCun, Y.: A convolutional neural network hand tracker. IEEE Proc.-Vision, Image. Signal Process. 141, 245–250 (1994)Google Scholar
  30. 30.
    Nowlan, S.J., Platt, J.C.: Original approach for the localisation of objects in images. Adv. Neural Inf. Process. Syst. 901–908 (1995)Google Scholar
  31. 31.
    Garcia, C., Delakis, M.: Convolutional face finder: a neural architecture for fast and robust face detection. IEEE Trans. Pattern Anal. Mach. Intell. 26, 14081423 (2004)CrossRefGoogle Scholar
  32. 32.
    Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1701–1708 (2014)Google Scholar
  33. 33.
    Garbin, D., Vianello, E., Bichler, O., Azzaz, M., Rafhay, Q., Candelier, P., Gamrat, C., Ghibaudo, G., DeSalvo, B., Perniola, L.: On the impact of OxRAM-based synapses variability on convolutional neural networks performance. In: 2015 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH), pp. 193–198 (2015)Google Scholar
  34. 34.
    LeCun, Y., Cortes, C., Burges, C.J.C.: The MNIST Database of Handwritten Digits.
  35. 35.
    Stallkamp, J., Schlipsing, M., Salmen, J., Igel, C.: The German traffic sign recognition benchmark: a multi-class classification competition. In: Proceedings of the IEEE International Joint Conference on Neural Networks, pp. 1453–1460 (2011)Google Scholar
  36. 36.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86, 2278–2324 (1998)CrossRefGoogle Scholar
  37. 37.
    Ciresan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3642–3649 (2012)Google Scholar
  38. 38.
    Bichler, O., Querlioz, D., Thorpe, S. J., Bourgoin, J.-P., Gamrat, C.: Unsupervised features extraction from asynchronous silicon retina through spike-timing-dependent plasticity. In: Proceedings of the IEEE International Joint Conference on Neural Networks, pp. 859–866 (2011)Google Scholar
  39. 39.
    Hayakawa, Y., Himeno, A., Yasuhara, R., Boullart, W., Vecchio, E., Vandeweyer, T., Witters, T., Crotti, D., Jurczak, M., Fujii, S., Ito, S., Kawashima, Y., Ikeda, Y., Kawahara, A., Kawai, K., Wei, K., Muraoka, S., Shimakawa, K., Mikawa, T., Yoneda, S.: Highly reliable TaO\(_x\) ReRAM with centralized filament for 28 nm embedded application. In: 2015 Symposium on VLSI Technology Digest of Technical Papers (2015)Google Scholar
  40. 40.
    Ueki, M., Takeuchi, K., Yamamoto, T., Tanabe, A., Ikarashi, N., Saitoh, M., Nagumo, T., Sunamura, H., Narihiro, M., Uejima, K., Masuzaki, K., Furutake, N., Saito, S., Yabe, Y., Mitsuiki, A., Takeda, K., Hase, T., Hayashi, Y.: Low-Power embedded ReRAM technology for IoT applications. In: 2015 Symposium on VLSI Technology Digest of Technical Papers (2015)Google Scholar
  41. 41.
    Vianello, E., Garbin, D., Jovanovic, N., Bichler, O., Thomas, O., Salvo, B., Perniola, L.: Oxide based resistive memories for low power embedded applications and neuromorphic systems. In: 2015 ECS Transactions (2015)Google Scholar

Copyright information

© Springer (India) Pvt. Ltd. 2017

Authors and Affiliations

  • E. Vianello
    • 1
  • D. Garbin
    • 1
  • O. Bichler
    • 2
  • G. Piccolboni
    • 1
  • G. Molas
    • 1
  • B. De Salvo
    • 1
  • L. Perniola
    • 1
  1. 1.CEA LETI MINATEC CampusGrenoble Cedex 9France
  2. 2.CEA LISTGif-sur-yvetteFrance

Personalised recommendations