Skip to main content
Book cover

Neural Networks and Deep Learning pp 419–458Cite as

Advanced Topics in Deep Learning

  • 418k Accesses

Abstract

This book will cover several advanced topics in deep learning, which either do not naturally fit within the focus of the previous chapters, or because their level of complexity requires separate treatment.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-94463-0_10
  • Chapter length: 40 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-94463-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Hardcover Book
USD   69.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Figure 10.1
Figure 10.2
Figure 10.3
Figure 10.4
Figure 10.5
Figure 10.6
Figure 10.7
Figure 10.8
Figure 10.9
Figure 10.10
Figure 10.11
Figure 10.12

Notes

  1. 1.

    The examples in Chapter 3 are given in a different context. Nevertheless, if we pretend that the loss function in Figure 3.17(b) represents J D, then the annotated saddle point in the figure is visually instructive.

  2. 2.

    It turns out that by modifying the discriminator to output classes (including the fake class), one can obtain state-of-the-art semi-supervised classification with very few labels [420]. However, using the generator to output the labels is not a good choice.

Bibliography

  1. M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. arXiv:1701.04862, 2017. https://arxiv.org/abs/1701.04862

  2. M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv:1701.07875, 2017. https://arxiv.org/abs/1701.07875

  3. J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. arXiv: 1412.7755, 2014. https://arxiv.org/abs/1412.7755

  4. D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. Also arXiv:1409.0473, 2014. https://arxiv.org/abs/1409.0473

  5. N. Butko and J. Movellan. I-POMDP: An infomax model of eye movement. IEEE International Conference on Development and Learning, pp. 139–144, 2008.

    Google Scholar 

  6. Y. Cao, Y. Chen, and D. Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1), 54–66, 2015.

    Google Scholar 

  7. Y. Chen, T. Krishna, J. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 52(1), pp. 127–138, 2017.

    Google Scholar 

  8. A. Coates and A. Ng. The importance of encoding versus training with sparse coding and vector quantization. ICML Confererence, pp. 921–928, 2011.

    Google Scholar 

  9. M. Courbariaux, Y. Bengio, and J.-P. David. BinaryConnect: Training deep neural networks with binary weights during propagations. arXiv:1511.00363, 2015. https://arxiv.org/pdf/1511.00363.pdf

  10. E. Denton, S. Chintala, and R. Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. NIPS Conference, pp. 1466–1494, 2015.

    Google Scholar 

  11. A. Dosovitskiy, J. Tobias Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. CVPR Conference, pp. 1538–1546, 2015.

    Google Scholar 

  12. V. Dumoulin and F. Visin. A guide to convolution arithmetic for deep learning. arXiv:1603.07285, 2016. https://arxiv.org/abs/1603.07285

  13. S. Essar et al. Convolutional neural networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Science of the United States of America, 113(41), pp. 11441–11446, 2016.

    Google Scholar 

  14. L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE TPAMI, 28(4), pp. 594–611, 2006.

    Google Scholar 

  15. B. Fritzke. A growing neural gas network learns topologies. NIPS Conference, pp. 625–632, 1995.

    Google Scholar 

  16. S. Gallant. Neural network learning and expert systems. MIT Press, 1993.

    Google Scholar 

  17. A. Gersho and R. M. Gray. Vector quantization and signal compression. Springer Science and Business Media, 2012.

    Google Scholar 

  18. I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160, 2016. https://arxiv.org/abs/1701.00160

  19. I. Goodfellow et al. Generative adversarial nets. NIPS Conference, 2014.

    Google Scholar 

  20. A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv:1410.5401, 2014. https://arxiv.org/abs/1410.5401

  21. A. Graves et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538.7626, pp. 471–476, 2016.

    Google Scholar 

  22. S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. Dally. EIE: Efficient Inference Engine for Compressed Neural Network. ACM SIGARCH Computer Architecture News, 44(3), pp. 243–254, 2016.

    CrossRef  Google Scholar 

  23. S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. NIPS Conference, pp. 1135–1143, 2015.

    Google Scholar 

  24. F. Iandola, S. Han, M. Moskewicz, K. Ashraf, W. Dally, and K. Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size. arXiv:1602.07360, 2016. https://arxiv.org/abs/1602.07360

  25. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.

    Google Scholar 

  26. P. Isola, J. Zhu, T. Zhou, and A. Efros. Image-to-image translation with conditional adversarial networks. arXiv:1611.07004, 2016. https://arxiv.org/abs/1611.07004

  27. L. Kaiser and I. Sutskever. Neural GPUs learn algorithms. arXiv:1511.08228, 2015. https://arxiv.org/abs/1511.08228

  28. T. Kohonen. The self-organizing map. Neurocomputing, 21(1), pp. 1–6, 1998.

    MathSciNet  CrossRef  Google Scholar 

  29. T. Kohonen. Self-organization and associative memory. Springer, 2012.

    Google Scholar 

  30. T. Kohonen. Self-organizing maps, Springer, 2001.

    Google Scholar 

  31. A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. NIPS Conference, pp. 1097–1105. 2012.

    Google Scholar 

  32. A. Kumar et al. Ask me anything: Dynamic memory networks for natural language processing. ICML Confererence, 2016.

    Google Scholar 

  33. B. Lake, T. Ullman, J. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pp. 1–101, 2016.

    Google Scholar 

  34. H. Larochelle and G. E. Hinton. Learning to combine foveal glimpses with a third-order Boltzmann machine. NIPS Conference, 2010.

    Google Scholar 

  35. W. Levy and R. Baxter. Energy efficient neural codes. Neural Computation, 8(3), pp. 531–543, 1996.

    CrossRef  Google Scholar 

  36. J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. NIPS Conference, pp. 289–297, 2016.

    Google Scholar 

  37. M. Luong, H. Pham, and C. Manning. Effective approaches to attention-based neural machine translation. arXiv:1508.04025, 2015. https://arxiv.org/abs/1508.04025

  38. A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv:1511.05644, 2015. https://arxiv.org/abs/1511.05644

  39. T. Martinetz, S. Berkovich, and K. Schulten. ‘Neural-gas’ network for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Network, 4(4), pp. 558–569, 1993.

    Google Scholar 

  40. M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv:1511.054, 2015. https://arxiv.org/abs/1511.05440

  41. M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014. https://arxiv.org/abs/1411.1784

  42. V. Mnih, N. Heess, and A. Graves. Recurrent models of visual attention. NIPS Conference, pp. 2204–2212, 2014.

    Google Scholar 

  43. M. Palatucci, D. Pomerleau, G. Hinton, and T. Mitchell. Zero-shot learning with semantic output codes. NIPS Conference, pp. 1410–1418, 2009.

    Google Scholar 

  44. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. CVPR Conference, 2016.

    Google Scholar 

  45. A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434, 2015. https://arxiv.org/abs/1511.06434

  46. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525–542, 2016.

    Google Scholar 

  47. S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. ICML Conference, pp. 1060–1069, 2016.

    Google Scholar 

  48. S. Reed and N. de Freitas. Neural programmer-interpreters. arXiv:1511.06279, 2015.

    Google Scholar 

  49. M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. NIPS Conference, pp. 2953–2961, 2015.

    Google Scholar 

  50. B. Romera-Paredes and P. Torr. An embarrassingly simple approach to zero-shot learning. ICML Confererence, pp. 2152–2161, 2015.

    Google Scholar 

  51. D. Rumelhart, D. Zipser, and J. McClelland. Parallel Distributed Processing, MIT Press, pp. 151–193, 1986.

    Google Scholar 

  52. D. Rumelhart and D. Zipser. Feature discovery by competitive learning. Cognitive science, 9(1), pp. 75–112, 1985.

    CrossRef  Google Scholar 

  53. A. M. Rush, S. Chopra, and J. Weston. A Neural Attention Model for Abstractive Sentence Summarization. arXiv:1509.00685, 2015. https://arxiv.org/abs/1509.00685

  54. A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One shot learning with memory-augmented neural networks. arXiv: 1605:06065, 2016. https://www.arxiv.org/pdf/1605.06065.pdf

  55. T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. NIPS Conference, pp. 2234–2242, 2016.

    Google Scholar 

  56. H. Siegelmann and E. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1), pp. 132–150, 1995.

    MathSciNet  CrossRef  Google Scholar 

  57. Socher, Richard, Milind Ganjoo, Christopher D. Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NIPS Conference, pp. 935–943, 2013.

    Google Scholar 

  58. S. Sukhbaatar, J. Weston, and R. Fergus. End-to-end memory networks. NIPS Conference, pp. 2440–2448, 2015.

    Google Scholar 

  59. S. Thrun and L. Platt. Learning to learn. Springer, 2012.

    Google Scholar 

  60. O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one-shot learning. NIPS Conference, pp. 3530–3638, 2016.

    Google Scholar 

  61. X. Wang and A. Gupta. Generative image modeling using style and structure adversarial networks. ECCV, 2016.

    Google Scholar 

  62. J. Weston, S. Chopra, and A. Bordes. Memory networks. ICLR, 2015.

    Google Scholar 

  63. C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML Confererence, pp. 2397–2406, 2016.

    Google Scholar 

  64. K. Xu et al. Show, attend, and tell: Neural image caption generation with visual attention. ICML Confererence, 2015.

    Google Scholar 

  65. Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29, 2016.

    Google Scholar 

  66. X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9), pp. 1423–1447, 1999.

    CrossRef  Google Scholar 

  67. L. Yu, W. Zhang, J. Wang, and Y. Yu. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. AAAI Conference, pp. 2852–2858, 2017.

    Google Scholar 

  68. W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv:1505.00521, 2015.

    Google Scholar 

  69. W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus. Learning simple algorithms from examples. ICML Confererence, pp. 421–429, 2016.

    Google Scholar 

  70. J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. arXiv:1609.03126, 2016. https://arxiv.org/abs/1609.03126

  71. https://github.com/Element-Research/rnn/blob/master/examples/

  72. https://github.com/lmthang/nmt.matlab

  73. https://github.com/carpedm20/NTM-tensorflow

  74. https://github.com/camigord/Neural-Turing-Machine

  75. https://github.com/SigmaQuan/NTM-Keras

  76. https://github.com/snipsco/ntm-lasagne

  77. https://github.com/kaishengtai/torch-ntm

  78. https://github.com/facebook/MemNN

  79. https://github.com/carpedm20/MemN2N-tensorflow

  80. https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano

  81. https://github.com/carpedm20/DCGAN-tensorflow

  82. https://github.com/carpedm20

  83. https://github.com/jacobgil/keras-dcgan

  84. https://github.com/wiseodd/generative-models

  85. https://github.com/paarthneekhara/text-to-image

Download references

Author information

Authors and Affiliations

  1. IBM T. J. Watson Research Center, International Business Machines, Yorktown Heights, NY, USA

    Charu C. Aggarwal

Authors
  1. Charu C. Aggarwal
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Aggarwal, C.C. (2018). Advanced Topics in Deep Learning. In: Neural Networks and Deep Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-94463-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-94463-0_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-94462-3

  • Online ISBN: 978-3-319-94463-0

  • eBook Packages: Computer ScienceComputer Science (R0)