Advanced Topics in Deep Learning

  • Charu C. Aggarwal


This book will cover several advanced topics in deep learning, which either do not naturally fit within the focus of the previous chapters, or because their level of complexity requires separate treatment.


  1. [11]
    M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. arXiv:1701.04862, 2017.
  2. [12]
    M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv:1701.07875, 2017.
  3. [15]
    J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. arXiv: 1412.7755, 2014.
  4. [18]
    D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. Also arXiv:1409.0473, 2014.
  5. [59]
    N. Butko and J. Movellan. I-POMDP: An infomax model of eye movement. IEEE International Conference on Development and Learning, pp. 139–144, 2008.Google Scholar
  6. [60]
    Y. Cao, Y. Chen, and D. Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1), 54–66, 2015.Google Scholar
  7. [68]
    Y. Chen, T. Krishna, J. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 52(1), pp. 127–138, 2017.Google Scholar
  8. [75]
    A. Coates and A. Ng. The importance of encoding versus training with sparse coding and vector quantization. ICML Confererence, pp. 921–928, 2011.Google Scholar
  9. [83]
    M. Courbariaux, Y. Bengio, and J.-P. David. BinaryConnect: Training deep neural networks with binary weights during propagations. arXiv:1511.00363, 2015.
  10. [95]
    E. Denton, S. Chintala, and R. Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. NIPS Conference, pp. 1466–1494, 2015.Google Scholar
  11. [103]
    A. Dosovitskiy, J. Tobias Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. CVPR Conference, pp. 1538–1546, 2015.Google Scholar
  12. [109]
    V. Dumoulin and F. Visin. A guide to convolution arithmetic for deep learning. arXiv:1603.07285, 2016.
  13. [114]
    S. Essar et al. Convolutional neural networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Science of the United States of America, 113(41), pp. 11441–11446, 2016.Google Scholar
  14. [116]
    L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE TPAMI, 28(4), pp. 594–611, 2006.Google Scholar
  15. [126]
    B. Fritzke. A growing neural gas network learns topologies. NIPS Conference, pp. 625–632, 1995.Google Scholar
  16. [129]
    S. Gallant. Neural network learning and expert systems. MIT Press, 1993.Google Scholar
  17. [136]
    A. Gersho and R. M. Gray. Vector quantization and signal compression. Springer Science and Business Media, 2012.Google Scholar
  18. [145]
    I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160, 2016.
  19. [149]
    I. Goodfellow et al. Generative adversarial nets. NIPS Conference, 2014.Google Scholar
  20. [158]
    A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv:1410.5401, 2014.
  21. [159]
    A. Graves et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538.7626, pp. 471–476, 2016.Google Scholar
  22. [168]
    S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. Dally. EIE: Efficient Inference Engine for Compressed Neural Network. ACM SIGARCH Computer Architecture News, 44(3), pp. 243–254, 2016.CrossRefGoogle Scholar
  23. [169]
    S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. NIPS Conference, pp. 1135–1143, 2015.Google Scholar
  24. [213]
    F. Iandola, S. Han, M. Moskewicz, K. Ashraf, W. Dally, and K. Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size. arXiv:1602.07360, 2016.
  25. [214]
    S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.Google Scholar
  26. [215]
    P. Isola, J. Zhu, T. Zhou, and A. Efros. Image-to-image translation with conditional adversarial networks. arXiv:1611.07004, 2016.
  27. [229]
    L. Kaiser and I. Sutskever. Neural GPUs learn algorithms. arXiv:1511.08228, 2015.
  28. [248]
    T. Kohonen. The self-organizing map. Neurocomputing, 21(1), pp. 1–6, 1998.MathSciNetCrossRefGoogle Scholar
  29. [249]
    T. Kohonen. Self-organization and associative memory. Springer, 2012.Google Scholar
  30. [250]
    T. Kohonen. Self-organizing maps, Springer, 2001.Google Scholar
  31. [255]
    A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. NIPS Conference, pp. 1097–1105. 2012.Google Scholar
  32. [257]
    A. Kumar et al. Ask me anything: Dynamic memory networks for natural language processing. ICML Confererence, 2016.Google Scholar
  33. [261]
    B. Lake, T. Ullman, J. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pp. 1–101, 2016.Google Scholar
  34. [266]
    H. Larochelle and G. E. Hinton. Learning to combine foveal glimpses with a third-order Boltzmann machine. NIPS Conference, 2010.Google Scholar
  35. [289]
    W. Levy and R. Baxter. Energy efficient neural codes. Neural Computation, 8(3), pp. 531–543, 1996.CrossRefGoogle Scholar
  36. [299]
    J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. NIPS Conference, pp. 289–297, 2016.Google Scholar
  37. [302]
    M. Luong, H. Pham, and C. Manning. Effective approaches to attention-based neural machine translation. arXiv:1508.04025, 2015.
  38. [311]
    A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv:1511.05644, 2015.
  39. [317]
    T. Martinetz, S. Berkovich, and K. Schulten. ‘Neural-gas’ network for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Network, 4(4), pp. 558–569, 1993.Google Scholar
  40. [319]
    M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv:1511.054, 2015.
  41. [331]
    M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014.
  42. [338]
    V. Mnih, N. Heess, and A. Graves. Recurrent models of visual attention. NIPS Conference, pp. 2204–2212, 2014.Google Scholar
  43. [364]
    M. Palatucci, D. Pomerleau, G. Hinton, and T. Mitchell. Zero-shot learning with semantic output codes. NIPS Conference, pp. 1410–1418, 2009.Google Scholar
  44. [370]
    D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. CVPR Conference, 2016.Google Scholar
  45. [384]
    A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434, 2015.
  46. [389]
    M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525–542, 2016.Google Scholar
  47. [392]
    S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. ICML Conference, pp. 1060–1069, 2016.Google Scholar
  48. [393]
    S. Reed and N. de Freitas. Neural programmer-interpreters. arXiv:1511.06279, 2015.Google Scholar
  49. [395]
    M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. NIPS Conference, pp. 2953–2961, 2015.Google Scholar
  50. [403]
    B. Romera-Paredes and P. Torr. An embarrassingly simple approach to zero-shot learning. ICML Confererence, pp. 2152–2161, 2015.Google Scholar
  51. [410]
    D. Rumelhart, D. Zipser, and J. McClelland. Parallel Distributed Processing, MIT Press, pp. 151–193, 1986.Google Scholar
  52. [411]
    D. Rumelhart and D. Zipser. Feature discovery by competitive learning. Cognitive science, 9(1), pp. 75–112, 1985.CrossRefGoogle Scholar
  53. [413]
    A. M. Rush, S. Chopra, and J. Weston. A Neural Attention Model for Abstractive Sentence Summarization. arXiv:1509.00685, 2015.
  54. [416]
    A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One shot learning with memory-augmented neural networks. arXiv: 1605:06065, 2016.
  55. [420]
    T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. NIPS Conference, pp. 2234–2242, 2016.Google Scholar
  56. [444]
    H. Siegelmann and E. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1), pp. 132–150, 1995.MathSciNetCrossRefGoogle Scholar
  57. [462]
    Socher, Richard, Milind Ganjoo, Christopher D. Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NIPS Conference, pp. 935–943, 2013.Google Scholar
  58. [473]
    S. Sukhbaatar, J. Weston, and R. Fergus. End-to-end memory networks. NIPS Conference, pp. 2440–2448, 2015.Google Scholar
  59. [497]
    S. Thrun and L. Platt. Learning to learn. Springer, 2012.Google Scholar
  60. [507]
    O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one-shot learning. NIPS Conference, pp. 3530–3638, 2016.Google Scholar
  61. [518]
    X. Wang and A. Gupta. Generative image modeling using style and structure adversarial networks. ECCV, 2016.Google Scholar
  62. [528]
    J. Weston, S. Chopra, and A. Bordes. Memory networks. ICLR, 2015.Google Scholar
  63. [539]
    C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML Confererence, pp. 2397–2406, 2016.Google Scholar
  64. [540]
    K. Xu et al. Show, attend, and tell: Neural image caption generation with visual attention. ICML Confererence, 2015.Google Scholar
  65. [542]
    Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29, 2016.Google Scholar
  66. [543]
    X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9), pp. 1423–1447, 1999.CrossRefGoogle Scholar
  67. [546]
    L. Yu, W. Zhang, J. Wang, and Y. Yu. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. AAAI Conference, pp. 2852–2858, 2017.Google Scholar
  68. [550]
    W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv:1505.00521, 2015.Google Scholar
  69. [551]
    W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus. Learning simple algorithms from examples. ICML Confererence, pp. 421–429, 2016.Google Scholar
  70. [562]
    J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. arXiv:1609.03126, 2016.
  71. [627]
  72. [628]
  73. [629]
  74. [630]
  75. [631]
  76. [632]
  77. [633]
  78. [634]
  79. [635]
  80. [636]
  81. [637]
  82. [638]
  83. [639]
  84. [640]
  85. [641]

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Charu C. Aggarwal
    • 1
  1. 1.IBM T. J. Watson Research CenterInternational Business MachinesYorktown HeightsUSA

Personalised recommendations