Abstract
This book will cover several advanced topics in deep learning, which either do not naturally fit within the focus of the previous chapters, or because their level of complexity requires separate treatment.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
It turns out that by modifying the discriminator to output classes (including the fake class), one can obtain state-of-the-art semi-supervised classification with very few labels [420]. However, using the generator to output the labels is not a good choice.
Bibliography
M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks. arXiv:1701.04862, 2017. https://arxiv.org/abs/1701.04862
M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein gan. arXiv:1701.07875, 2017. https://arxiv.org/abs/1701.07875
J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. arXiv: 1412.7755, 2014. https://arxiv.org/abs/1412.7755
D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. ICLR, 2015. Also arXiv:1409.0473, 2014. https://arxiv.org/abs/1409.0473
N. Butko and J. Movellan. I-POMDP: An infomax model of eye movement. IEEE International Conference on Development and Learning, pp. 139–144, 2008.
Y. Cao, Y. Chen, and D. Khosla. Spiking deep convolutional neural networks for energy-efficient object recognition. International Journal of Computer Vision, 113(1), 54–66, 2015.
Y. Chen, T. Krishna, J. Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 52(1), pp. 127–138, 2017.
A. Coates and A. Ng. The importance of encoding versus training with sparse coding and vector quantization. ICML Confererence, pp. 921–928, 2011.
M. Courbariaux, Y. Bengio, and J.-P. David. BinaryConnect: Training deep neural networks with binary weights during propagations. arXiv:1511.00363, 2015. https://arxiv.org/pdf/1511.00363.pdf
E. Denton, S. Chintala, and R. Fergus. Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks. NIPS Conference, pp. 1466–1494, 2015.
A. Dosovitskiy, J. Tobias Springenberg, and T. Brox. Learning to generate chairs with convolutional neural networks. CVPR Conference, pp. 1538–1546, 2015.
V. Dumoulin and F. Visin. A guide to convolution arithmetic for deep learning. arXiv:1603.07285, 2016. https://arxiv.org/abs/1603.07285
S. Essar et al. Convolutional neural networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Science of the United States of America, 113(41), pp. 11441–11446, 2016.
L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE TPAMI, 28(4), pp. 594–611, 2006.
B. Fritzke. A growing neural gas network learns topologies. NIPS Conference, pp. 625–632, 1995.
S. Gallant. Neural network learning and expert systems. MIT Press, 1993.
A. Gersho and R. M. Gray. Vector quantization and signal compression. Springer Science and Business Media, 2012.
I. Goodfellow. NIPS 2016 tutorial: Generative adversarial networks. arXiv:1701.00160, 2016. https://arxiv.org/abs/1701.00160
I. Goodfellow et al. Generative adversarial nets. NIPS Conference, 2014.
A. Graves, G. Wayne, and I. Danihelka. Neural turing machines. arXiv:1410.5401, 2014. https://arxiv.org/abs/1410.5401
A. Graves et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538.7626, pp. 471–476, 2016.
S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. Horowitz, and W. Dally. EIE: Efficient Inference Engine for Compressed Neural Network. ACM SIGARCH Computer Architecture News, 44(3), pp. 243–254, 2016.
S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. NIPS Conference, pp. 1135–1143, 2015.
F. Iandola, S. Han, M. Moskewicz, K. Ashraf, W. Dally, and K. Keutzer. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 0.5 MB model size. arXiv:1602.07360, 2016. https://arxiv.org/abs/1602.07360
S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03167, 2015.
P. Isola, J. Zhu, T. Zhou, and A. Efros. Image-to-image translation with conditional adversarial networks. arXiv:1611.07004, 2016. https://arxiv.org/abs/1611.07004
L. Kaiser and I. Sutskever. Neural GPUs learn algorithms. arXiv:1511.08228, 2015. https://arxiv.org/abs/1511.08228
T. Kohonen. The self-organizing map. Neurocomputing, 21(1), pp. 1–6, 1998.
T. Kohonen. Self-organization and associative memory. Springer, 2012.
T. Kohonen. Self-organizing maps, Springer, 2001.
A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. NIPS Conference, pp. 1097–1105. 2012.
A. Kumar et al. Ask me anything: Dynamic memory networks for natural language processing. ICML Confererence, 2016.
B. Lake, T. Ullman, J. Tenenbaum, and S. Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, pp. 1–101, 2016.
H. Larochelle and G. E. Hinton. Learning to combine foveal glimpses with a third-order Boltzmann machine. NIPS Conference, 2010.
W. Levy and R. Baxter. Energy efficient neural codes. Neural Computation, 8(3), pp. 531–543, 1996.
J. Lu, J. Yang, D. Batra, and D. Parikh. Hierarchical question-image co-attention for visual question answering. NIPS Conference, pp. 289–297, 2016.
M. Luong, H. Pham, and C. Manning. Effective approaches to attention-based neural machine translation. arXiv:1508.04025, 2015. https://arxiv.org/abs/1508.04025
A. Makhzani, J. Shlens, N. Jaitly, I. Goodfellow, and B. Frey. Adversarial autoencoders. arXiv:1511.05644, 2015. https://arxiv.org/abs/1511.05644
T. Martinetz, S. Berkovich, and K. Schulten. ‘Neural-gas’ network for vector quantization and its application to time-series prediction. IEEE Transactions on Neural Network, 4(4), pp. 558–569, 1993.
M. Mathieu, C. Couprie, and Y. LeCun. Deep multi-scale video prediction beyond mean square error. arXiv:1511.054, 2015. https://arxiv.org/abs/1511.05440
M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv:1411.1784, 2014. https://arxiv.org/abs/1411.1784
V. Mnih, N. Heess, and A. Graves. Recurrent models of visual attention. NIPS Conference, pp. 2204–2212, 2014.
M. Palatucci, D. Pomerleau, G. Hinton, and T. Mitchell. Zero-shot learning with semantic output codes. NIPS Conference, pp. 1410–1418, 2009.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. CVPR Conference, 2016.
A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434, 2015. https://arxiv.org/abs/1511.06434
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. European Conference on Computer Vision, pp. 525–542, 2016.
S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. Generative adversarial text to image synthesis. ICML Conference, pp. 1060–1069, 2016.
S. Reed and N. de Freitas. Neural programmer-interpreters. arXiv:1511.06279, 2015.
M. Ren, R. Kiros, and R. Zemel. Exploring models and data for image question answering. NIPS Conference, pp. 2953–2961, 2015.
B. Romera-Paredes and P. Torr. An embarrassingly simple approach to zero-shot learning. ICML Confererence, pp. 2152–2161, 2015.
D. Rumelhart, D. Zipser, and J. McClelland. Parallel Distributed Processing, MIT Press, pp. 151–193, 1986.
D. Rumelhart and D. Zipser. Feature discovery by competitive learning. Cognitive science, 9(1), pp. 75–112, 1985.
A. M. Rush, S. Chopra, and J. Weston. A Neural Attention Model for Abstractive Sentence Summarization. arXiv:1509.00685, 2015. https://arxiv.org/abs/1509.00685
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One shot learning with memory-augmented neural networks. arXiv: 1605:06065, 2016. https://www.arxiv.org/pdf/1605.06065.pdf
T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. NIPS Conference, pp. 2234–2242, 2016.
H. Siegelmann and E. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1), pp. 132–150, 1995.
Socher, Richard, Milind Ganjoo, Christopher D. Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. NIPS Conference, pp. 935–943, 2013.
S. Sukhbaatar, J. Weston, and R. Fergus. End-to-end memory networks. NIPS Conference, pp. 2440–2448, 2015.
S. Thrun and L. Platt. Learning to learn. Springer, 2012.
O. Vinyals, C. Blundell, T. Lillicrap, and D. Wierstra. Matching networks for one-shot learning. NIPS Conference, pp. 3530–3638, 2016.
X. Wang and A. Gupta. Generative image modeling using style and structure adversarial networks. ECCV, 2016.
J. Weston, S. Chopra, and A. Bordes. Memory networks. ICLR, 2015.
C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. ICML Confererence, pp. 2397–2406, 2016.
K. Xu et al. Show, attend, and tell: Neural image caption generation with visual attention. ICML Confererence, 2015.
Z. Yang, X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29, 2016.
X. Yao. Evolving artificial neural networks. Proceedings of the IEEE, 87(9), pp. 1423–1447, 1999.
L. Yu, W. Zhang, J. Wang, and Y. Yu. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient. AAAI Conference, pp. 2852–2858, 2017.
W. Zaremba and I. Sutskever. Reinforcement learning neural turing machines. arXiv:1505.00521, 2015.
W. Zaremba, T. Mikolov, A. Joulin, and R. Fergus. Learning simple algorithms from examples. ICML Confererence, pp. 421–429, 2016.
J. Zhao, M. Mathieu, and Y. LeCun. Energy-based generative adversarial network. arXiv:1609.03126, 2016. https://arxiv.org/abs/1609.03126
https://github.com/Element-Research/rnn/blob/master/examples/
https://github.com/YerevaNN/Dynamic-memory-networks-in-Theano
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this chapter
Cite this chapter
Aggarwal, C.C. (2018). Advanced Topics in Deep Learning. In: Neural Networks and Deep Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-94463-0_10
Download citation
DOI: https://doi.org/10.1007/978-3-319-94463-0_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-94462-3
Online ISBN: 978-3-319-94463-0
eBook Packages: Computer ScienceComputer Science (R0)