Advertisement

From Maxout to Channel-Out: Encoding Information on Sparse Pathways

  • Qi Wang
  • Joseph JaJa
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8681)

Abstract

Motivated by an important insight from neural science that “functionality is determined by pathway”, we propose a new deep network framework, called “channel-out network”, which encodes information on sparse pathways. We argue that the recent success of maxout networks can also be explained by its ability of encoding information on sparse pathways, while channel-out network does not only select pathways at training time but also at inference time. From a mathematical perspective, channel-out networks can represent a wider class of piece-wise continuous functions, thereby endowing the network with more expressive power than that of maxout networks. We test our channel-out networks on several well-known image classification benchmarks, achieving new state-of-the-art performances on CIFAR-100 and STL-10.

Keywords

deep networks pathway selection sparse pathway encoding channel-out 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Hinton, G.E., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.R.: Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 (2012)Google Scholar
  2. 2.
    Goodfellow, I.J., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. arXiv preprint arXiv:1302.4389 (2013)Google Scholar
  3. 3.
    Kandel, E.R., Schwartz, J.H., Jessell, T.M., et al.: Principles of neural science, vol. 4. McGraw-Hill, New York (2000)Google Scholar
  4. 4.
    Srivastava, N.: Improving neural networks with dropout. PhD thesis, University of Toronto (2013)Google Scholar
  5. 5.
    Wan, L., Zeiler, M., Zhang, S., Cun, Y.L., Fergus, R.: Regularization of neural networks using dropconnect. In: Proceedings of the 30th International Conference on Machine Learning (ICML 2013), pp. 1058–1066 (2013)Google Scholar
  6. 6.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25, pp. 1106–1114 (2012)Google Scholar
  7. 7.
    Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. arXiv preprint arXiv:1206.2944 (2012)Google Scholar
  8. 8.
    Zeiler, M.D., Fergus, R.: Stochastic pooling for regularization of deep convolutional neural networks. arXiv preprint arXiv:1301.3557 (2013)Google Scholar
  9. 9.
    Malinowski, M., Fritz, M.: Learning smooth pooling regions for visual recognition. In: British Machine Vision Conference (2013)Google Scholar
  10. 10.
    Coates, A., Ng, A.Y., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: International Conference on Artificial Intelligence and Statistics, pp. 215–223 (2011)Google Scholar
  11. 11.
    Bo, L., Ren, X., Fox, D.: Unsupervised feature learning for rgb-d based object recognition. ISER (June 2012)Google Scholar
  12. 12.
    Gens, R., Domingos, P.: Discriminative learning of sum-product networks. In: Advances in Neural Information Processing Systems, pp. 3248–3256 (2012)Google Scholar
  13. 13.
    Srivastava, R.K., Masci, J., Kazerounian, S., Gomez, F., Schmidhuber, J.: Compete to compute. technical report (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Qi Wang
    • 1
  • Joseph JaJa
    • 1
  1. 1.Department of Electrical and Computer Engineering, University of Maryland Institute of Advanced Computer StudiesUniversity of MarylandCollege ParkUSA

Personalised recommendations