Information-Based Learning of Deep Architectures for Feature Extraction

  • Stefano Melacci
  • Marco Lippi
  • Marco Gori
  • Marco Maggini
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8157)


Feature extraction is a crucial phase in complex computer vision systems. Mainly two different approaches have been proposed so far. A quite common solution is the design of appropriate filters and features based on image processing techniques, such as the SIFT descriptors. On the other hand, machine learning techniques can be applied, relying on their capabilities to automatically develop optimal processing schemes from a significant set of training examples. Recently, deep neural networks and convolutional neural networks have been shown to yield promising results in many computer vision tasks, such as object detection and recognition. This paper introduces a new computer vision deep architecture model for the hierarchical extraction of pixel–based features, that naturally embed scale and rotation invariances. Hence, the proposed feature extraction process combines the two mentioned approaches, by merging design criteria derived from image processing tools with a learning algorithm able to extract structured feature representations from data. In particular, the learning algorithm is based on information-theoretic principles and it is able to develop invariant features from unsupervised examples. Preliminary experimental results on image classification support this new challenging research direction, when compared with other deep architectures models.


Convolutional Neural Network Rotation Invariance Computational Unit Deep Neural Network Local Entropy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bengio, Y.: Learning deep architectures for ai. Foundations and Trends in Machine Learning 2(1), 1–127 (2009)CrossRefzbMATHMathSciNetGoogle Scholar
  2. 2.
    Farabet, C., Couprie, C., Najman, L., LeCun, Y.: Scene parsing with multiscale feature learning, purity trees, and optimal covers. In: Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26-July 1. / Omni Press (2012)Google Scholar
  3. 3.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Computer Vision and Image Understanding 106(1), 59–70 (2007)CrossRefGoogle Scholar
  4. 4.
    Gori, M., Melacci, S., Lippi, M., Maggini, M.: Information theoretic learning for pixel-based visual agents. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 864–875. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  5. 5.
    Hinton, G.E., Salakhutdinov, R.R.: Reducing the Dimensionality of Data with Neural Networks. Science 313(5786), 504–507 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  6. 6.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  7. 7.
    Melacci, S., Gori, M.: Unsupervised learning by minimal entropy encoding. IEEE Transactions on Neural Networks and Learning Systems 23(12), 1849–1861 (2012)CrossRefGoogle Scholar
  8. 8.
    Riedmiller, M., Braun, H.: A direct algorithm method for faster backpropagation learning: RPROP. In: Int. Conf. on Neural Networks, pp. 586–591 (1993)Google Scholar
  9. 9.
    Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual cortex. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 994–1000 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Stefano Melacci
    • 1
  • Marco Lippi
    • 1
  • Marco Gori
    • 1
  • Marco Maggini
    • 1
  1. 1.Dipartimento di Ingegneria dell’Informazione e Scienze MatematicheUniversità degli Studi di SienaSienaItaly

Personalised recommendations