Fusing Color and Shape for Bag-of-Words Based Object Recognition

  • Joost van de Weijer
  • Fahad Shahbaz Khan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7786)


In this article we provide an analysis of existing methods for the incorporation of color in bag-of-words based image representations. We propose a list of desired properties on which bases fusing methods can be compared. We discuss existing methods and indicate shortcomings of the two well-known fusing methods, namely early and late fusion. Several recent works have addressed these shortcomings by exploiting top-down information in the bag-of-words pipeline: color attention which is motivated from human vision, and Portmanteau vocabularies which are based on information theoretic compression of product vocabularies. We point out several remaining challenges in cue fusion and provide directions for future research.


object recognition color features bag-of-words image classification 


  1. 1.
    Fergus, R., Perona, P., Zisserman, A.: Object class recognition by unsupervised scale-invariant learning. In: IEEE Conference on Computer Vision and Patter Recognition, vol. 2, pp. 264–271 (June 2003)Google Scholar
  2. 2.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE Trans. on Pattern Analysis and Machine Intelligence 27(10), 1615–1630 (2005)CrossRefGoogle Scholar
  3. 3.
    Vedaldi, A., Gulshan, V., Varma, M., Zisserman, A.: Multiple kernels for object detection. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 606–613. IEEE (2009)Google Scholar
  4. 4.
    Nister, D., Stewenius, H.: Scalable recognition with a vocabulary tree. In: Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2006, vol. 2, pp. 2161–2168. IEEE Computer Society (2006)Google Scholar
  5. 5.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision (IJCV) 60(2), 91–110 (2004)CrossRefGoogle Scholar
  6. 6.
    van de Weijer, J., Schmid, C.: Coloring local feature extraction. In: Proc. of the European Conference on Computer Vision, Graz, Austria, vol. 2, pp. 334–348 (2006)Google Scholar
  7. 7.
    Bosch, A., Zisserman, A., Muñoz, X.: Scene Classification Via pLSA. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 517–530. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  8. 8.
    van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluating color descriptors for object and scene recognition. PAMI 32(9), 1582–1596 (2010)CrossRefGoogle Scholar
  9. 9.
    Bach, F.: Exploring large feature spaces with hierarchical multiple kernel learning. In: NIPS (2008)Google Scholar
  10. 10.
    Gehler, P.V., Nowozin, S.: On feature combination for multiclass object classification. In: Proc. International Conference on Computer Vision (2009)Google Scholar
  11. 11.
    Fernando, B., Fromont, E., Muselet, D., Sebban, M.: Discriminative feature fusion for image classification. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3434–3441. IEEE (2012)Google Scholar
  12. 12.
    Burghouts, G., Geusebroek, J.: Performance evaluation of local colour invariants. Computer Vision and Image Understanding 113(1), 48–62 (2009)CrossRefGoogle Scholar
  13. 13.
    Khan, F., Van de Weijer, J., Bagdanov, A., Vanrell, M.: Portmanteau vocabularies for multi-cue image representation. In: Twenty-Fifth Annual Conference on Neural Information Processing Systems (NIPS 2011) (2011)Google Scholar
  14. 14.
    Khan, F.S., van de Weijer, J., Vanrell, M.: Modulating shape features by color attention for object recognition. International Journal of Computer Vision (IJCV) 98(1), 49–64 (2012)CrossRefGoogle Scholar
  15. 15.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: IEEE Conference on Computer Vision and Patter Recognition, pp. 2169–2178 (2006)Google Scholar
  16. 16.
    van de Weijer, J., Schmid, C.: Applying color names to image description. In: IEEE International Conference on Image Processing (ICIP), San Antonio, USA (2007)Google Scholar
  17. 17.
    van de Weijer, J., Schmid, C., Verbeek, J., Larlus, D.: Learning color names for real-world applications. IEEE Transactions on Image Processing 18(7), 1512–1524 (2009)MathSciNetCrossRefGoogle Scholar
  18. 18.
    Zhang, J., Barhomi, Y., Serre, T.: A New Biologically Inspired Color Image Descriptor. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 312–324. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  19. 19.
    Rojas-Vigo, D., Khan, F.S., van de Weijer, J., Gevers, T.: The impact of color on bag-of-words based object recognition. In: Int. Conference on Pattern Recognition, ICPR (2010)Google Scholar
  20. 20.
    Treisman, A.: The binding problem. Current Opinion in Neurobiology 6, 171–178 (1996)CrossRefGoogle Scholar
  21. 21.
    Treisman, A., Gelade, G.: A feature integration theory of attention. Cogn. Psych. 12, 97–136 (1980)CrossRefGoogle Scholar
  22. 22.
    Wolfe, J.M.: Visual Search. In: Pashler, H. (ed.) Attention, Psychology Press Ltd. (1998)Google Scholar
  23. 23.
    Wolfe, J.M., Horowitz, T.: What attributes guide the deployment of visual attention and how do they do it? Nature Reviews Neuroscience 5, 1–7 (2004)CrossRefGoogle Scholar
  24. 24.
    Dhillon, I., Mallela, S., Kumar, R.: A divisive information-theoretic feature clustering algorithm for text classification. Journal of Machine Learning Research (JMLR) 3, 1265–1287 (2003)zbMATHMathSciNetGoogle Scholar
  25. 25.
    Li, L., Yuan, C., Hu, W., Li, B.: Top-Down Cues for Event Recognition. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part III. LNCS, vol. 6494, pp. 691–702. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  26. 26.
    Elfiky, N., Khan, F.S., van de Weijer, J., Gonzalez, J.: Discriminative compact pyramids for object and scene recognition. Pattern Recognition (PR) 45(4), 1627–1636 (2012)zbMATHCrossRefGoogle Scholar
  27. 27.
    Khan, F., Anwer, R., van de Weijer, J., Bagdanov, A., Vanrell, M., Lopez, A.: Color attributes for object detection. In: IEEE Conference on Computer Vision and Patter Recognition (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Joost van de Weijer
    • 1
  • Fahad Shahbaz Khan
    • 2
  1. 1.Computer Vision Center BarcelonaEdifici O, Campus UABBellaterraSpain
  2. 2.Computer Vision LaboratoryLinköping UniversitySweden

Personalised recommendations