Advertisement

In the Era of Deep Convolutional Features: Are Attributes Still Useful Privileged Data?

  • Viktoriia Sharmanska
  • Novi Quadrianto
Chapter
Part of the Advances in Computer Vision and Pattern Recognition book series (ACVPR)

Abstract

Our answer is, if used for challenging computer vision tasks, attributes are useful privileged data. We introduce a learning framework called learning using privileged information (LUPI) to the computer vision field to solve the object recognition task in images. We want computers to be able to learn more efficiently at the expense of providing extra information during training time. In this chapter, we focus on semantic attributes as a source of additional information about image data. This information is privileged to image data as it is not available at test time. Recently, image features from deep convolutional neural networks (CNNs) have become primary candidates for many visual recognition tasks. We will therefore analyze the usefulness of attributes as privileged information in the context of deep CNN features as image representation. We explore two maximum-margin LUPI techniques and provide a kernelized version of them to handle nonlinear binary classification problems. We interpret LUPI methods as learning to identify easy and hard objects in the privileged space and transferring this knowledge to train a better classifier in the original data space. We provide a thorough analysis and comparison of information transfer from privileged to the original data spaces for two maximum-margin LUPI methods and a recently proposed probabilistic LUPI method based on Gaussian processes. Our experiments show that in a typical recognition task such as deciding whether an object is “present” or “not present” in an image, attributes do not lead to improvement in the prediction performance when used as privileged information. In an ambiguous vision task such as determining how “easy” or “difficult” it is to spot an object in an image, we show that attribute representation is useful privileged information for deep CNN image features.

Keywords

Support Vector Machine Giant Panda Object Recognition Task Standard Support Vector Machine Original Feature Space 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

We would like to thank Christoph Lampert, Kristian Kersting, and Daniel Hernández-Lobato for discussions and collaborative work on the LUPI framework. We also thank Rogerio Feris for his editorial comments on the manuscript.

References

  1. 1.
    Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embeddings for fine-grained image classification. In: Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  2. 2.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (SURF). Comput. Vision Image Underst. (CVIU) 110(3), 346–359 (2008)CrossRefGoogle Scholar
  3. 3.
    Blaschko, M.B., Lampert, C.H.: Correlational spectral clustering. In: Computer Vision and Pattern Recognition (CVPR) (2008)Google Scholar
  4. 4.
    Chapelle, O.: Training a support vector machine in the primal. Neural Comput. 19(5), 1155–1178 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Chen, C.Y., Grauman, K.: Clues from the beaten path: location estimation with bursty sequences of tourist photos. In: Computer Vision and Pattern Recognition (CVPR) (2011)Google Scholar
  6. 6.
    Chen, K., Kämäräinen, J.K.: Learning to count with back-propagated information. In: International Conference on Pattern Recognition (ICPR) (2014)Google Scholar
  7. 7.
    Chen, J., Liu, X., Lyu, S.: Boosting with side information. In: Asian Conference on Computer Vision (ACCV) (2013)Google Scholar
  8. 8.
    Christoudias, M., Urtasun, R., Darrell, T.: Multi-view learning in the presence of view disagreement. In: Uncertainty in Artificial Intelligence (UAI) (2008)Google Scholar
  9. 9.
    Dance, C., Willamowski, J., Fan, L., Bray, C., Csurka, G.: Visual categorization with bags of keypoints. In: European Conference on Computer Vision (ECCV) (2004)Google Scholar
  10. 10.
    Donahue, J., Grauman, K.: Annotator rationales for visual recognition. In: International Conference on Computer Vision (ICCV) (2011)Google Scholar
  11. 11.
    Everingham, M., Gool, L.J.V., Williams, C.K.I., Winn, J.M., Zisserman, A.: The Pascal VOC challenge. Int. J. Comput. Vision (IJCV) 88, 303–338 (2010)CrossRefGoogle Scholar
  12. 12.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: LIBLINEAR: a library for large linear classification. J. Mach. Lear. Res. (JMLR) 9, 1871–1874 (2008)zbMATHGoogle Scholar
  13. 13.
    Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.A.: Describing objects by their attributes. In: Computer Vision and Pattern Recognition (CVPR) (2009)Google Scholar
  14. 14.
    Feyereisl, J., Aickelin, U.: Privileged information for data clustering. Inf. Sci. 194, 4–23 (2012)CrossRefGoogle Scholar
  15. 15.
    Feyereisl, J., Kwak, S., Son, J., Han, B.: Object localization based on structural SVM using privileged information. In: Conference on Neural Information Processing Systems (NIPS) (2014)Google Scholar
  16. 16.
    Fouad, S., Tino, P., Raychaudhury, S., Schneider, P.: Incorporating privileged information through metric learning. IEEE Trans. Neural Netw. Learn. Syst. 24(7), 1086–1098 (2013)CrossRefGoogle Scholar
  17. 17.
    Gehler, P., Nowozin, S.: On feature combination for multiclass object classification. In: International Conference on Computer Vision (ICCV) (2009)Google Scholar
  18. 18.
    Hernández-Lobato, D., Sharmanska, V., Kersting, K., Lampert, C.H., Quadrianto, N.: Mind the nuisance: Gaussian process classification using privileged noise. In: Conference on Neural Information Processing Systems (NIPS) (2014)Google Scholar
  19. 19.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: ACM Multimedia (ACM MM) (2014)Google Scholar
  20. 20.
    Kalal, Z., Matas, J., Mikolajczyk, K.: Online learning of robust object detectors during unstable tracking. In: ICCV Workshop On-line learning for Computer Vision (2009)Google Scholar
  21. 21.
    Khamis, S., Lampert, C.H.: CoConut: co-classification with output space regularization. In: British Machine Vision Conference (BMVC) (2014)Google Scholar
  22. 22.
    Kovashka, A., Parikh, D., Grauman, K.: Whittlesearch: interactive image search with relative attribute feedback. Int. J. Comput. Vision (IJCV) 115(2), 185–210 (2015)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Conference on Neural Information Processing Systems (NIPS) (2012)Google Scholar
  24. 24.
    Kuettel, D., Guillaumin, M., Ferrari, V.: Segmentation propagation in ImageNet. In: European Conference on Computer Vision (ECCV) (2012)Google Scholar
  25. 25.
    Kulkarni, G., Premraj, V., Ordonez, V., Dhar, S., Li, S., Choi, Y., Berg, A.C., Berg, T.L.: Babytalk: understanding and generating simple image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 35(12), 2891–2903 (2013)Google Scholar
  26. 26.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Attribute-based classification for zero-shot visual object categorization. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 36(3), 453–465 (2013)Google Scholar
  27. 27.
    Lapin, M., Hein, M., Schiele, B.: Learning using privileged information: SVM+ and weighted SVM. Neural Netw. 53, 95–108 (2014)CrossRefzbMATHGoogle Scholar
  28. 28.
    Li, W., Niu, L., Xu, D.: Exploiting privileged information from web data for image categorization. In: European Conference on Computer Vision (ECCV) (2014)Google Scholar
  29. 29.
    Lopez-Paz, D., Bottou, L., Schölkopf, B., Vapnik, V.: Unifying distillation and privileged information. In: International Conference on Learning Representations (ICLR) (2016)Google Scholar
  30. 30.
    Maron, O., Ratan, A.L.: Multiple-instance learning for natural scene classification. In: International Conference on Machine Learning (ICML) (1998)Google Scholar
  31. 31.
    Niu, L., Li, W., Xu, D.: Exploiting privileged information from web data for action and event recognition. Int. J. Comput. Vision (IJCV) 118(2), 130–150 (2016)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Ouyang, W., Li, H., Zeng, X., Wang, X.: Learning deep representation with large-scale attributes. In: International Conference on Computer Vision (ICCV) (2015)Google Scholar
  33. 33.
    Pechyony, D., Vapnik, V.: On the theory of learning with privileged information. In: Conference on Neural Information Processing Systems (NIPS) (2010)Google Scholar
  34. 34.
    Pechyony, D., Vapnik, V.: Fast optimization algorithms for solving SVM+. In: Statistical Learning and Data Science (2011)Google Scholar
  35. 35.
    Quadrianto, N., Lampert, C.H.: Learning multi-view neighborhood preserving projections. In: International Conference on Machine Learning (ICML) (2011)Google Scholar
  36. 36.
    Rasmussen, C.E., Williams, C.K.: Gaussian Processes for Machine Learning. The MIT Press (2006)Google Scholar
  37. 37.
    Russakovsky, O., Lin, Y., Yu, K., Fei-Fei, L.: Object-centric spatial pooling for image classification. In: European Conference on Computer Vision (ECCV) (2012)Google Scholar
  38. 38.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision (IJCV) 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Schölkopf, B., Smola, A.J.: Learning with Kernels. The MIT Press (2002)Google Scholar
  40. 40.
    Shalev-Shwartz, S., Singer, Y., Srebro, N.: Pegasos: Primal estimated sub-GrAdient solver for SVM. In: International Conference on Machine Learning (ICML) (2007)Google Scholar
  41. 41.
    Sharmanska, V., Quadrianto, N.: Learning from the mistakes of others: matching errors in cross dataset learning. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  42. 42.
    Sharmanska, V., Quadrianto, N., Lampert, C.H.: Learning to rank using privileged information. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar
  43. 43.
    Sharmanska, V., Hernández-Lobato, D., Hernández-Lobato, J.M., Quadrianto, N.: Ambiguity helps: classification with disagreements in crowdsourced annotations. In: Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  44. 44.
    Sharmanska, V., Quadrianto, N., Lampert, C.H.: Learning to transfer privileged information. arXiv:1410.0389 (2014)
  45. 45.
    Snoek, C.G., Worring, M., Smeulders, A.W.: Early versus late fusion in semantic video analysis. In: ACM Multimedia (ACM MM) (2005)Google Scholar
  46. 46.
    Tommasi, T., Quadrianto, N., Caputo, B., Lampert, C.: Beyond dataset bias: multi-task unaligned shared knowledge transfer. In: Asian Conference on Computer Vision (ACCV) (2012)Google Scholar
  47. 47.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer (1999)Google Scholar
  48. 48.
    Vapnik, V., Izmailov, R.: Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res. (JMLR) 16, 2023–2049 (2015)MathSciNetzbMATHGoogle Scholar
  49. 49.
    Vapnik, V., Vashist, A.: A new learning paradigm: learning using privileged information. Neural Netw. 22(5–6), 544–557 (2009)CrossRefzbMATHGoogle Scholar
  50. 50.
    Vedaldi, A., Gulshan, V., Varma, M., Zisserman, A.: Multiple kernels for object detection. In: International Conference on Computer Vision (ICCV) (2009)Google Scholar
  51. 51.
    Wang, Y., Mori, G.: A discriminative latent model of object classes and attributes. In: European Conference on Computer Vision (ECCV) (2010)Google Scholar
  52. 52.
    Yang, H., Patras, I.: Privileged information-based conditional regression forest for facial feature detection. In: International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013)Google Scholar
  53. 53.
    Zhang, Z., Luo, P., Loy, C.C., Tang, X.: Learning deep representation for face alignment with auxiliary attributes. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 38(5), 918–930 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.SMiLe CLiNiCUniversity of SussexBrightonUK

Personalised recommendations