A Testbed for Cross-Dataset Analysis

  • Tatiana TommasiEmail author
  • Tinne Tuytelaars
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8927)


Despite the increasing interest towards domain adaptation and transfer learning techniques to generalize over image collections and overcome their biases, the visual community misses a large scale testbed for cross-dataset analysis. In this paper we discuss the challenges faced when aligning twelve existing image databases in a unique corpus, and we propose two cross-dataset setups that introduce new interesting research questions. Moreover, we report on a first set of experimental domain adaptation tests showing the effectiveness of iterative self-labeling for large scale problems.


Dataset Bias Domain Adaptation Iterative Self-Labeling 


  1. 1.
    Bergamo, A., Torresani, L.: Exploiting weakly-labeled web images to improve object classification: a domain adaptation approach. In: NIPS (2010)Google Scholar
  2. 2.
    Bruzzone, L., Marconcini, M.: Domain adaptation problems: A dasvm classification technique and a circular validation strategy. IEEE Trans. PAMI 32(5), 770–787 (2010)CrossRefGoogle Scholar
  3. 3.
    Chen, M., Weinberger, K.Q., Blitzer, J.: Co-training for domain adaptation. In: NIPS (2011)Google Scholar
  4. 4.
    Deng, J., Berg, A.C., Li, K., Fei-Fei, L.: What Does Classifying More Than 10,000 Image Categories Tell Us? In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part V. LNCS, vol. 6315, pp. 71–84. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  5. 5.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: CVPR (2009)Google Scholar
  6. 6.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531 (2013)
  7. 7.
    Everingham, M., Gool, L.V., Williams, C.K., Winn, J., Zisserman, A.: The Pascal Visual Object Classes (VOC) Challenge. IJCV 88(2) (2010)Google Scholar
  8. 8.
    Fang, C., Xu, Y., Rockmore, D.N.: Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In: ICCV (2013)Google Scholar
  9. 9.
    Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: CVPR (2009)Google Scholar
  10. 10.
    Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. Comput. Vis. Image Underst. 106(1), 59–70 (2007)CrossRefGoogle Scholar
  11. 11.
    Fernando, B., Habrard, A., Sebban, M., Tuytelaars, T.: Unsupervised visual domain adaptation using subspace alignment. In: ICCV (2013)Google Scholar
  12. 12.
    Gong, B., Sha, F., Grauman, K.: Overcoming dataset bias: An unsupervised domain adaptation approach. In: NIPS Workshop on Large Scale Visual Recognition and Retrieval (2012)Google Scholar
  13. 13.
    Gong, B., Shi, Y., Sha, F., Grauman, K.: Geodesic flow kernel for unsupervised domain adaptation. In: CVPR (2012)Google Scholar
  14. 14.
    Gong, B., Grauman, K., Sha, F.: Connecting the dots with landmarks: Discriminatively learning domain-invariant features for unsupervised domain adaptation. In: ICML (2013)Google Scholar
  15. 15.
    Gong, B., Grauman, K., Sha, F.: Reshaping visual datasets for domain adaptation. In: NIPS (2013)Google Scholar
  16. 16.
    Griffin, G., Holub, A., Perona, P.: Caltech 256 object category dataset. Tech. Rep. UCB/CSD-04-1366, California Institue of Technology (2007)Google Scholar
  17. 17.
    Hand, D.J.: Classifier Technology and the Illusion of Progress. Stat. Sci. 21, 1–15 (2006)CrossRefzbMATHMathSciNetGoogle Scholar
  18. 18.
    Hand, D.J.: Academic obsessions and classification realities: ignoring practicalities in supervised classification. In: Classification, Clustering, and Data Mining Applications, pp. 209–232 (2004)Google Scholar
  19. 19.
    Hoffman, J., Kulis, B., Darrell, T., Saenko, K.: Discovering Latent Domains for Multisource Domain Adaptation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part II. LNCS, vol. 7573, pp. 702–715. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  20. 20.
    Khosla, A., Zhou, T., Malisiewicz, T., Efros, A.A., Torralba, A.: Undoing the Damage of Dataset Bias. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part I. LNCS, vol. 7572, pp. 158–171. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  21. 21.
    Lai, K., Bo, L., Ren, X., Fox, D.: A large-scale hierarchical multi-view rgb-d object dataset. In: ICRA (2011)Google Scholar
  22. 22.
    Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between class attribute transfer. In: CVPR (2009)Google Scholar
  23. 23.
    Leibe, B., Schiele, B.: Analyzing appearance and contour based methods for object categorization. In: CVPR (2003)Google Scholar
  24. 24.
    Microsoft: Microsoft Research Cambridge Object Recognition Image Database. (2005)
  25. 25.
    Ordonez, V., Deng, J., Choi, Y., Berg, A.C., Berg, T.L.: From large scale image categorization to entry-level categories. In: ICCV (2013)Google Scholar
  26. 26.
    Patricia, N., Caputo, B.: Learning to learn, from transfer learning to domain adaptation: A unifying perspective. In: CVPR (2014)Google Scholar
  27. 27.
    Qiu, Q., Patel, V.M., Turaga, P., Chellappa, R.: Domain Adaptive Dictionary Learning. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part IV. LNCS, vol. 7575, pp. 631–645. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  28. 28.
    Rodner, E., Hoffman, J., Donahue, J., Darrell, T., Saenko, K.: Towards adapting imagenet to reality: Scalable domain adaptation with implicit low-rank transformations. CoRR abs/1308.4200 (2013)Google Scholar
  29. 29.
    Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting Visual Category Models to New Domains. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part IV. LNCS, vol. 6314, pp. 213–226. Springer, Heidelberg (2010) CrossRefGoogle Scholar
  30. 30.
    Sanchez, J., Perronnin, F.: High-dimensional signature compression for large-scale image classification. In: CVPR (2011)Google Scholar
  31. 31.
    Tommasi, T., Caputo, B.: Frustratingly easy nbnn domain adaptation. In: ICCV (2013)Google Scholar
  32. 32.
    Tommasi, T., Quadrianto, N., Caputo, B., Lampert, C.H.: Beyond Dataset Bias: Multi-task Unaligned Shared Knowledge Transfer. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part I. LNCS, vol. 7724, pp. 1–15. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  33. 33.
    Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR (2011)Google Scholar
  34. 34.
    Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: CVPR (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.ESAT-PSI/VISICS - iMindsKU LeuvenBelgium

Personalised recommendations