Multimedia Tools and Applications

, Volume 78, Issue 2, pp 1635–1648 | Cite as

Fused feature encoding in convolutional neural network

  • Lu HuoEmail author
  • Tianrong Rao
  • Leijie Zhang


Recently, deep hashing (DH) methods have been proposed to learn specific image representations and a series of hash functions. However, existing DH methods mainly use convolutional neural networks (CNN) to extract global features, losing some local information. What’s more, the pairwise or triplet wise model applied in DH methods increases computational complexity and storage requirements. In this paper, we propose a new DH method called fused feature encoding (FFE). In FFE, we introduce a bypass from the intermediate convolutional layer to extract images’ local information and unify local and global information into one neural network to explore richer semantic information within the image. In our model, the number of neurons in the global or local encoding layer corresponds to the number of global or local encoding bits respectively. We also apply a new method to update the weights in our network to improve the efficiency. Experimental results show the superiority of the proposed approach over the state-of-the-arts.


Image retrieval Convolutional neural network Fused feature 



This work was supported in part by the National Natural Science Foundation of China (No. 61502129), the Zhejiang Provincial Natural Science Foundation of China (No. LQ16F020004). The authors would like to thank the reviewers in advance for their comments and suggestions. In addition, special thanks should go to Prof. Qin and Yuan Yong for their scientific advice and technical editing of the manuscript.


  1. 1.
    Abate AF, Nappi M, Tortora G, Tucci M (1999) Ime: an image management environment with content-based access. Image Vis Comput 17(13):967–980CrossRefGoogle Scholar
  2. 2.
    Babenko A, Lempitsky V (2015) Aggregating local deep features for image retrieval. In: Proceedings of the IEEE international conference on computer vision, pp 1269–1277Google Scholar
  3. 3.
    Babenko A, Slesarev A, Chigorin A, Lempitsky V (2014) Neural codes for image retrieval. In: European conference on computer vision. Springer, pp 584–599Google Scholar
  4. 4.
    Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828CrossRefGoogle Scholar
  5. 5.
    Chen Y, Zhou XS, Huang T (2001) One-class svm for learning in image retrieval. In: International conference on image processing, vol 1. IEEE, pp 34–37Google Scholar
  6. 6.
    Datar M, Immorlica N, Indyk P, Mirrokni VS (2004) Locality-sensitive hashing scheme based on p-stable distributions. In: Proceedings of the twentieth annual symposium on computational geometry. ACM, pp 253–262Google Scholar
  7. 7.
    Datta R, Joshi D, Li J, Wang J (2008) Image retrieval: ideas, influences, and trends of the new age. ACM Comput Surv 40(2):5CrossRefGoogle Scholar
  8. 8.
    Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Darrell T (2014) Decaf: a deep convolutional activation feature for generic visual recognition. In: International conference on machine learning, pp 647–655Google Scholar
  9. 9.
    Finkel RA, Bentley JL (1974) Quad trees a data structure for retrieval on composite keys. Acta Inform 4(1):1–9CrossRefGoogle Scholar
  10. 10.
    Ge T, He K, Ke Q, Sun J (2013) Optimized product quantization for approximate nearest neighbor search, pp 2946–2953Google Scholar
  11. 11.
    Gionis A, Indyk P, Motwani R et al (1999) Similarity search in high dimensions via hashing. VLDB 99:518–529Google Scholar
  12. 12.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587Google Scholar
  13. 13.
    Gong Y, Lazebnik S, Gordo A, Perronnin F (2013) Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans Pattern Anal Mach Intell 35(12):2916–2929CrossRefGoogle Scholar
  14. 14.
    Guttman A (1984) R-trees: a dynamic index structure for spatial searching, vol 14. ACMGoogle Scholar
  15. 15.
    He J, Chang SF, Radhakrishnan R, Bauer C (2011) Compact hashing with joint optimization of search accuracy and time. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 753– 760Google Scholar
  16. 16.
    Heo JP, Lee Y, He J, Chang SF, Yoon SE (2012) Spherical hashing. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 2957–2964Google Scholar
  17. 17.
    Jain AK, Vailaya A (1996) Image retrieval using color and shape. Pattern Recogn 29(8):1233–1244CrossRefGoogle Scholar
  18. 18.
    Jegou H, Douze M, Schmid C (2011) Product quantization for nearest neighbor search. IEEE Trans Pattern Anal Mach Intell 33(1):117–128CrossRefGoogle Scholar
  19. 19.
    Jin Z, Li C, Lin Y, Cai D (2014) Density sensitive hashing. IEEE Trans Cybern 44(8):1362–1371CrossRefGoogle Scholar
  20. 20.
    Ke Y, Sukthankar R, Huston L, Ke Y, Sukthankar R (2004) Efficient near-duplicate detection and sub-image retrieval. In: ACM multimedia, vol 4. Citeseer, p 5Google Scholar
  21. 21.
    Knuth D (1968) The art of computer programming 1: fundamental algorithms 2: seminumerical algorithms 3: sorting and searching. Addison-Wesley, Reading, p 30Google Scholar
  22. 22.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  23. 23.
    Kulis B, Grauman K (2009) Kernelized locality-sensitive hashing for scalable image search. In: International conference on computer vision. IEEE, pp 2130–2137Google Scholar
  24. 24.
    Lai H, Pan Y, Liu Y, Yan S (2015) Simultaneous feature learning and hash coding with deep neural networks. arXiv:1504.03410
  25. 25.
    Li WJ, Wang S, Kang WC (2015) Feature learning based deep supervised hashing with pairwise labels. arXiv:1511.03855
  26. 26.
    Li Q, Sun Z, He R, Tan T (2017) Deep supervised discrete hashing. In: Advances in neural information processing systems, pp 2479–2488Google Scholar
  27. 27.
    Lin K, Yang HF, Hsiao JH, Chen CS (2015) Deep learning of binary hash codes for fast image retrieval. In: IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 27–35Google Scholar
  28. 28.
    Liong VE, Lu J, Wang G, Moulin P, Zhou J et al (2015) Deep hashing for compact binary codes learning. In: CVPR, vol 1, p 3Google Scholar
  29. 29.
    Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282CrossRefGoogle Scholar
  30. 30.
    Liu W, Wang J, Kumar S, Chang SF (2011) Hashing with graphs. In: Proceedings of international conference on machine learning. Citeseer, pp 1–8Google Scholar
  31. 31.
    Ng JYH, Yang F, Davis LS (2015) Exploiting local features from deep networks for image retrieval. arXiv:1504.05133
  32. 32.
    Perronnin F, Liu Y, Sánchez J, Poirier H (2010) Large-scale image retrieval with compressed fisher vectors. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 3384–3391Google Scholar
  33. 33.
    Razavian AS, Azizpour H, Sullivan J, Carlsson S (2014) Cnn features off-the-shelf: an astounding baseline for recognition. In: IEEE conference on computer vision and pattern recognition workshops. IEEE, pp 512–519Google Scholar
  34. 34.
    Razavian AS, Azizpour H, Maki A, Sullivan J, Ek CH, Carlsson S (2015) Persistent evidence of local image properties in generic convnets. In: Scandinavian conference on image analysis. Springer, pp 249–262Google Scholar
  35. 35.
    Razavian AS, Sullivan J, Carlsson S, Maki A (2016) Visual instance retrieval with deep convolutional networks. ITE Trans Media Technol Appl 4(3):251–258CrossRefGoogle Scholar
  36. 36.
    Robinson JT (1981) The kdb-tree: a search structure for large multidimensional dynamic indexes. In: Proceedings of ACM international conference on Management of data. ACM, pp 10–18Google Scholar
  37. 37.
    Rui Y, Huang T, Chang SF (1999) Image retrieval: current techniques, promising directions, and open issues. J Vis Commun Image Represent 10(1):39–62CrossRefGoogle Scholar
  38. 38.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252MathSciNetCrossRefGoogle Scholar
  39. 39.
    Sasso G, Marsiglia HR, Pigatto F, Basilicata A, Gargiulo M, Abate AF, Nappi M, Pulley J, Sasso FS (2005) A visual query-by-example image database for chest ct images: potential role as a decision and educational support tool for radiologists. J Digit Imaging 18(1):78–84CrossRefGoogle Scholar
  40. 40.
    Schmid C, Mohr R (1997) Local grayvalue invariants for image retrieval. IEEE Trans Pattern Anal Mach Intell 19(5):530–535CrossRefGoogle Scholar
  41. 41.
    Seddati O, Dupont S, Mahmoudi S, Parian M, Dolez B (2017) Towards good practices for image retrieval based on cnn features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1246–1255Google Scholar
  42. 42.
    Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  43. 43.
    Velmurugan K, Baboo LDSS (2011) Content-based image retrieval using surf and colour moments. Global J Comp Sci Technol 11(1):4Google Scholar
  44. 44.
    Wang J, Kumar S, Chang SF (2010) Semi-supervised hashing for scalable image retrieval. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 3424–3431Google Scholar
  45. 45.
    Wan J, Wang D, Hoi SCH, Wu P, Zhu J, Zhang Y, Li J (2014) Deep learning for content-based image retrieval: A comprehensive study. In: Proceedings of ACM international conference on multimedia. ACM, pp 157–166Google Scholar
  46. 46.
    Wang X, Shi Y, Kitani KM (2016) Deep supervised hashing with triplet labels. In: Asian conference on computer vision. Springer, pp 70–84Google Scholar
  47. 47.
    Weiss Y, Torralba A, Fergus R (2009) Spectral hashing. In: Advances in neural information processing systems, pp 1753–1760Google Scholar
  48. 48.
    Xia R, Pan Y, Lai H, Liu C, Yan S (2014) Supervised hashing for image retrieval via image representation learning. In: AAAI, vol 1, p 2Google Scholar
  49. 49.
    Yan K, Wang Y, Liang D, Huang T, Tian Y (2016) Cnn vs. sift for image retrieval: alternative or complementary?. In: Proceedings of ACM on multimedia conference. ACM, pp 407–411Google Scholar
  50. 50.
    Yang HF, Lin K, Chen CS (2018) Supervised learning of semantics-preserving hash via deep convolutional neural networks. IEEE Trans Pattern Anal Mach Intell 40 (2):437–451CrossRefGoogle Scholar
  51. 51.
    Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. Neural Inf Process Syst 27:3320–3328Google Scholar
  52. 52.
    Zhang R, Lin L, Zhang R, Zuo W, Zhang L (2015) Bit-scalable deep hashing with regularized similarity learning for image retrieval and person re-identification. IEEE Trans Image Process 24(12):4766–4779MathSciNetCrossRefGoogle Scholar
  53. 53.
    Zhao F, Huang Y, Wang L, Tan T (2015) Deep semantic ranking based hashing for multi-label image retrieval. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 1556–1564Google Scholar
  54. 54.
    Zheng L, Wang S, Tian Q (2014) Coupled binary embedding for large-scale image retrieval. IEEE Trans Image Process 23(8):3368–3380MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Hangzhou Dianzi UniversityHangzhou CityChina
  2. 2.University of Technology SydneyUltimoAustralia

Personalised recommendations