Skip to main content
SpringerLink
Log in

Retrieving Aerial Scene Images with Learned Deep Image-Sketch Features

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

This paper investigates the problem of retrieving aerial scene images by using semantic sketches, since the state-of-the-art retrieval systems turn out to be invalid when there is no exemplar query aerial image available. However, due to the complex surface structures and huge variations of resolutions of aerial images, it is very challenging to retrieve aerial images with sketches and few studies have been devoted to this task. In this article, for the first time to our knowledge, we propose a framework to bridge the gap between sketches and aerial images. First, an aerial sketch-image database is collected, and the images and sketches it contains are augmented to various levels of details. We then train a multi-scale deep model by the new dataset. The fully-connected layers of the network in each scale are finally connected and used as cross-domain features, and the Euclidean distance is used to measure the cross-domain similarity between aerial images and sketches. Experiments on several commonly used aerial image datasets demonstrate the superiority of the proposed method compared with the traditional approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hu F, Xia G S, Wang Z, Huang X, Zhang L, Sun H. Unsupervised feature learning via spectral clustering of multidimensional patches for remotely sensed scene classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(5): 2015-2030.

    Article  Google Scholar 

  2. Xia G S, Wang Z, Xiong C, Zhang L. Accurate annotation of remote sensing images via active spectral clustering with little expert knowledge. Remote Sensing, 2015, 7(11): 15014-15045.

    Article  Google Scholar 

  3. Hu F, Xia G S, Hu J, Zhang L. Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sensing, 2015, 7(11): 14680-14707.

    Article  Google Scholar 

  4. Aptoula E. Remote sensing image retrieval with global morphological texture descriptors. IEEE Transactions on Geoscience and Remote Sensing, 2014, 52(5): 3023-3034.

    Article  Google Scholar 

  5. Demir B, Bruzzone L. A novel active learning method in relevance feedback for content-based remote sensing image retrieval. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(5): 2323-2334.

    Article  Google Scholar 

  6. Demir B, Bruzzone L. Hashing-based scalable remote sensing image search and retrieval in large archives. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(2): 892-904.

    Article  Google Scholar 

  7. Ozkan S, Ates T, Tola E, Soysal M et al. Performance analysis of state-of-the-art representation methods for geographical image retrieval and categorization. IEEE Geoscience and Remote Sensing Letters, 2014, 11(11): 1996-2000.

    Article  Google Scholar 

  8. Ferecatu M, Boujemaa N. Interactive remote-sensing image retrieval using active relevance feedback. IEEE Transactions on Geoscience Remote Sensing, 2007, 45(4): 818-826.

    Article  Google Scholar 

  9. Du Z, Li X, Lu X. Local structure learning in high resolution remote sensing image retrieval. Neurocomputing, 2016, 207: 813-822.

    Article  Google Scholar 

  10. Yang Y, Newsam S. Geographic image retrieval using local invariant features. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(2): 818-832.

    Article  Google Scholar 

  11. Liu T, Zhang L, Li P, Lin H. Remotely sensed image retrieval based on region-level semantic mining. EURASIP Journal on Image Video Processing, 2012, 2012(1): 4-15.

    Article  Google Scholar 

  12. Eitz M, Hays J, Alexa M. How do humans sketch objects? ACM Transactions on Graphics, 2012, 31(4): 44:1-44:10.

  13. Bai X, Latecki L J. Path similarity skeleton graph matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(7): 1282-1292.

    Article  Google Scholar 

  14. Wang X, Feng B, Bai X, LiuW, Latecki L J. Bag of contour fragments for robust shape classification. Pattern Recognition, 2014, 47(6): 2116-2125.

    Article  Google Scholar 

  15. Bai X, Bai S, Zhu Z, Latecki L J. 3D shape matching via two layer coding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(12): 2361-2373.

    Article  Google Scholar 

  16. Shen W, Wang X, Wang Y, Bai X, Zhang Z. DeepContour: A deep convolutional feature learned by positive-sharing loss for contour detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp.3982-3991.

  17. Eitz M, Hildebrand K, Boubekeur T, Alexa M. Sketchbased image retrieval: Benchmark and bag-of-features descriptors. IEEE Transactions on Visualization Computer Graphics, 2011, 17(11): 1624-1636.

    Article  Google Scholar 

  18. Hu R, Collomosse J. A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Computer Vision Image Understanding, 2013, 117(7): 790-806.

    Article  Google Scholar 

  19. Cao Y, Wang C, Zhang L, Zhang L. Edgel index for largescale sketch-based image search. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2011, pp.761-768.

  20. Chen T, Cheng M M, Tan P, Shamir A, Hu S M. Sketch2Photo: Internet image montage. ACM Transactions on Graphics, 2009, 28(5): 124:1-124:10.

  21. Yu Q, Liu F, Song Y Z, Xiang T, Hospedales T M, Loy C C. Sketch me that shoe. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp.799-807.

  22. Qi Y, Song Y Z, Zhang H, Liu J. Sketch-based image retrieval via Siamese convolutional neural network. In Proc. IEEE International Conference on Image Processing (ICIP), September 2016, pp.2460-2464.

  23. Wang X, Duan X, Bai X. Deep sketch feature for crossdomain image retrieval. Neurocomputing, 2016, 207: 387-397.

    Article  Google Scholar 

  24. Yang Y, Newsam S. Bag-of-visual-words and spatial extensions for land-use classification. In Proc. the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, November 2010, pp.270-279.

  25. Xia G S, Yang W, Delon J, Gousseau Y, Sun H, Maître H. Structural high-resolution satellite image indexing. In Proc. ISPRS TC VII Symposium-100 Years ISPRS, Volume 38, September 2010, pp.298-303.

  26. Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. Advances in Neural Information Processing Systems (NIPS), December 2012, pp.1097-1105.

  27. Zhang Q, Shen X, Xu L, Jia J. Rolling guidance filter. In Proc. the 13th European Conference on Computer Vision (ECCV), September 2014, pp.815-830.

  28. Hu J, Jiang T, Tong X, Xia G S, Zhang L. A benchmark for scene classification of high spatial resolution remote sensing imagery. In Proc. IEEE International Geoscience and Remote Sensing Symposium (IGARSS), July 2015, pp.5003-5006.

  29. Xia G S, Hu J, Hu F, Shi B, Bai X, Zhong Y, Zhang L. AID: A benchmark dataset for performance evaluation of aerial scene classification. arXiv:1608.05167, 2016. https://arxiv.org/abs/1608.05167v1, May 2017.

  30. Zitnick C L, Dollár P. Edge boxes: Locating object proposals from edges. In Proc. European Conference on Computer Vision (ECCV), September 2014, pp.391-405.

  31. Oliva A, Torralba A. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision, 2011, 42(3): 145-175.

    Article  MATH  Google Scholar 

  32. Lowe D G. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60(2): 91-110.

    Article  Google Scholar 

  33. Grauman K, Darrell T. The pyramid match kernel: Discriminative classification with sets of image features. In Proc. the 10th IEEE International Conference on Computer Vision (ICCV), October 2005, pp.1458-1465.

  34. Lazebnik S, Schmid C, Ponce J. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2006, pp.2169-2178.

  35. Dalal N, Triggs B. Histograms of oriented gradients for human detection. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2005, pp.886-893.

  36. Xie S, Tu Z. Holistically-nested edge detection. In Proc. IEEE International Conference on Computer Vision (ICCV), December 2015, pp.1395-1403.

  37. Babenko A, Slesarev A, Chigorin A, Lempitsky V. Neural codes for image retrieval. In Proc. European Conference on Computer Vision (ECCV), September 2014, pp.584-599.

  38. Mahendran A, Vedaldi A. Understanding deep image representations by inverting them. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp.5188-5196.

Download references

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations

  1. State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, 430079, China

    Tian-Bi Jiang, Gui-Song Xia & Wei-Ming Shen

  2. School of Electronic Information, Wuhan University, Wuhan, 430072, China

    Tian-Bi Jiang & Qi-Kai Lu

Authors
  1. Tian-Bi Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gui-Song Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qi-Kai Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei-Ming Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qi-Kai Lu.

Electronic supplementary material

Below is the link to the electronic supplementary material.

ESM 1

(PDF 593 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, TB., Xia, GS., Lu, QK. et al. Retrieving Aerial Scene Images with Learned Deep Image-Sketch Features. J. Comput. Sci. Technol. 32, 726–737 (2017). https://doi.org/10.1007/s11390-017-1754-7

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-017-1754-7

Keywords

Search

Navigation