Journal of Real-Time Image Processing

, Volume 16, Issue 1, pp 227–240 | Cite as

Partially shaded sketch-based image search in real mobile device environments via sketch-oriented compact neural codes

  • Jamil Ahmad
  • Khan Muhammad
  • Syed Inayat Ali Shah
  • Arun Kumar Sangaiah
  • Sung Wook BaikEmail author
Special Issue Paper


With the advent of touch screens in mobile devices, sketch-based image search is becoming the most intuitive method to query multimedia contents. Traditionally, sketch-based queries were formulated with hand-drawn shapes without any shades or colors. The absence of such critical information from sketches increased the ambiguity between natural images and their sketches. Although it was previously considered too cumbersome for users to add colors to hand-drawn sketches in image retrieval systems, the modern day touch input devices make it convenient to add shades or colors to query sketches. In this work, we propose deep neural codes extracted from partially colored sketches by an efficient convolutional neural network (CNN) fine-tuned on sketch-oriented augmented dataset. The training dataset is constructed with hand-drawn sketches, natural color images, de-colorized, and de-texturized images, coarse and fine edge maps, and flipped and rotated images. Fine-tuning CNN with augmented dataset enabled it to capture features effectively for representing partially colored sketches. We also studied the effects of shading and partial coloring on retrieval performance and show that the proposed method provides superior performance in sketch-based large-scale image retrieval on mobile devices as compared to other state-of-the-art methods.


Sketch-based query Image retrieval Hash codes Deep learning Convolutional neural network 



This work was supported by the National Research Foundation of Korea (NRF) Grant Funded by the Korea Government (MSIP) (No. 2016R1A2B4011712).


  1. 1.
    Ahmad, J., Muhammad, K., Lloret, J., Baik, S.W.: Efficient conversion of deep features to compact binary codes using fourier decomposition for multimedia big Data IEEE Trans. Ind. Inf. PP, 1–1 (2018)Google Scholar
  2. 2.
    Wang, S., Zhang, J., Han, T.X., Miao, Z.: Sketch-based image retrieval through hypothesis-driven object boundary selection with hlr descriptor. IEEE Trans. Multimed 17, 1045–1057 (2015)CrossRefGoogle Scholar
  3. 3.
    Ahmad, J., Sajjad, M., Rho, S., Baik, S.W.: Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems. Multimed. Tools Appl. 75, 12669–12692 (2016)CrossRefGoogle Scholar
  4. 4.
    Ahmad, J., Sajjad, M., Mehmood, I., Rho, S., Baik, S.W.: Saliency-weighted graphs for efficient visual content description and their applications in real-time image retrieval systems J. Real-Time Image Proc. 13, 431–447 (2017)CrossRefGoogle Scholar
  5. 5.
    Hu, R., Collomosse, J.: A performance evaluation of gradient field hog descriptor for sketch based image retrieval. Comput. Vis. Image Underst. 117, 790–806 (2013)CrossRefGoogle Scholar
  6. 6.
    Kim, S., Guy, S.J., Hillesland, K., Zafar, B., Gutub, A.A.-A., Manocha, D.: Velocity-based modeling of physical interactions in dense crowds. Vis. Comput. 31, 541–555 (2015)CrossRefGoogle Scholar
  7. 7.
    Tseng, K.-Y., Lin, Y.-L., Chen, Y.-H., Hsu, W.H.: Sketch-based image retrieval on mobile devices using compact hash bits. In Proceedings of the 20th ACM International Conference on Multimedia, pp. 913–916 (2012)Google Scholar
  8. 8.
    Al-Otaibi, N.A., Gutub, A.A.: 2-Leyer security system for hiding sensitive text data on personal computers. Lect. Notes Inf. Theory 2(2) (2014)Google Scholar
  9. 9.
    Abdelgawad, H., Shalaby, A., Abdulhai, B., Gutub, A.A.A.: Microscopic modeling of large-scale pedestrian–vehicle conflicts in the city of Madinah, Saudi Arabia. J. Adv. Transp. 48, 507–525 (2014)CrossRefGoogle Scholar
  10. 10.
    Eitz, M., Hays, J., Alexa, M.: How do humans sketch objects? ACM Trans. Graph. 31, 44:1–44:10 (2012)Google Scholar
  11. 11.
    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248–255 (2009)Google Scholar
  12. 12.
    Ahmad, J., Muhammad, K., Baik, S.W.: Medical image retrieval with compact binary codes generated in frequency domain using highly reactive convolutional features. J. Med. Syst. 42, 24 (2017)CrossRefGoogle Scholar
  13. 13.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 1, 1097–1105 (2012)Google Scholar
  14. 14.
    Ahmad, J., Muhammad, K., Bakshi, S., Baik, S.W.: Object-oriented convolutional features for fine-grained image retrieval in large surveillance datasets. Fut. Gen. Comput. Syst. 81, 314–330 (2018)CrossRefGoogle Scholar
  15. 15.
    Ahmad, J., Sajjad, M., Mehmood, I., Baik, S.W.: SiNC: saliency-injected neural codes for representation and efficient retrieval of medical radiographs. PLoS One 12, e0181707 (2017)CrossRefGoogle Scholar
  16. 16.
    Qi, Y., Song, Y.-Z., Zhang, H., Liu, J.: Sketch-based image retrieval via Siamese convolutional neural network. In Image Processing (ICIP), 2016 IEEE International Conference on, pp. 2460–2464 (2016)Google Scholar
  17. 17.
    Krizhevsky, A., Hinton, G.E.: Using very deep autoencoders for content-based image retrieval. In: Proceedings of the 19th European Symposium on Artificial Neural Networks, Bruges, Belgium (2011)Google Scholar
  18. 18.
    Babenko, A., Slesarev, A., Chigorin, A., Lempitsky, V.: Neural codes for image retrieval. In: Computer Vision–European Conference on Computer Vision (ECCV), Springer, pp. 584–599 (2014)Google Scholar
  19. 19.
    Gutub, A., Alharthi, N.: Improving Hajj and Umrah Services Utilizing Exploratory Data Visualization Techniques, presented at the Hajj Forum. Umm Al-Qura University–King Abdulaziz Historical Hall, Makkah (2016)Google Scholar
  20. 20.
    Gutub, A.: Exploratory data visualization for smart systems. Smart cities 2015-3rd annual digital grids and smart cities workshop, Burj Rafal Hotel Kempinski, Riyadh (2015)Google Scholar
  21. 21.
    Gutub, A.: Social media and its impact on e-Governance. ME smart cities 2015-4th middle east smart cities summit, 8–9 Dec, Dubai (2015)Google Scholar
  22. 22.
    Chen, T., Cheng, M.-M., Tan, P., Shamir, A., Hu, S.-M.: Sketch2photo: Internet image montage. ACM Trans. Graph. (TOG) 28, 124 (2009)Google Scholar
  23. 23.
    Fu, L., Kara, L.B.: Recognizing network-like hand-drawn sketches: a convolutional neural network approach. In ASME 2009 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, pp. 671–681 (2009)Google Scholar
  24. 24.
    Sarvadevabhatla, R.K., Babu, R.V.: Freehand sketch recognition using deep features. arXiv:1502.00254 (2015)Google Scholar
  25. 25.
    Al-Otaibi, N.A., Gutub, A.A.: Flexible stego-system for hiding text in images of personal computers based on user security priority. In: Proceedings of: 2014 International Conference on Advanced Engineering Technologies (AET-2014), pp. 250–256 (2014)Google Scholar
  26. 26.
    Seddati, O., Dupont, S., Mahmoudi, S.: Deepsketch: deep convolutional neural networks for sketch recognition and similarity search. In: Content-Based Multimedia Indexing (CBMI), 2015 13th International Workshop on, pp. 1–6 (2015)Google Scholar
  27. 27.
    Liu, L., Shen, F., Shen, Y., Liu, X., Shao, L.: Deep sketch hashing: fast free-hand sketch-based image retrieval. arXiv:1703.05605 (2017)Google Scholar
  28. 28.
    Wang, X., Duan, X., Bai, X.: Deep sketch feature for cross-domain image retrieval. Neurocomputing 207, 387–397 (2016)CrossRefGoogle Scholar
  29. 29.
    Ahmad, J., Mehmood, I., Baik, S.W.: Efficient object-based surveillance image search using spatial pooling of convolutional features. J. Vis. Commun. Image Rep. 45, 62–76 (2017)CrossRefGoogle Scholar
  30. 30.
    Iandola, F.N., Moskewicz, M.W., Ashraf, K., Han, S., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and < 1 MB model size. arXiv:1602.07360 (2016)Google Scholar
  31. 31.
    Ahmad, J., Muhammad, K., Baik, S.W.: Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. PLoS One 12, e0183838 (2017)CrossRefGoogle Scholar
  32. 32.
    Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 629–639 (1990)CrossRefGoogle Scholar
  33. 33.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)Google Scholar
  34. 34.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  35. 35.
    Lin, M., Chen, Q., Yan, S.: Network in network. arXiv:1312.4400 (2013)Google Scholar
  36. 36.
    Gschwend, D.: Zynqnet: an fpga-accelerated embedded convolutional neural network. MS thesis, Swiss Federal Institute of Technology Zurich (ETH-Zurich) (2016)Google Scholar
  37. 37.
    Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications arXiv:1704.04861 (2017)Google Scholar
  38. 38.
    Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR 2, abs/1510.00149 (2015)Google Scholar
  39. 39.
    MathWorks (2015) MATLAB. Available:
  40. 40.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., et al.: Caffe: Convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia, pp. 675–678 (2014)Google Scholar
  41. 41.
    Nvidia DIGITS: Available: (2016)
  42. 42.
    Caltech-256 Object Category Dataset: Available:
  43. 43.
    Çalışır, F., Baştan, M., Ulusoy, Ö., Güdükbay, U.: Mobile multi-view object image search. Multimed. Tool Appl. 76, 12433–12456 (2017)CrossRefGoogle Scholar
  44. 44.
    Yang, Y., Hospedales, T.M.: Deep neural networks for sketch recognition. arXiv:1501.07873 (2015)Google Scholar
  45. 45.
    Lin, K., Yang, H.-F., Hsiao, J.-H., Chen, C.-S., Deep learning of binary hash codes for fast image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 27–35 (2015)Google Scholar
  46. 46.
    Gionis, A., Indyk, P., Motwani, R.: Similarity search in high dimensions via hashing. VLDB, pp. 518–529 (1999)Google Scholar
  47. 47.
    Datar, M., Immorlica, N., Indyk, P., Mirrokni, V.S.: Locality-sensitive hashing scheme based on p-stable distributions. In: Proceedings of the Twentieth Annual Symposium on Computational Geometry, pp. 253–262 (2004)Google Scholar
  48. 48.
    Weiss, Y., Torralba, A., Fergus, R.: Spectral hashing. Advances in neural information processing systems, pp. 1753–1760 (2009)Google Scholar
  49. 49.
    Jin, Z., Li, C., Lin, Y., Cai, D.: Density sensitive hashing. IEEE Trans. Cybern. 44, 1362–1371 (2014)CrossRefGoogle Scholar
  50. 50.
    Heo, J.-P., Lee, Y., He, J., Chang, S.-F., Yoon, S.-E.: Spherical hashing. In: Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 2957–2964 (2012)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Jamil Ahmad
    • 1
  • Khan Muhammad
    • 2
  • Syed Inayat Ali Shah
    • 3
  • Arun Kumar Sangaiah
    • 4
  • Sung Wook Baik
    • 2
    Email author
  1. 1.Department of Computer ScienceIslamia CollegePeshawarPakistan
  2. 2.Digital Contents Research InstituteSejong UniversitySeoulSouth Korea
  3. 3.Department of MathematicsIslamia College PeshawarPeshawarPakistan
  4. 4.School of Computing Science and EngineeringVIT UniversityVelloreIndia

Personalised recommendations