Multimedia Tools and Applications

, Volume 78, Issue 12, pp 15751–15777 | Cite as

Object detection and classification: a joint selection and fusion strategy of deep convolutional neural network and SIFT point features

  • Muhammad Rashid
  • Muhammad Attique KhanEmail author
  • Muhammad Sharif
  • Mudassar Raza
  • Muhammad Masood Sarfraz
  • Farhat Afza


In the area of machine learning and pattern recognition, object classification is getting an attraction due to its range of applications such as visual surveillance. In recent times, numerous deep learning-based methods are presented for object classification but still, set of problems/concerns exists which reduce the overall classification accuracy. Complex background, congest situtaions, and similarity among different objects are few challenging issues. To tackle such problems, we propose a technique by using deep convolutional neural network (DCNN) and scale invariant features transform (SIFT). First, an improved saliency method is implemented, and the point features are extracted. Then, DCNN features are extracted from two deep CNN models like VGG and AlexNet. Thereafter, Reyni entropy-controlled method is implemented on DCNN pooling and the SIFT point matrix to select the robust features. Finally, the selected robust features are fused in a matrix by a serial approach, which is later fed to ensemble classifier for recognition. The proposed method is evaluated on three publically available datasets including Caltech101, Barkley 3D, and Pascal 3D and obtained classification accuracy of 93.8%, 99%, and 88.6% - clearly showing the exceptional performance compared to existing methods.


Object detection Hand crafted features Deep CNN Features fusion Features reduction Classification 



  1. 1.
    Achanta R et al (2012) SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 34(11):2274–2282CrossRefGoogle Scholar
  2. 2.
    Agrawal S et al (2018) A comparative study of fuzzy PSO and fuzzy SVD-based RBF neural network for multi-label classification. Neural Comput & Applic 29(1):245–256CrossRefGoogle Scholar
  3. 3.
    Akcay S, et al (2018) Using deep convolutional neural network architectures for object classification and detection within X-ray baggage security imagery. IEEE Trans Inf Forensics SecurGoogle Scholar
  4. 4.
    Akram T, et al (2018) Skin lesion segmentation and recognition using multichannel saliency estimation and M-SVM on selected serially fused features. J Ambient Intell Humaniz Comput:1–20Google Scholar
  5. 5.
    Arel I, Rose DC, Karnowski TP (2010) Deep machine learning-a new frontier in artificial intelligence research [research frontier]. IEEE Comput Intell Mag 5(4):13–18CrossRefGoogle Scholar
  6. 6.
    Chen S et al (2018) Local patch vectors encoded by fisher vectors for image classification. Information 9(2):38MathSciNetCrossRefGoogle Scholar
  7. 7.
    Cheng G et al (2016) Study on planetary gear fault diagnosis based on entropy feature fusion of ensemble empirical mode decomposition. Measurement 91:140–154CrossRefGoogle Scholar
  8. 8.
    Dong H, et al (2018) A novel hybrid genetic algorithm with granular information for feature selection and optimization. Appl Soft ComputGoogle Scholar
  9. 9.
    Ejbali R, Zaied M (2018) A dyadic multi-resolution deep convolutional neural wavelet network for image classification. Multimed Tools Appl 77(5):6149–6163CrossRefGoogle Scholar
  10. 10.
    Esteva A et al (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115CrossRefGoogle Scholar
  11. 11.
    Everingham, M., et al., The pascal visual object classes challenge: a retrospective. Int J Comput Vis, 2015. 111(1): p. 98–136Google Scholar
  12. 12.
  13. 13.
    Fang X, et al (2018) Approximate low-rank projection learning for feature extraction. IEEE Transactions on Neural Networks and Learning SystemsGoogle Scholar
  14. 14.
    Fei-Fei L, Fergus R, Perona P (2006) One-shot learning of object categories. IEEE Trans Pattern Anal Mach Intell 28(4):594–611CrossRefGoogle Scholar
  15. 15.
    Fondón I, et al (2018) Automatic classification of tissue malignancy for breast carcinoma diagnosis. J Comput Biol MedGoogle Scholar
  16. 16.
    Ghose U, Mehta R (2018) Attribute reduction method using the combination of entropy and fuzzy entropy. In: Networking Communication and Data Knowledge Engineering. Springer, p 169–177Google Scholar
  17. 17.
    Gomathi D, Seetharaman K Object classification techniques using tree based classifiersGoogle Scholar
  18. 18.
    Grabner A, Roth PM, Lepetit V (2018) 3d pose estimation and 3d model retrieval for objects in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  19. 19.
    He K, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  20. 20.
    Hu F et al (2015) Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery. Remote Sens 7(11):14680–14707CrossRefGoogle Scholar
  21. 21.
    Janoch A, et al (2013) A category-level 3d object dataset: putting the kinect to work. In: Consumer depth cameras for computer vision. Springer, p 141–165Google Scholar
  22. 22.
    Juuti M, Corona F, Karhunen J (2018) Stochastic discriminant analysis for linear supervised dimension reduction. NeurocomputingGoogle Scholar
  23. 23.
    Khan MA et al (2017) License number plate recognition system using entropy-based features selection approach with SVM. IET Image Process 12(2):200–209CrossRefGoogle Scholar
  24. 24.
    Khan MA, et al (2018) An implementation of optimized framework for action classification using multilayers neural network on selected fused features. Pattern Anal Applic:1–21Google Scholar
  25. 25.
    Khan MA et al (2018) CCDF: automatic system for segmentation and recognition of fruit crops diseases based on correlation coefficient and deep CNN features. Comput Electron Agric 155:220–236CrossRefGoogle Scholar
  26. 26.
    Khan MA et al (2018) An implementation of normal distribution based segmentation and entropy controlled features selection for skin lesion detection and classification. BMC Cancer 18(1):638CrossRefGoogle Scholar
  27. 27.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systemsGoogle Scholar
  28. 28.
    Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Computer vision and pattern recognition, 2006 IEEE computer society conference on. IEEEGoogle Scholar
  29. 29.
    Leng L, et al (2010) Dynamic weighted discrimination power analysis in DCT domain for face and palmprint recognition. In: Information and Communication Technology Convergence (ICTC), 2010 International Conference on. IEEEGoogle Scholar
  30. 30.
    Leng L, Zhang J, Khan MK, Chen X, Alghathbar K (2010) Dynamic weighted discrimination power analysis: a novel approach for face and palmprint recognition in DCT domain. International Journal of Physical Sciences 5(17):2543–2554Google Scholar
  31. 31.
    Leng L, et al (2011) Two dimensional PalmPhasor enhanced by multi-orientation score level fusion. In: FTRA International Conference on Secure and Trust Computing, Data Management, and Application. SpringerGoogle Scholar
  32. 32.
    Leng L, et al (2011) Two-directional two-dimensional random projection and its variations for face and palmprint recognition. In: International Conference on Computational Science and Its Applications. SpringerGoogle Scholar
  33. 33.
    Leng L, et al (2012) Two-dimensional cancelable biometric scheme. In: Wavelet Analysis and Pattern Recognition (ICWAPR), 2012 International Conference on. IEEEGoogle Scholar
  34. 34.
    Leng L, Li M, Teoh ABJ (2013) Conjugate 2dpalmhash code for secure palm-print-vein verification. In: Image and Signal Processing (CISP), 2013 6th International Congress on. IEEEGoogle Scholar
  35. 35.
    Leng L et al (2017) Dual-source discrimination power analysis for multi-instance contactless palmprint recognition. Multimed Tools Appl 76(1):333–354CrossRefGoogle Scholar
  36. 36.
    Li B, et al (2017) Skeleton based action recognition using translation-scale invariant image mapping and multi-scale deep CNN. In: Multimedia & Expo Workshops (ICMEW), 2017 IEEE International Conference on. IEEEGoogle Scholar
  37. 37.
    Li K et al (2018) Multi-modal feature fusion for geographic image annotation. Pattern Recogn 73:1–14CrossRefGoogle Scholar
  38. 38.
    Li Q, et al (2018) Improving image classification accuracy with ELM and CSIFT. Comput Sci EngGoogle Scholar
  39. 39.
    Liaqat A, et al (2018) Automated ulcer and bleeding classification from wce images using multiple features fusion and selection. Journal of Mechanics in Medicine and Biology:1850038Google Scholar
  40. 40.
    Liu L, Wang L, Liu X (2011) In defense of soft-assignment coding. In: Computer Vision (ICCV), 2011 IEEE International Conference on. IEEEGoogle Scholar
  41. 41.
    Liu W et al (2017) A survey of deep neural network architectures and their applications. Neurocomputing 234:11–26CrossRefGoogle Scholar
  42. 42.
    Liu W, Yang X, Tao D, Cheng J, Tang Y (2018) Multiview dimension reduction via hessian multiset canonical correlations. Information Fusion 41:119–128Google Scholar
  43. 43.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110CrossRefGoogle Scholar
  44. 44.
    Naeini AA et al (2018) Particle swarm optimization for object-based feature selection of VHSR satellite images. IEEE Geosci Remote Sens Lett 15(3):379–383CrossRefGoogle Scholar
  45. 45.
    Pan Y, et al Locality constrained encoding of frequency and spatial information for image classification. Multimed Tools Appl:1–17Google Scholar
  46. 46.
    Qin C, Sun M, Chang C-C (2018) Perceptual hashing for color images based on hybrid extraction of structural features. Signal Process 142:194–205CrossRefGoogle Scholar
  47. 47.
    Rastegari M, et al (2016) Xnor-net: imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision. SpringerGoogle Scholar
  48. 48.
    Raza M et al (2018) Appearance based pedestrians’ gender recognition by employing stacked auto encoders in deep learning. Futur Gener Comput Syst 88:28–39CrossRefGoogle Scholar
  49. 49.
    Roy PK, Om H (2018) Suspicious and violent activity detection of humans using HOG features and SVM classifier in surveillance videos. In: Advances in soft computing and machine learning in image processing. Springer, p 277–294Google Scholar
  50. 50.
    Sankar AS, Nair SS, Dharan VS, Sankaran P (2015) Wavelet sub band entropy based feature extraction method for BCI. Procedia Computer Science 46:1476–1482Google Scholar
  51. 51.
    Sharif M, Khan MA, Akram T, Javed MY, Saba T, Rehman A (2017) A framework of human detection and action recognition based on uniform segmentation and combination of Euclidean distance and joint entropy-based features selection. EURASIP Journal on Image and Video Processing 2017(1):89Google Scholar
  52. 52.
    Sharif M, et al. (2018) A framework for offline signature verification system: best features selection approach. Pattern Recogn LettGoogle Scholar
  53. 53.
    Sharif M et al (2018) Detection and classification of citrus diseases in agriculture based on optimized weighted segmentation and feature selection. Comput Electron Agric 150:220–234CrossRefGoogle Scholar
  54. 54.
    Siddiqui S, Khan MA, Bashir K, Sharif M, Azam F, Javed MY (2018) Human action recognition: a construction of codebook by discriminative features selection approach. International Journal of Applied Pattern Recognition 5(3):206–228Google Scholar
  55. 55.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556Google Scholar
  56. 56.
    Singh C, Walia E, Kaur KP (2018) Enhancing color image retrieval performance with feature fusion and non-linear support vector machine classifier. Optik 158:127–141CrossRefGoogle Scholar
  57. 57.
    Song J, et al (2018) Structure preserving dimensionality reduction for visual object recognition. Multimed Tools Appl:1–17Google Scholar
  58. 58.
    Szegedy C, et al (2015) Going deeper with convolutions. CvprGoogle Scholar
  59. 59.
    Wei G et al (2018) Content-based image retrieval for lung nodule classification using texture features and learned distance metric. J Med Syst 42(1):13CrossRefGoogle Scholar
  60. 60.
    Yu W, et al (2018) Hierarchical semantic image matching using CNN feature pyramid. Comput Vis Image UnderstGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Muhammad Rashid
    • 1
  • Muhammad Attique Khan
    • 2
    Email author
  • Muhammad Sharif
    • 1
  • Mudassar Raza
    • 1
  • Muhammad Masood Sarfraz
    • 3
  • Farhat Afza
    • 1
  1. 1.Department of Computer ScienceCOMSATS University IslamabadIslamabadPakistan
  2. 2.Department of Computer Science and EngineeringHITEC UniversityTaxilaPakistan
  3. 3.Department of Electrical EngineeringCOMSATS University IslamabadIslamabadPakistan

Personalised recommendations