Improving Deep Neural Network Performance with Kernelized Min-Max Objective

  • Kai Yao
  • Kaizhu HuangEmail author
  • Rui Zhang
  • Amir Hussain
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11301)


In this paper, we present a novel training strategy using kernelized Min-Max objective to enable improved object recognition performance on deep neural networks (DNN), e.g., convolutional neural networks (CNN). Without changing the other part of the original model, the kernelized Min-Max objective works by combining the kernel trick with the Min-Max objective and being embedded into a high layer of the networks in the training phase. The proposed kernelized objective explicitly enforces the learned object feature maps to maintain in a kernel space the least compactness for each category manifold and the biggest margin among different category manifolds. With very few additional computation costs, the proposed strategy can be widely used in different DNN models. Extensive experiments with shallow convolutional neural network model, deep convolutional neural network model, and deep residual neural network model on two benchmark datasets show that the proposed approach outperforms those competitive models.



The work was partially supported by the following: National Natural Science Foundation of China under grant no. 61473236 and 61876155; Natural Science Fund for Colleges and Universities in Jiangsu Province under grant no. 17KJD520010; Suzhou Science and Technology Program under grant no. SYG201712, SZS201613; Jiangsu University Natural Science Research Programme under grant no. 17KJB- 520041; Key Program Special Fund in XJTLU under no. KSF-A-01 and KSF-P-02.


  1. 1.
    Goodfellow, I., Wardefarley, D., Mirza, M., Courvile, A., Bengio, Y.: Maxout networks. In: ICML (2013)Google Scholar
  2. 2.
    He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 346–361. Springer, Cham (2014). Scholar
  3. 3.
    Huang, K., Yang, H., King, I., Lyu, M.R.: Learning classifiers from imbalanced data based on biased minimax probability machine. Proc. CVPR 2, 558–563 (2004)Google Scholar
  4. 4.
    Huang, K., Yang, H., King, I., Lyu, M.R.: Machine Learning: Modeling Data Locally and Globally. Springer, Heidelberg (2008). ISBN 3-5407-9451-4CrossRefzbMATHGoogle Scholar
  5. 5.
    Huang, K., Yang, H., King, I., Lyu, M.R.: Maxi-min margin machine: learning large margin classifiers globally and locally. IEEE Trans. Neural Netw. 19, 260–272 (2008)CrossRefGoogle Scholar
  6. 6.
    Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)Google Scholar
  7. 7.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  8. 8.
    Lee, C., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: NIPS (2014)Google Scholar
  9. 9.
    Lin, M., Chen, Q., Yan, S.: Network in network. In: ICLR (2014)Google Scholar
  10. 10.
    Lyu, C., Huang, K., Liang, H.N.: A unified gradient regularization family for adversarial examples. In: 2015 IEEE International Conference on Data Mining (ICDM), pp. 301–309. IEEE (2015)Google Scholar
  11. 11.
    Shi, W., Gong, Y., Wang, J.: Improving CNN performance with min-max objective. In: International Joint Conference on Artificial Intelligence (2016)Google Scholar
  12. 12.
    Snock, J., Larochelle, H., Adams, R.: Practical Bayesian optimization of machine learning algorithm. In: NIPS (2012)Google Scholar
  13. 13.
    Springenberg, J., Riedmiller, M.: Improving deep neural networks with probabilistic maxout units. In: ICLR (2014)Google Scholar
  14. 14.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. In: JMLR (2014)Google Scholar
  15. 15.
    Wang, J., Zhang, B., Sun, Z., Hao, W., Sun, Q.: A novel conjugate gradient method with generalized Armijo search for efficient training of feedforward neural networks. Neurocomputing 275, 308–316 (2018)CrossRefGoogle Scholar
  16. 16.
    Xu, B., Huang, K., Liu, C.L.: Maxi-min discriminant analysis via online learning. Neural Netw. 34, 56–64 (2012)CrossRefGoogle Scholar
  17. 17.
    Yan, S., Xu, D., Zhang, B., Zhang, H., Yang, Q., Lin, S.: Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans. PAMI 29, 40–51 (2007)CrossRefGoogle Scholar
  18. 18.
    Zeiler, M., Fergus, R.: Stochastic pooling for regularization of deep convolutional neural networks. In: ICLR (2013)Google Scholar
  19. 19.
    Zhang, S., Huang, K., Zhang, R., Hussain, A.: Learning from few samples with memory network. Cogn. Comput. 10(1), 15–22 (2018)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Kai Yao
    • 1
  • Kaizhu Huang
    • 1
    Email author
  • Rui Zhang
    • 2
  • Amir Hussain
    • 3
  1. 1.Department of EEEXi’an Jiaotong-Liverpool UniversitySuzhouPeople’s Republic of China
  2. 2.Department of MSXi’an Jiaotong-Liverpool UniversitySuzhouPeople’s Republic of China
  3. 3.Division of Computing Science and Maths, School of Natural SciencesUniversity of StirlingStirlingUK

Personalised recommendations