Robust multi-layer extreme learning machine using bias-variance tradeoff

基于偏差-方差权衡的多层鲁棒极限学习机模型

Abstract

As a new neural network model, extreme learning machine (ELM) has a good learning rate and generalization ability. However, ELM with a single hidden layer structure often fails to achieve good results when faced with large-scale multi-featured problems. To resolve this problem, we propose a multi-layer framework for the ELM learning algorithm to improve the model’s generalization ability. Moreover, noises or abnormal points often exist in practical applications, and they result in the inability to obtain clean training data. The generalization ability of the original ELM decreases under such circumstances. To address this issue, we add model bias and variance to the loss function so that the model gains the ability to minimize model bias and model variance, thus reducing the influence of noise signals. A new robust multi-layer algorithm called ML-RELM is proposed to enhance outlier robustness in complex datasets. Simulation results show that the method has high generalization ability and strong robustness to noise.

摘要

极限学习机(ELM)作为一种新型的神经网络模型, 具有良好的学习速度和泛化能力. 然而, 单隐层结构的 ELM 在面临大规模多特征问题时往往不能取得良好的效果, 为了解决这个问题, 提出一个新型的多层 ELM 学习算法框架来提高模型的泛化能力. 此外, 在实际应用中, 经常会因为出现噪声或异常点而导致训练数据被污染, 面对被污染的数据, 普通的 ELM 的泛化能力会下降. 为了解决这个问题, 利用偏差-方差分解理论, 在损失函数中加入模型的偏差和方差, 使模型获得最小化模型偏差和模型方差的能力, 从而降低噪声信号的影响. 我们提出一种新的鲁棒多层算法 ML-RELM, 来提升在含有离群点的复杂数据集中的鲁棒性. 仿真结果表明, 该方法具有较强的泛化能力和较强的抗噪声能力.

This is a preview of subscription content, access via your institution.

References

  1. [1]

    HUANG Guang-bin, ZHU Qin-yu, SIEW C K. Extreme learning machine: A new learning scheme of feedforward neural networks [C]// 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541). 2004, 2: 985–990.

    Google Scholar 

  2. [2]

    HUANG Guang-bin, ZHU Qin-yu, SIEW C K. Extreme learning machine: Theory and applications [J]. Neurocomputing, 2006, 70(1–3): 489–501.

    Article  Google Scholar 

  3. [3]

    HUANG Zhi-yong, YU Yuan-long, GU J, LIU Hua-ping. An efficient method for traffic sign recognition based on extreme learning machine [J]. IEEE Transactions on Cybernetics, 2017, 47(4): 920–933. DOI: https://doi.org/10.1109/TCYB.2016.2533424.

    Article  Google Scholar 

  4. [4]

    IOSIFIDIS A, TEFAS A, PITAS I. Approximate kernel extreme learning machine for large scale data classification [J]. Neurocomputing, 2017, 219: 210–220. DOI: https://doi.org/10.1016/j.neucom.2016.09.023.

    Article  Google Scholar 

  5. [5]

    YANG Yi-min, WU Q M J. Extreme learning machine with subnetwork hidden nodes for regression and classification [J]. IEEE Transactions on Cybernetics, 2016, 46(12): 2885–2898. DOI: https://doi.org/10.1109/tcyb.2015.2492468.

    Article  Google Scholar 

  6. [6]

    XU Xin-zheng, SHAN Dong, LI Shan, SUN Tong-feng, XIAO Peng-cheng, FAN Jian-ping. Multi-label learning method based on ML-RBF and laplacian ELM [J]. Neurocomputing, 2019, 331: 213–219. DOI: https://doi.org/10.1016/j.neucom.2018.11.018.

    Article  Google Scholar 

  7. [7]

    INABA F K, TEATINI SALLES E O, PERRON S, CAPOROSSI G. DGR-ELM-distributed generalized regularized ELM for classification [J]. Neurocomputing, 2018, 275: 1522–1530. DOI: https://doi.org/10.1016/j.neucom.2017.09.090.

    Article  Google Scholar 

  8. [8]

    ZHANG Yu, WANG Yu, ZHOU Guo-xu, JIN Jing, WANG Bei, WANG Xing-yu, CICHOCKI A. Multi-kernel extreme learning machine for EEG classification in brain-computer interfaces [J]. Expert Systems with Applications, 2018, 96: 302–310. DOI: https://doi.org/10.1016/j.eswa.2017.12.015.

    Article  Google Scholar 

  9. [9]

    DAI Hao-zhen, CAO Jiu-wen, WANG Tian-lei, DENG Mu-qing, YANG Zhi-xin. Multilayer one-class extreme learning machine [J]. Neural Networks, 2019, 115: 11–22. DOI: https://doi.org/10.1016/j.neunet.2019.03.004.

    Article  Google Scholar 

  10. [10]

    CHYZHYK D, SAVIO A, GRANA M. Computer aided diagnosis of schizophrenia on resting state fMRI data by ensembles of ELM [J]. Neural Networks, 2015, 68: 23–33. DOI: https://doi.org/10.1016/j.neunet.2015.04.002.

    Article  Google Scholar 

  11. [11]

    WONG P K, YANG Zhi-xin, VONG C M, ZHONG Jian-hua. Real-time fault diagnosis for gas turbine generator systems using extreme learning machine [J]. Neurocomputing, 2014, 128: 249–257. DOI: https://doi.org/10.1016/j.neucom.2013.03.059.

    Article  Google Scholar 

  12. [12]

    DU Fang, ZHANG Jiang-she, JI Nan-nan, SHI Guang, ZHANG Chun-xia. An effective hierarchical extreme learning machine based multimodal fusion framework [J]. Neurocomputing, 2018, 322: 141–150. DOI: https://doi.org/10.1016/j.neucom.2018.09.005.

    Article  Google Scholar 

  13. [13]

    CHEN Zhi-cong, WU Li-jun, CHENG Shu-ying, LIN Pei-jie, WU Yue, LIN Wen-cheng. Intelligent fault diagnosis of photovoltaic arrays based on optimized kernel extreme learning machine and I–V characteristics [J]. Applied Energy, 2017, 204: 912–931. DOI: https://doi.org/10.1016/j.apenergy.2017.05.034.

    Article  Google Scholar 

  14. [14]

    DENG Wan-yu, ZHENG Qing-hua, CHEN Lin, XU Xue-bin. Research on extreme learning of neural networks [J]. Chinese Journal of Computers, 2010, 33(2): 279–287. DOI: https://doi.org/10.3724/SP.J.1016.2010.00279. (in Chinese)

    MathSciNet  Article  Google Scholar 

  15. [15]

    ZHANG Kai, LUO Min-xia. Outlier-robust extreme learning machine for regression problems [J]. Neurocomputing, 2015, 151: 1519–1527. DOI: https://doi.org/10.1016/j.neucom.2014.09.022.

    Article  Google Scholar 

  16. [16]

    LU Xin-jiang, MING Li, LIU Wen-bo, LI Han-xiong. Probabilistic regularized extreme learning machine for robust modeling of noise data [J]. IEEE Transactions on Cybernetics, 2018, 48(8): 2368–2377. DOI: https://doi.org/10.1109/tcyb.2017.2738060.

    Article  Google Scholar 

  17. [17]

    ZHAO Yong-ping, HU Qian-kun, XU Jian-guo, LI Bing, HUANG Gong, PAN Ying-ting. A robust extreme learning machine for modeling a small-scale turbojet engine [J]. Applied Energy, 2018, 218: 22–35. DOI: https://doi.org/10.1016/j.apenergy.2018.02.175.

    Article  Google Scholar 

  18. [18]

    HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets [J]. Neural Comput, 2006, 18(7): 1527–1554. DOI: https://doi.org/10.1162/neco.2006.18.7.1527.

    MathSciNet  Article  Google Scholar 

  19. [19]

    KASUN L L C, ZHOU Hong-ming, HUANG Guang-bin, VONG C. Representational learning with extreme learning machine for big data [J]. IEEE Intelligent System, 2013(4): 1–4.

    Google Scholar 

  20. [20]

    TANG Jie-xiong, DENG Chen-wei, HUANG Guang-bin. Extreme learning machine for multilayer perceptron [J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(4): 809–821. DOI: https://doi.org/10.1109/tnnls.2015.2424995.

    MathSciNet  Article  Google Scholar 

  21. [21]

    CHEN Liang-jun, HONEINE P, QU Hua, ZHAO Ji-hong, SUN Xia. Correntropy-based robust multilayer extreme learning machines [J]. Pattern Recognition, 2018, 84: 357–370. DOI: https://doi.org/10.1016/j.patcog.2018.07.011.

    Article  Google Scholar 

  22. [22]

    LE CUN Y, HUANG F J, BOTTOU L. Gradient-based learning applied to document recognition [C]// Proceedings of the IEEE. 1998, 86(11): 2278–2324. DOI: https://doi.org/10.1109/5.726791.

    Article  Google Scholar 

  23. [23]

    LE CUN Y, HUANG F J, BOTTOU L. Learning methods for generic object recognition with invariance to pose and lighting [C]// Conference on Computer Vision and Pattern Recognition. Washington DC: IEEE Comp Soc, 2004, 2: 97–104.

    Google Scholar 

  24. [24]

    LICHMAN M. UCI machine learning repository [D]. California, Irvine, CA, USA: School Inf Comput Sci Univ 2013. http://archive.ics.uci.edu/ml.

    Google Scholar 

  25. [25]

    WIENS T S, DALE B C, BOYCE M S. Three-way k-fold cross-validation of resource selection functions [J]. Ecological Modelling, 2008, 212(3, 4): 244–255. DOI: https://doi.org/10.1016/j.ecolmodel.2007.10.005.

    Article  Google Scholar 

  26. [26]

    SUYKENS J A K, VANDEWALLE J. Least squares support vector machine classifiers [J]. Neural Process Lett, 1999, 9(3): 293–300. DOI: https://doi.org/10.1023/A:1018628609742.

    Article  Google Scholar 

  27. [27]

    ZHOU Hong-ming, HUANG Guang-bin, LIN Zhi-ping, WANG Han, SOH Y C. Stacked extreme learning machines [J]. IEEE Transactions on Cybernetics, 2014, 45(9): 2013–2025. DOI: https://doi.org/10.1109/TCYB.2014.2363492.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Contributions

YU Tian-jun provided the concept, conducted the literature review, wrote and edited the draft of the manuscript. YAN Xue-feng edited the draft of the manuscript and provided directional guidance and financial support for the whole study.

Corresponding author

Correspondence to Xue-feng Yan 颜学峰.

Ethics declarations

YU Tian-jun and YAN Xue-feng declare that they have no conflict of interest.

Additional information

Foundation item: Project(21878081) supported by the National Natural Science Foundation of China; Project(222201917006) supported by the Fundamental Research Funds for the Central Universities, China

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yu, Tj., Yan, Xf. Robust multi-layer extreme learning machine using bias-variance tradeoff. J. Cent. South Univ. 27, 3744–3753 (2020). https://doi.org/10.1007/s11771-020-4574-9

Download citation

Key words

  • extreme learning machine
  • deep neural network
  • robustness
  • unsupervised feature learning

关键词

  • 极限学习机
  • 深度神经网络
  • 鲁棒性
  • 无监督学习