Abstract
Despite the great success of neural networks in recent years, they are not providing useful extrapolation. In regression tasks, the popular Rectified Linear Units do enable unbounded linear extrapolation by neural networks, but their extrapolation behaviour varies widely and is largely independent of the training data. Our goal is instead to continue the local linear trend at the margin of the training data. Here we introduce ReLEx, a regularising method composed of a set of loss terms design to achieve this goal and reduce the variance of the extrapolation. We present a ReLEx implementation for single input, single output, and single hidden layer feed-forward networks. Our results demonstrate that ReLEx has little cost in terms of standard learning, i.e. interpolation, but enables controlled univariate linear extrapolation with ReLU neural networks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The source code and more supporting material can be found at https://github.com/EnricoLope/PhD_ReLEx.
References
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control, Signals Syst. 2(4), 303–314 (1989)
Douglas, S.C., Yu, J.: Why ReLU units sometimes die: analysis of single-unit error backpropagation in neural networks. In: 52nd Asilomar Conference on Signals, Systems, and Computers, pp. 864–868. IEEE (2018)
Lake, B., Baroni, M.: Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks. PMLR. 80, 2873–2882 (2018)
Lake, B.M.: Compositional generalization through meta sequence-to-sequence learning. In: NeurIPS, pp. 9791–9801 (2019)
Leshno, M., Lin, V.Y., Pinkus, A., Schocken, S.: Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw. 6(6), 861–867 (1993)
Marcus, G.: Deep learning: a critical appraisal. CoRR abs/1801.00631 (2018)
Martius, G., Lampert, C.H.: Extrapolation and learning equations. In: ICLR, Workshop Track Proceedings (2017)
Mitchell, J., Minervini, P., Stenetorp, P., Riedel, S.: Extrapolation in NLP. In: Proceedings of the Workshop on Generalization in the Age of Deep Learning (2018)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why Should I Trust You?”: explaining the predictions of any classifier. In: ACM KDD, pp. 1135–1144 (2016)
Trask, A., Hill, F., Reed, S.E., Rae, J., Dyer, C., Blunsom, P.: Neural arithmetic logic units. In: NeurIPS, pp. 8035–8044 (2018)
Weyde, T., Kopparti, R.M.: Feed-forward neural networks need inductive bias to learn equality relations. In: NeurIPS Workshop on Relational Representation Learning (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Lopedoto, E., Weyde, T. (2020). ReLEx: Regularisation for Linear Extrapolation in Neural Networks with Rectified Linear Units. In: Bramer, M., Ellis, R. (eds) Artificial Intelligence XXXVII. SGAI 2020. Lecture Notes in Computer Science(), vol 12498. Springer, Cham. https://doi.org/10.1007/978-3-030-63799-6_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-63799-6_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-63798-9
Online ISBN: 978-3-030-63799-6
eBook Packages: Computer ScienceComputer Science (R0)