Three Conjectures on Neural Network Implementations of Volterra Models (Mappings)

Abstract

Three conjectures are presented that form the methodological link between Volterra models (mappings) and a popular class of artificial neural networks (multi-layer perceptrons). The first conjecture elucidates the equivalence between these two types of nonlinear mapping and shows how we can achieve a network implementation of a Volterra model by employing proper linear input transformations and polynomial activation functions in the hidden units; while the output unit(s) may be simple adder(s). The second conjecture outlines the trade-offs between general polynomial and fixed sigmoidal activation functions traditionally used in multilayer perceptrons. The former are more flexible in defining nonlinear mappings; while thelatter, being far more restrictive, lead to increased numbers of hidden units and heavier computational burden during training via back-propagation. In general, an infinite number of sigmoidal hidden units is required to represent exactly a Volterra model (mapping) or a network with (finite) polynomial hidden units. The third conjecture extends the results for continous-output models to binary-(or spike-)output models/mappings, often encountered in neural networks. These conjectures collectively point to the potential versatility and efficiency of a class of networks that utilize polynomial activation functions in the hidden units and linear output unit(s) with fixed weights. Practical procedures for optimal use of these networks are currently developed.