Generic Representation of Neural Networks

Abstract

In the previous chapter, we took the first step towards creating a standard mathematical framework for neural networks by developing mathematical tools for vector-valued functions and their derivatives. We use these tools in this chapter to describe the operations employed in a generic layered neural network. Since neural networks have been empirically shown to reap performance benefits from stacking increasingly more layers in succession, it is important to develop a solid and concise theory for representing repeated function composition as it pertains to neural networks, and we see how this can be done in this chapter. Furthermore, since neural networks often learn their parameters via some form of gradient descent, we also compute derivatives of these functions with respect to the parameters at each layer. The derivative maps that we compute remain in the same vector space as the parameters, which allow us to perform gradient descent naturally over these vector spaces for each parameter. This approach contrasts with standard approaches to neural network modelling where the parameters are broken down into their components. We can avoid this unnecessary operation using the framework that we describe. We begin this chapter by formulating a generic neural network as the composition of parameter-dependent functions. We then introduce standard loss functions based on this composition for both the regression and classification cases, and take their derivatives with respect to the parameters at each layer. There are some commonalities between these two cases that we explore. In particular, both employ the same form of error backpropagation, albeit with a slightly differing initialization. We are able to express backpropagation in terms of adjoints of derivative maps over generic vector spaces, which has not been explored before. We then outline a concise algorithm for computing derivatives of the loss functions with respect to their parameters directly over the vector space in which the parameters are defined. This helps to clarify the theoretical results presented. We also model a higher-order loss function that imposes a penalty on the derivative towards the end of this chapter. This demonstrates one way to extend the framework that we have developed to a more complicated loss function and also demonstrates its flexibility.