Model Adequacy Testing

Overview

For a ‘correctly specified’ neural network model the non-parametric residuals

$$ {{y}_{i}} - g(x;{{\hat{w}}_{n}}) = {{e}_{i}}i = 1,2,...,n $$

are such that ${{e}_{i}}\tilde{ = }{{\varepsilon }_{i}} = {{y}_{i}} - \phi ({{x}_{i}}).$The residuals {e _i } can be used to perform meaningful diagnostic tests about the initial assumptions regarding the error term ε in equation (1.7). However, because of the non-parametric nature of neural networks, satisfying those tests is a ‘necessary but not sufficient’ condition for model adequacy. Let us suppose that we are considering a hierarchy of nested ANN classes S ₁⊂S ₂ ⊂…⊂S _k …, where S _λ is the class comprising all models g _λ (x; w) with the same architecture A _λ , which is uniquely determined by the number of hidden units λ, i.e. one-hidden-layer networks¹. Then, assuming that there exists a unique optimal number of hidden units λ ^*, which corresponds to the simplest class which still contains a correctly specified model, if λ<λ ^* the complexity of models comprising the class SA would not be adequate to represent ø(x), and the residuals e _i will not satisfy the tests. On the other hand, if λ≥λ the tests will be successful, since g_it (x; w) will be either ‘correctly specified’ if. λ=λ ^* or ‘over-parametrized’ ifλ<λ ^*.In the latter case, when λ≫λ ^* the residuals will be underestimated, i.e. e _i ≪ε _i (in some cases even e _i → 0 will be the case²). Clearly, satisfying the diagnostic tests does not indicate a faithful model. Furthermore, because of the sensitivity of the training algorithms to initial conditions and the existence of local minima, unsuccessful diagnostic tests do not necessarily imply thatλ<λ ^*. In the manner specified by Box and Jenkins for ARMA models (Box and Jenkins, 1978), the stage of diagnostic checking should be an integral part of the model specification procedure, but it can not replace it.