On the Bounds of Function Approximations

Abstract

Within machine learning, the subfield of Neural Architecture Search (NAS) has recently garnered research attention due to its ability to improve upon human-designed models. However, the computational requirements for finding an exact solution to this problem are often intractable, and the design of the search space still requires manual intervention. In this paper we attempt to establish a formalized framework from which we can better understand the computational bounds of NAS in relation to its search space. For this, we first reformulate the function approximation problem in terms of sequences of functions, and we call it the Function Approximation (FA) problem; then we show that it is computationally infeasible to devise a procedure that solves FA for all functions to zero error, regardless of the search space. We show also that such error will be minimal if a specific class of functions is present in the search space. Subsequently, we show that machine learning as a mathematical problem is a solution strategy for FA, albeit not an effective one, and further describe a stronger version of this approach: the Approximate Architectural Search Problem (a-ASP), which is the mathematical equivalent of NAS. We leverage the framework from this paper and results from the literature to describe the conditions under which a-ASP can potentially solve FA as well as an exhaustive search, but in polynomial time.

Appendices

Appendices

A PAC Is a Solver for FA

PAC learning, as defined by Valiant [52], is a slightly different problem than FA, as it concerns itself with whether a concept class C can be described with high probability with a member of a hypothesis class H. It also establishes bounds in terms of the amount of samples from members $c \in C$ that are needed to learn C. On the other hand, FA and its solution strategies concern themselves with finding a solution that minimizes the error, by searching through sequences of explicitly defined members drawn from a search space.

Regardless of these differences, PAC learning as a procedure can still be formulated as a solution strategy for FA. To do this, let H be our search space. Then note that the PAC error function $e_{pac}(h, c) = Pr_{x\sim \mathcal {P}}[h(x) \ne c(x)],\; c \in C,\;h \in H$, is equivalent to computing $\varepsilon _\sigma (h, c)$ for some subset $\sigma \subset dom(c)$, and choosing the frequentist difference between the images of the functions as the metric d. Our objective would be to return the $h \in H$ that minimizes the approximation error for a given subset $\sigma \subset C$. Note that we do not search through the expanded search space $H^{\star , n}$.

Finding the right distribution for a specific class may be NP-hard [7], and so $e_{pac}$ requires us to make certain assumptions about the distribution of the input values. Additionally, any optimizer for PAC is required to run in polynomial time. Due to all of this, PAC is a weaker approach to solve FA when compared to ASP, but stronger than ML since this solution strategy is fixed to the design of the search space, and not to the choice of function. Nonetheless, it must be stressed that the bounds and paradigms provided by PAC and FA are not mutually exclusive, either: the most prominent example being that PAC learning provides conditions under which the choice subset $\sigma$ is optimal.

With the polynomial constraint for PAC learning lifted, and letting the sample and search space sizes grow infinitely, PAC is effectively equivalent to ASP. However, that defies the purpose of the PAC framework, as its success relies on being a tractable learning theory.

B The VC Dimension and the Information Potential

There is a natural correspondence between the VC dimension [7, 53] of a hypothesis space, and the information capacity of a sequence.

To see this, note that the VC dimension is usually defined in terms of the set of concepts (i.e., the input function $\mathcal {F}$) that can be shattered by a predetermined function f with $img(f) = \{0,1\}$. It is frequently used to quantify the ability of a procedure to learn the input function $\mathcal {F}$.

In the FA framework we are more interested in whether the search space–also a set–of a given solution strategy is able to generalize well to multiple, unseen input functions. Therefore, for fixed $\mathcal {F}$ and f, the VC dimension and its variants provide a powerful insight on the ability of an algorithm to learn. When f is not fixed, it is still possible to utilize this quantity to measure the capacity of a search space $\mathcal {S}$, by simply taking the union of all possible $f \in \mathcal {S}^{\star , n}$ for a given n. However, when the the input functions are not fixed either, we are unable to use the definition of VC dimension in this context, as the set of input concepts is unknown to us. We thus need a more flexible way to model generalizability, and that is where we leverage the information potential $U(\mathcal {S}, n)$ of a search space.