Approximation by Ridge Functions and Neural Networks

Source abstract

We investigate the efficiency of approximation by linear combinations of ridge functions in the metric of L 2 (B d ) with B d the unit ball in R d . If X n is an n-dimensional linear space of univariate functions in L 2 (I), I=[-1,1], and $\Omega$ is a subset of the unit sphere S d-1 in R d of cardinality m, then the space Y n :={span}\{r({\bf x}\cdot\xi):r\in X_n,\omega\in\Omega\}$is a linear space of ridge functions of dimension$\le mn$. We show that if X n provides order of approximation O(n -r ) for univariate functions with r derivatives in L 2 (I), and$\Omega$are properly chosen sets of cardinality O(n d-1 ), then Y n will provide approximation of order O(n -r-d/2+1/2 ) for every function$f\in L 2 (B d ) with smoothness of order r+d/2-1/2 in L 2 (B d ). Thus, the theorems we obtain show that this form of ridge approximation has the same efficiency of approximation as other more traditional methods of multivariate approximation such as polynomials, splines, or wavelets. The theorems we obtain can be applied to show that a feed-forward neural network with one hidden layer of computational nodes given by certain sigmoidal function $\sigma$ will also have this approximation efficiency. Minimal requirements are made of the sigmoidal functions and in particular our results hold for the unit-impulse function $\sigma=\chi_{\raisebox{-.6ex}{$\scriptstyle [0, \infty)$}}$.