The injectivity threshold of a Gaussian ReLU layer

Source abstract

We determine how wide a randomly initialized ReLU layer must be to distinguish every pair of inputs. For independent Gaussian weights and zero bias, as the dimensions grow at a fixed output-to-input ratio, the probability of global injectivity tends to zero below a sharp threshold and to one above it. We characterize this threshold exactly through a variational formula for the limiting minimum number of active neurons per input coordinate. The formula accounts for correlations among input directions at arbitrarily many levels. We prove this formula rigorously, establishing a prediction from the statistical physics of the spherical perceptron. The proof compares systems of nearby dimensions and controls a Gaussian interpolation by calibrating responses to auxiliary fields. A uniform volume estimate connects the soft minimum to the worst input direction. An exact-arithmetic calculation places the threshold below seven output neurons per input coordinate. Numerical evaluations suggest a value near 6.698.