Multilinear Compressive Sensing and an Application to Convolutional Linear Networks

Source abstract

We study a deep linear network endowed with the following structure: A matrix $X$ is obtained by multiplying $K$ matrices (called factors and corresponding to the action of the layers). The action of each layer (i.e., factor) is obtained by applying a fixed linear operator to a vector of parameters satisfying a constraint. The number of layers is not limited. Assuming that $X$ is given and factors have been estimated, the error between the product of the estimated factors and $X$ (i.e., the reconstruction error) is either the statistical or the empirical risk. We provide necessary and sufficient conditions on the network topology under which a stability property holds. The stability property requires that the error on the parameters defining the near-optimal factors scales linearly with the reconstruction error (i.e., the risk). Therefore, under these conditions on the network topology, any successful learning task leads to stably defined features that can be interpreted. In order to do so, we first evaluate how the Segre embedding and its inverse distort distances. Then we show that any deep structured linear network can be cast as a generic multilinear problem that uses the Segre embedding. This is the tensorial lifting. Using the tensorial lifting, we provide a necessary and sufficient condition for the identifiability of the factors up to a scale rearrangement. We finally provide a necessary and sufficient condition called the deep-Null Space Property (because of the analogy with the usual Null Space Property in the compressed sensing framework) which guarantees that the stability property holds. We illustrate the theory with a practical example where the deep structured linear network is a convolutional linear network. We obtain a condition on the scattering of the supports which is strong but not empty. A simple test on the network topology can be implemented to test whether the condition holds.