Sparse Inverse Cholesky Factorization of Dense Kernel Matrices by Greedy Conditional Selection

Source abstract

Abstract. Dense kernel matrices resulting from pairwise evaluations of a kernel function arise naturally in machine learning and statistics. Previous work in constructing sparse approximate inverse Cholesky factors of such matrices by minimizing Kullback–Leibler divergence recovers the Vecchia approximation for Gaussian processes. These methods rely only on the geometry of the evaluation points to construct the sparsity pattern. In this work, we instead construct the sparsity pattern by leveraging a greedy selection algorithm that maximizes mutual information with target points, conditional on all points previously selected. For selecting [Formula: see text] points out of [Formula: see text], the naive time complexity is [Formula: see text], but by maintaining a partial Cholesky factor we reduce this to [Formula: see text]. Furthermore, for multiple ([Formula: see text]) targets we achieve a time complexity of [Formula: see text], which is maintained in the setting of aggregated Cholesky factorization where a selected point need not condition every target. We apply the selection algorithm to image classification and recovery of sparse Cholesky factors. By minimizing Kullback–Leibler divergence, we apply the algorithm to Cholesky factorization, Gaussian process regression, and preconditioning the conjugate gradient method, improving over [Formula: see text]-nearest neighbors selection.