Kernel Density Estimators in Large Dimensions

Source abstract

Abstract. This paper studies kernel density estimation for a high-dimensional distribution [Formula: see text]. Traditional approaches have focused on the limit of a large number of data points [Formula: see text] and fixed dimension [Formula: see text]. We analyze instead the regime where both the number [Formula: see text] of data points [Formula: see text] and their dimensionality [Formula: see text] grow with [Formula: see text] fixed. Our study reveals three distinct statistical regimes for the kernel-based estimate of the density [Formula: see text] depending on the bandwidth [Formula: see text]: a classical regime for large bandwidth where the central limit theorem (CLT) holds, which is akin to the one found in traditional approaches. Below, a certain value of the bandwidth, [Formula: see text], we find that the CLT breaks down. The statistics of [Formula: see text] for a fixed [Formula: see text] drawn from [Formula: see text] is given by a heavy-tailed distribution (an alpha-stable distribution). In particular, below a value [Formula: see text], we find that [Formula: see text] is governed by extreme value statistics: only a few points in the database matter and give the dominant contribution to the density estimator. We provide a detailed analysis for high-dimensional multivariate Gaussian data. We show that the optimal bandwidth threshold based on Kullback–Leibler divergence lies in the new statistical regime identified in this paper. As known by practitioners, when decreasing the bandwidth, a Kernel-estimated density changes from a smooth curve to a collections of peaks centered on the data points. Our findings reveal that this general phenomenon is related to sharp transitions between phases characterized by different statistical properties and offer new insights for kernel density estimation in high-dimensional settings.