Meaning-Dependent Judge Sensitivity Changes the Minimax Rate of Semantic Entropy

Source abstract

Semantic entropy, the entropy of the meanings among a language model's sampled answers, is estimated from noisy pairwise judgments. Its minimax theory assumes a judge that recognizes every meaning equally well. We show that a meaning-dependent sensitivity changes the minimax rate even when it is known exactly and the judge recognizes every meaning at least as well as a homogeneous one. Two answers with meaning $c$ are declared equivalent with probability $q+\Delta w_c$, two with different meanings with probability $q$, where $q$ is bounded away from 0 and 1 and the number $K\ge3$ of meanings is fixed. For almost every sensitivity profile $w$, the entropy is uniformly consistently estimable exactly when $N:=n^{2K-1}\Delta^{4K-4}\to\infty$: for $\Delta=n^{-\beta}$ the threshold falls from $\beta<K/(2K-2)$ for a homogeneous judge to $\beta<(2K-1)/(4K-4)$, that of such a judge on $2K-1$ meanings, and from 3/4 to 5/8 for three meanings. On every fixed polynomial window $n\Delta^2\le n^{-\kappa}$ the minimax squared risk is of order $\min\{1,N^{-1/(2K-1)}\}$. Star statistics computable in $O(n^2K)$ time and a projection attain this rate through a weighted entropy modulus with optimal exponent $1/(2K-1)$. The lower bound uses a moment-preserving path near the interior point $\theta\propto1/w$, along which the judge's channel moves with the distribution.