Semantic Prior Smoothing for Conformal Prediction in Multiple-Choice Question Answering with API-Only Large Language Models

Source abstract

We study conformal prediction (CP) for multiple-choice question answering (MCQA) with large language models (LLMs) accessed only through application programming interfaces (APIs), where internal logits and token probabilities are unavailable. Candidate probabilities are often estimated from repeated answer samples, but finite sampling yields coarse estimates and tied nonconformity scores. We examine an extreme consequence of this discreteness, which we call efficiency collapse. It occurs when the correct option receives zero count on enough calibration examples, forcing the threshold to its maximum and producing full prediction sets. We derive the exact finite-sample condition and show that the problem stems from the empirical probability proxy rather than the split conformal rule. To address this issue, we propose Semantic Prior Smoothing (SemPrior), which uses semantic evidence from candidate rationales to allocate a nonuniform Dirichlet prior and combines it with observed answer counts. SemPrior leaves the standard nonconformity score and split conformal calibration unchanged. Experiments on four datasets show that SemPrior generally produces smaller prediction sets while keeping empirical coverage close to nominal levels. Results are also stable across the tested prior settings and calibration-set sizes.