Semiparametric learning from open-set label shift data

Source abstract

Abstract We study the open-set label shift problem, where the test data may include a novel class absent from the training data. This setting is challenging because both the class proportions and the distribution of the novel class are not identifiable without extra assumptions. Existing approaches often rely on restrictive separability conditions, prior knowledge, or computationally infeasible procedures, and some may lack theoretical guarantees. We propose a semiparametric density ratio model framework that ensures identifiability while allowing overlap between novel and known classes. Within this framework, we develop maximum empirical likelihood estimators and confidence intervals for class proportions, establish their asymptotic validity, and design a stable Expectation–Maximization algorithm for computation. We also develop a goodness-of-fit test for model assessment and an approximately optimal classifier with theoretical guarantees. Simulations and a real-data application confirm that our methods improve both estimation accuracy and classification performance compared with existing approaches.