Asymptotic standard errors for reliability coefficients in item response theory

Source abstract

Abstract In a recent review, Liu et al. ( Psychological Methods , 2025b) classified reliability coefficients into two types: classical test theory (CTT) reliability and proportional reduction in mean squared error (PRMSE). This article focuses on quantifying the sampling variability of these coefficients under item response theory (IRT) models. While some existing standard error (SE) formulas are accurate when variability arises only from item parameter estimation, the reliability estimators considered in our work involve additional variability from substituting population moments with sample moments. We propose a general strategy to derive SEs that incorporates both sources of sampling error simultaneously, enabling the estimation of model‐based reliability coefficients and their SEs in such settings. We then apply our general theory to derive SEs for two specific estimators under the graded response model: (1) CTT reliability for the expected a posteriori score of the latent variable and (2) PRMSE for the latent variable. Simulation results show that the derived SEs accurately capture the sampling variability across various test lengths in moderate to large samples. We conclude with an empirical illustration and directions for future research.