ARBITER: retrieval-augmented generation for automated PROBAST+AI risk-of-bias assessment of HIV, HBV and HCV prediction models: a proof-of-concept validation study

Source abstract

Abstract Clinical prediction models for blood-borne virus infections — HIV, hepatitis B (HBV), and hepatitis C (HCV) — are published widely, yet appraising their quality remains labour-intensive and inconsistently applied. We developed the Automated Risk-of-Bias Inspection Tool for Evidence Review (ARBITER) to characterise studies, separate model-development and model-evaluation pathways, and generate a categorical judgment with verbatim supporting evidence from manuscript text for each PROBAST+AI signalling question. We illustrate the application of ARBITER (using the Google Gemini 2.5 Pro API) to ten prediction model studies for HIV, HBV and HCV, comparing its accuracy against Claude Sonnet 4.6 (via its chat interface) and a human statistical expert; one study (Kerschberger 2021) informed pipeline refinement before assessment of the remaining nine. All three assessors achieved substantial agreement with expert consensus: ARBITER Cohen κ=0.701 (95%CI 0.650–0.754), Manual Reviewer κ=0.724 (95%CI 0.674–0.774), and Claude Sonnet 4.6 κ=0.714 (95%CI 0.617–0.794). Error analysis revealed characteristic, partly complementary patterns: ARBITER drew incorrect conclusions from correctly retrieved evidence; the manual reviewer overlooked briefly-reported detail both automated assessors captured; and Claude Sonnet 4.6 erred on the most demanding statistical questions. This reproducible, auditable approach improves through human feedback and, with further validation, could support evidence synthesis and reduce reviewer burden.