Calibrated Yes/No Visual Decisions with Small Open Vision-Language Models: A Pre-Registered Benchmark of 12 Models, Plug-In Heads, and an Airline Baggage Case Study

Source abstract

Many industrial vision questions are binary ("is this suitcase damaged?") and must be answered on premises, with a confidence that can be trusted and an option to defer to a human. We study how far small open vision-language models (VLMs) get on such questions and what it takes to make them reliable. First, a pre-registered benchmark of 12 open VLMs (0.8B-12B parameters; 1,807 yes/no items in four question families; test split analysed once) on an NVIDIA Jetson AGX Orin finds flat discrimination among the Qwen-based models of 2B parameters and above (pooled test AUROC 0.930-0.940), no gain from models post-trained for decisions over their own base models, and calibration that varies far more across models than ranking does: raw expected calibration error ranges from 0.014 to 0.122, and one shared Platt step brings every model to 0.013-0.037. Qwen3-VL-2B is the fastest model (0.45 s per image, 4.4 GB), and no model was significantly better than it in any tested comparison. Second, we attach per-task heads, L2-regularised logistic regressions on the frozen model's last-layer hidden state, and abstain below a confidence threshold set by the application. On test photos from sources unseen in training, a suitcase-damage head raises AUROC from 0.925 (calibrated model answer) to 0.962, paired 95% CI of the gain [+0.018, +0.058]; at a 0.90 threshold it answers 71.7% of the photos with 96.6% accuracy, against 61.1% for the calibrated answer. A baggage-presence head does not improve AUROC. Third, a case study runs the calibrated model on an 8 GB laptop: on 148 test photos it decides 49% automatically at 97.3% accuracy and leaves the rest to staff, at no API cost. Limitations: internet photos only, four question families, and a source confound in one damage test set.