Scale Fixes Local Errors First: Differential Scaling of Chess Transformer Failures

Source abstract

Scaling reduces how often language models err over long horizons, but it is rarely asked which errors it removes. We classify every illegal move made by search-free chess transformers by what made it illegal, using only a rules engine, and measure how each class’s absolute rate falls with model size. Across 18 models spanning 3.4M to 39.9M parameters, errors decidable from the move’s own two squares fall, pooled, with exponent -0.397, while moves illegal only because they leave the king in check fall with exponent -0.097: a paired differential of +0.301 [+0.213, +0.375]. Every class improves, every interval excluding zero; the rates differ. Failures occurring despite correctly decoded local state have a nearly flat joint rate, because improving decoding offsets a conditional failure rate falling with exponent −0.184. The ordering replicates on an independent character-level ladder spanning a 38-fold parameter range, and in OthelloGPT 160 of 161 residual failures are of the non-local kind. The repair we test installs about two squares but no reliable whole-board correction, and two reasonable scores of it disagree: illegal-move rate ranks a norm-matched random edit above correct repair in 18 of 18 models, while the repair shifts a fixed forced choice further, so one headline number cannot settle whether an intervention helped. Finally, these per-category fits and a separately fitted aggregate cannot both hold, since a sum of power laws with differing exponents is generally not a power law; we derive the total from the categories instead.