From Draft-Then-Verify to Systems Co-Design: A Survey of Speculative Decoding for Autoregressive Large Language Models

Source abstract

Speculative decoding accelerates autoregressive large language model inference by producing low-cost draft work and committing accepted portions after parallel target-side verification. This draft-then-verify process defines the broad family, while recent methods differ in where draft work comes from, how candidate units are logically organized across drafting and verification, what fidelity claim licenses commitment, and how runtime systems realize committed state, control speculation, execute serving paths, and evaluate end-to-end performance. We therefore develop a systems co-design taxonomy with four interdependent aspects---Draft Source, Draft Geometry, Verification Fidelity, and Runtime Execution---and 19 categories. Applying it to 304 works yields Method Profiles that trace individual works through the design space and help researchers navigate the literature, compare methods consistently, and select category-matched baselines. We synthesize recurring combinations and interactions among the four aspects into Co-Design Gates---Representability, Verifiability, State Realizability, and Profitability---that organize open challenges and future directions. Together, the taxonomy, Method Profiles, and Gates frame speculative decoding as a systems co-design problem in which these choices must align to deliver the stated fidelity and end-to-end benefit.