Data-Centric Recursive Improvement for Foundation Models: A Survey

Source abstract

Data-centric methods are central to foundation model development, but data construction and evaluation have often been treated as separate stages. Recent model-development pipelines increasingly couple the two: evaluation results expose capability gaps and failure modes, which then guide data selection, filtering, synthesis, and post-training interventions that shape the behavior of subsequent model versions. This survey studies this trend as data-centric recursive improvement, with data-evaluation co-evolution as its core mechanism. In this mechanism, evaluation produces diagnostic signals, orchestration determines whether and how these signals should be trusted and used, and execution updates mutable data-related objects. These updated objects are then used in later training, adaptation, retrieval, or memory stages, enabling the system to change in subsequent rounds. The framework centers on three questions: what signal diagnoses the current system, who decides how to act on that signal, and what object is updated to affect the next round. We organize the literature around this signal-decision-update loop, covering feedback signals, control decisions, and data updates across major stages of foundation model development. We further examine failure modes in closed-loop improvement, such as unreliable feedback, feedback overfitting, unstable updates, erosion of data support, and irreproducible system changes. By providing a structured framework for understanding and designing data-evaluation co-evolution, this survey outlines a roadmap toward more adaptive, robust, and accountable foundation model development, where evaluation becomes an active driver of recursive model improvement rather than a passive measure of progress and ultimately sheds light on possible pathways toward Artificial Super Intelligence (ASI), in which a model continually evolve and surpass human-level performance across diverse tasks.