Value Matrix Dynamics and Attractor Selection in a Reduced Transformer Model

Source abstract

We study an interacting particle system on the unit sphere Sd−1 motivated by the forward-pass dynamics of a transformer model that combines a value matrix V and a u-aligned multi-layer perceptron (MLP) Φ, a particular form of MLP specified below. For a symmetric value matrix and this MLP, the one-cluster dynamic is the Riemannian gradient flow of an explicit Lyapunov function L=fV+g, which combines the Rayleigh quotient of V with the MLP potential. Every trajectory converges to a fixed point, limit cycles are excluded, and the fixed points are the critical points of L: local maxima are stable attractors and saddles are unstable. Isolating the value matrix (Φ≡0, with V possibly non-symmetric and a real strictly dominant eigenvalue λ1), we show that the dominant eigenvector v1 is the selected attractor and that its basin is almost the whole sphere, the two basins of ±v1 being separated by an invariant subsphere carried by the left eigenvector of V. The subdominant eigenvectors form a hierarchical structure of saddles, and a complex dominant pair replaces the point attractor by a great-circle limit cycle. When the tokens move independently, softmax attention contributes no first-order clustering force: the tokens collapse to ±v1 at a consensus rate equal to λ1 and independent of the attention matrix, and whether two clusters in opposite basins coexist or merge is governed by the single order parameter s=v1⊤Av1. Thus, the value matrix, rather than the attention, controls the location and the selection of the limiting cluster. All results are confirmed by numerical integration of the continuous-time ordinary differential equation (ODE).