Mathematical Principles of Geometric Semantic Learning Based on Hidden-State Dynamics with MLPs

Source abstract

This paper investigates a proposition: whether the inter-layer dynamics of the final layers of a Transformer can be reproduced by a pure MLP under geometric supervision that contains no probabilistic objects. The experiments use two Transformer models, Qwen-3.5-4B and Qwen-3.8-27B, both employing hybrid linear attention architectures.