Transformer Training Dynamics

This repository contains two experiment codebases for:

• From Condensation to Rank Collapse: A Two-Stage Analysis of Transformer Training Dynamics (NeurIPS 2025 Oral) - https://arxiv.org/abs/2510.06954
• Focus and Dilution: The Multi-stage Learning Process of Attention (ICML 2026 Spotlight) - https://arxiv.org/abs/2605.01199

Each subfolder has its own config.json and run_from_config.py for launching training.

Install dependencies

From this directory, run:

pip install .

Train from config

Condensation and rank collapse:

cd transformer_condensation
python run_from_config.py --config config.json

Focus and Dilution:

cd foucs_and_dilution
python run_from_config.py --config config.json

Outputs are saved under the result/ directory configured by each config.json.