Table 1; Extended Data 7
Low-rank adaptation and preservation
On the held-out 70M study, tuned AdamW with explicit KL regularisation has 1.57 times the preservation-frontier area of natural-gradient LoRA.
Loading recorded measurements…
Matched-gain optimizer comparison
Loading recorded comparison…
A factorised metric can trade off training and held-out cost
Loading recorded comparison…
Approximating the metric with token groups
Loading recorded comparison…
Refining groups by value and direction
Loading recorded comparison…
Methods and interpretation
Rank four, three target gains, eight paired seeds and the stated prompt set. Preservation is measured by next-token KL on the held-out prompts. The supporting 410M sweep has a different comparator.
Procedure in the paper: Reproduce the fixed training and preservation prompts, rank-four adapters, paired seeds, calibration-only tuning and matched-gain attainment. Evaluate the preservation frontier from next-token distributions on the fixed held-out prompts.
Matched training gain
Eight paired seeds, two methods and three targets. Each connected line is one seed and method. Preservation measures next-token distributions on the fixed held-out prompts.
A seed fixes a repeat of the stochastic training procedure. Preservation is measured on the specified held-out prompts.
Code and data
v0.1.0 · 75 KB · View source on GitHub ↗
The primary Pythia-70M preservation frontier: natural-gradient LoRA and KL-regularised AdamW across eight paired seeds.
v0.1.0 · 68 KB · View source on GitHub ↗
Pythia-410M Adam, both-factor K-FAC and coarse-metric adaptation comparisons.
Each standalone package includes code, shared helpers, required small inputs and reference results, with setup and commands in its README. Model weights and public datasets are obtained separately where needed.
Download complete source (v0.1.0) for all experiments and the companion website.