Figure 6; SI: acquisition
Forecasting factual acquisition
An ex-ante n-gram predictor explains fresh margin trajectories at three model sizes without refitting. A separate common 300-fact census is more closely aligned by loss than by training step in the quantified 70M-to-160M comparison.
Predicting a preference before measuring it.
Corpus word-sequence statistics forecast the model’s preference for one continuation over another. The forecast is fixed before evaluation on these fresh examples.
Loading forecasts…
The same learning, seen in two coordinates.
Follow a shared set of 300 continuation preferences across three model sizes. Change the axis to compare training time with held-out predictive loss.
Comparing learning schedules
Loading recorded comparison…
Methods and interpretation
Forecast batteries differ between 70M/160M and 410M. The census uses a shared 300-fact set. Persistent acquisition requires all later recorded checkpoints to remain above threshold; unacquired cases are censored.
Procedure in the paper: Construct corpus n-gram margins, retain the frozen calibration and fresh fact batteries, extract checkpoint margins and held-out loss, and apply the exact persistent-threshold and censoring analysis.
Code and data
v0.1.0 · 119 KB · View source on GitHub ↗
The two-size Pythia-70M/160M, nine-run, 20-checkpoint margin harvest and four-component acquisition-schedule analysis, conditional on the original fixed discovery calibration and scalar loss coordinates.
v0.1.0 · 376 KB · View source on GitHub ↗
Pythia-410M, nine-run, 20-checkpoint measurement on the same 300-fact battery, frozen-discovery forecast, original crossed bootstrap and non-rescuing seed4 sensitivity, plus census projection on fixed loss coordinates.
v0.1.0 · 100 KB · View source on GitHub ↗
The fixed 180-fact battery, 400 model checkpoints and zero-refit acquisition predictions, including timing, class-separation and permutation controls.
v0.1.0 · 146 KB · View source on GitHub ↗
Continuous score/chord bounds, induction motion and cross-seed discrete flip curves.
Each standalone package includes code, shared helpers, required small inputs and reference results, with setup and commands in its README. Model weights and public datasets are obtained separately where needed.
Download complete source (v0.1.0) for all experiments and the companion website.