Figure 4b; SI: stability and attenuation
Predicting cross-model agreement
The cross-fitted exchangeable prediction has pooled median absolute error 0.009 across 96 defined cells. All cells are defined from checkpoint 256.
Loading recorded measurements…
Methods and interpretation
The 24 early undefined cases are retained as missing predictions, with their reasons. Checkpoints and comparisons are repeated observations.
Procedure in the paper: Use the frozen context split and model caches, estimate the six variance/covariance terms on the fit half, and evaluate the exchangeable restriction on the held-out half without refitting.
Defined predictions
96 predictions are defined. The data retain the 24 undefined A→B cases at early checkpoints as missing values; the scatter shows the defined predictions.
A context is the text preceding a prediction. A checkpoint is a saved version of a model during training.
Code and data
v0.1.0 · 81 KB · View source on GitHub ↗
Complete four-seed, ten-checkpoint next-token measurement and cross-fitted agreement prediction: both 100-context directions,120 full attempts and120,000 substream attempts, preserving every undefined case.
Each standalone package includes code, shared helpers, required small inputs and reference results, with setup and commands in its README. Model weights and public datasets are obtained separately where needed.
Download complete source (v0.1.0) for all experiments and the companion website.