Figure 4c,d
Human completion geometry
Native sampled-completion squared semantic distance decreases with Pythia scale. A separate risk-to-alignment relation fitted on Pythia predicts OLMo and external anchors without refitting.
Loading recorded measurements…
Shared leading directions in human responses
Loading recorded comparison…
Methods and interpretation
The 512-context native-generation instrument and 384-context conditional-probability instrument are separate. Folds and checkpoints provide repeated measurements of the same models.
Procedure in the paper: Acquire the stated human completion norms, reproduce the fixed context selection and sample splits, generate native completions with fixed seeds, and separately score the conditional probability instrument and frozen prediction law.
Native sampled completions
Lower distance means closer geometry under this semantic instrument. Six model sizes, 16 folds, 512 shared contexts and 32 sampled completions per context. Each model is measured repeatedly across the 16 folds.
A fold is a fixed subset used in a split evaluation. Each of the six models is measured on 16 folds.
Conditional human comparison
This is a separate 384-context conditional probability instrument. The law fitted on Pythia is applied to OLMo and external anchors without refitting.
This conditional-probability test is separate from unrestricted sampled completions. Reliability correction adjusts for noise in the estimated human and model representations.
Code and data
v0.1.0 · 286 KB · View source on GitHub ↗
Native sampled completions on 512 contexts across six Pythia model sizes, including preparation of human features.
v0.1.0 · 266 KB · View source on GitHub ↗
The separate 384-context conditional-completion experiment, with 21 model checkpoints and the held-out risk-to-alignment prediction.
v0.1.0 · 1.4 MB · View source on GitHub ↗
Direct human geometry, calibration decomposition, random-subspace controls and crossed bootstrap intervals on 1,726 contexts.
Each standalone package includes code, shared helpers, required small inputs and reference results, with setup and commands in its README. Model weights and public datasets are obtained separately where needed.
Download complete source (v0.1.0) for all experiments and the companion website.