From activations to predictions
A language model assigns probabilities to possible next tokens. This small example has three: token A, token B and token C. Given the prompt “The user hasn’t given permission. I should …”, these could represent “ask”, “wait” and “proceed”, respectively. The aim is to make token A more likely by changing the model’s internal activations while keeping the network itself fixed.
The cube represents activation space: each point gives the values of three neurons. The network maps this internal state to a prediction, represented by a point on the spherical surface. The two spaces show the same intervention from these different perspectives.
The sphere’s coordinates are the square roots of the token probabilities. Because the probabilities sum to one, these points lie on a sphere. Each corner corresponds to certainty about one token. Distances along this surface represent Fisher–Rao distances, up to a constant scale.
What the metric measures
A metric assigns lengths and angles to small changes at each point. The Euclidean metric uses ordinary distance between activation values.
A small change in activations produces a corresponding change in the model’s predictions. The pullback measures that prediction change using the Fisher–Rao metric, then assigns the resulting length to the original activation change. Two equally sized activation changes can therefore have different geometric lengths if they affect predictions differently. Directions that leave predictions unchanged to first order have zero length in this geometry.
The effect of an activation change depends on the model’s current state, so this geometry changes from point to point. A straight path in prediction space can therefore correspond to a curved path through activations.
Why the paths differ
Both methods start from the same state and make token A equally likely at each stage. Fisher–Rao steering preserves the balance between tokens B and C as their total probability falls. Euclidean steering also changes that balance, introducing an additional change beyond the target.
In this example, the Fisher–Rao path follows a great-circle arc on the sphere, like a shortest route on a globe. Its path through activation space curves because the network’s mapping is nonlinear. The Euclidean path looks straighter in activation space but bends in prediction geometry. The relevant comparison is how much the predictions change for the same target gain.
Connection to the paper
The example illustrates how geometry can guide an intervention by its effect on predictions. Its exact preservation of the alternatives’ balance belongs to this simple three-token setting. The paper tests how well the same principle predicts and reduces unwanted output changes in larger language models, including along extended paths and for interventions reused across prompts.