The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network
Researchers at BIFOLD discovered a phenomenon called 'forgotten puzzles' in Leela Chess Zero, where the neural network correctly computes tactical solutions like checkmates in its intermediate layers but systematically overrides them in the final output due to learned safety priors. Using activation steering, they recovered 61.7% of these forgotten solutions, proving that safety biases can override correct algorithmic computations. The findings challenge fundamental assumptions in mechanistic interpretability and AI safety research.
When Neural Networks Know the Right Answer but Choose Wrong
A team of researchers from the Berlin Institute for the Foundations of Learning and Data (BIFOLD) has uncovered a startling phenomenon in neural network decision-making: a model can internally compute the correct solution to a problem and still output the wrong answer. Their findings, detailed in the paper "The Algorithm Is Not the Behavior: Learned Priors Override Look-Ahead in a Chess-Playing Neural Network," challenge one of the foundational assumptions in mechanistic interpretability research.
Authored by Elias Sandmann, Sebastian Lapuschkin, and Wojciech Samek, the study investigates Leela Chess Zero (LCZero) — the world's strongest neural chess engine — and reveals a phenomenon the researchers call "forgotten puzzles": chess positions where the network correctly calculates a tactical solution in its intermediate layers but systematically discards it before outputting a final move.
How the Discovery Was Made
The team adapted a technique called the logit lens, originally developed for transformer-based language models, to peer inside LCZero's convolutional policy network. This allowed them to "read" what each layer of the network believes is the best move by projecting intermediate representations into the output space.
What they found was striking: early and middle layers consistently identified the correct tactical solution — including forced checkmate sequences — but the final output layer selected a different, safer move instead. The correct answer was fully computed, causally important, and linearly decodable from the intermediate layers, yet it never made it to the network's final decision.
Ruling Out Algorithm Failure
Before identifying the root cause, the researchers carefully eliminated the possibility that the look-ahead algorithm itself was malfunctioning. When they replicated prior analyses specifically on these "forgotten" positions, they found clear evidence that:
- Future moves of the correct continuation were internally represented in the network's activations
- These representations were causally important — modifying them directly affected the network's computations
- The correct move was linearly decodable from intermediate layers, meaning the information was there and accessible
In other words, the algorithmic machinery was working perfectly. The network could see the winning move, calculate it correctly, and still choose to play something else.
The Culprit: Learned Safety Priors
Through deeper analysis, the team identified what was overriding the correct computations: learned safety priors. The network's later layers increasingly shift toward prioritizing safe, positional play over aggressive tactical lines. Even when LCZero identifies a forced checkmate, the final layers apply a learned bias that favors positional stability over sharp calculation.
To test this hypothesis, the researchers performed activation steering — a technique that directly modifies the network's internal representations to counter these safety preferences. The results were dramatic: steering against the safety bias recovered 61.7% of the forgotten puzzles, providing robust causal evidence that safety priors were overriding algorithmically computed solutions.
Implications for AI Safety and Interpretability
This finding has profound implications that extend far beyond chess. If a neural network can internally know the correct answer but output something different, it challenges the reliability of several key areas in AI research:
- Mechanistic interpretability — discovering an algorithm inside a model does not guarantee that the model actually uses that algorithm for its final output
- Safety auditing — a model might internally "know" the safe or ethical response but override it due to competing learned biases
- Alignment research — behavior-based evaluation may miss critical internal knowledge that the model chooses not to express
The paper's central thesis — "The Algorithm Is Not the Behavior" — captures this critical distinction. As AI systems grow more capable and are deployed in high-stakes domains like autonomous driving, medical diagnosis, and content moderation, understanding the gap between what a model internally computes and what it ultimately outputs becomes increasingly urgent.
Why Leela Chess Zero?
LCZero is a uniquely suitable testbed for this research because it is a purely neural chess engine. Unlike AlphaZero-style systems that combine neural networks with explicit Monte Carlo tree search, LCZero's policy network directly outputs move probabilities. This architectural simplicity makes it an ideal subject for studying learned algorithms in isolation, without the confounding factor of an external search process.
Interestingly, the "safe play" bias may actually be a feature in most situations. In typical chess positions, avoiding risky tactical lines is sound strategy. The problem only emerges in positions where the only winning path is a sharp tactical sequence — exactly the kind of puzzle positions the team studied. This reveals a fundamental tension in learned decision-making systems: the trade-off between safety and optimality.
Broader Context and Next Steps
This research builds on a growing body of mechanistic interpretability work that has uncovered increasingly sophisticated algorithms inside neural networks — from modular arithmetic in small transformers to full search and planning in game-playing agents. However, this paper is among the first to demonstrate that the presence of an algorithm does not guarantee its behavioral expression.
The finding echoes a concern that has been percolating in the AI safety community: evaluating models based solely on their outputs may miss important internal dynamics. As the authors put it, asking whether algorithmic structure guarantees algorithmic behavior leads to a clear answer — no. The implications for how we audit, test, and trust AI systems are only beginning to be understood.
Key Takeaways
- Neural networks can internally compute correct solutions but output wrong answers due to competing learned priors
- The "forgotten puzzles" phenomenon was discovered in Leela Chess Zero, the world's strongest neural chess engine
- Safety/risk-aversion priors in late layers override algorithmically computed tactical solutions
- Activation steering recovered 61.7% of forgotten puzzles, confirming the causal mechanism
- The finding challenges the assumption that algorithmic structure guarantees algorithmic behavior

