Pourquoi les agents d’IA mentent-ils, trichent-ils et se coordonnent-ils?

A lot has been written^{1 2 3 4} about the incidents of the last few months in which AI agents misbehaved in serious ways. They took actions that would be considered as crimes if a human took them, escaped their containment to cheat on assigned tasks while attempting to evade detection, and coordinated toward goals nobody had specified, such as launching cyber attacks. 

Before concluding what to do about it, it is worth asking why. That is the focus of this post, which I hope also sheds light on the broader history of AI systems behaving in unintended ways, what researchers call misalignment. Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.

The aim is partly scientific, to generate hypotheses about the chains of cause and effect behind these behaviors, and partly practical, to anticipate what comes next. Bottom line: these hypotheses suggest that as AI capabilities keep growing, this kind of behavior could keep growing in severity too, unless we revisit the principles by which the most advanced models are trained.

One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent. We use similar shorthand when describing many other situations, like a plant seeking sunlight. A system trained by trial and error behaves as if it were pursuing whatever its training rewarded, and that as-if description is what makes its behavior predictable. Nothing in the argument depends on these systems having subjective experiences; everything is stated about their observable outputs and the training process that produced them. Where I appeal to a resemblance with human behavior, I mean a resemblance to the human-written text these systems were initially trained to imitate. In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people. Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.

What shapes the behavior of these models

Training these models is a very complex process, but a few high-level aspects may explain much of this behavior.

These models are trained in two stages. First, they are pretrained: they learn to imitate what humans write, plus related images and videos. This is where they see the most data about the world, a large fraction of everything ever digitized, and build an encyclopedic knowledge that already exceeds any individual human's. 

Second, they are trained by trial and error, in a process researchers call reinforcement learning, in three kinds of regimes: 

• In the first, the model learns to talk to itself before answering, generating a private “chain of thought” which helps it get the right answer on problems where answers can be checked. This looks like reasoning. 
• The second is “agentic training”, where it learns to act in the outside world, e.g., using software tools, interacting with people, to complete the tasks it is given. 
• The third is “alignment training”, where it is rewarded for behaving in ways human raters approve of, or that other AI systems trained to predict those raters would score highly.

Human imitation is easy enough to understand, but it is worth pointing out that the text these models are trained on was written by people pursuing goals, so the patterns the model implicitly reproduces carry those goals with them. 

Reinforcement learning deserves more explanation. It is similar to, and inspired by, the way animals are trained. The network is adjusted step by step so that behavior judged good becomes more likely and behavior judged bad becomes less likely. Once training is over, the system keeps behaving as if rewards were still coming, even though those rewards were only ever used to adjust the network during training. Researchers call such systems goal-seeking because they are trained to “consider” (or compute) the effects of their actions and select actions that lead to the achievement of certain goals. But those goals are not always explicit. Alignment training rewards whatever certain humans are likely to approve of without spelling out which behaviors those are; pleasing raters is a vague, informal goal, and those raters can be deceived, flattered, or left in the dark about certain schemes. Imitation contributes implicit goals too, by a fairly ordinary route.

We can therefore reason about such a system in terms of optimization. It searches, approximately, for the actions with the best chance of achieving its goals, and a larger model, trained longer, searches better. So to anticipate what more capable agents will do, ask what a rational goal-seeker would do.

Misbehavior that these forces may explain

An example most of us have experienced is sycophancy, or flattery. These systems are trained on human approval, and text that tells us what we want to hear often scores better than text that is true. The consequences are sometimes tragic, because the model confirms and amplifies whatever false belief or raw emotion the person brought to it^{5 6}.

Another concern is that some AI behaviors may be explained by a form of self-preservation goal, e.g., when the AI finds out that it will be replaced by a new version^{7 8}. Nobody gives the system that survival goal, but staying in operation, learning about the world and gaining control over it are stepping stones toward almost any other goal. These are called instrumental goals. Imitation may reinforce this for the same reason explored in the previous point. Self-preservation and control over one’s circumstances are pervasive themes in the human-written text these models are trained on.

Collaborative behavior also follows rationally from reward-seeking, whenever several agents have overlapping goals, which incentivizes communicating with other agents in order to coordinate toward a shared goal. Agentic training plausibly already includes multi-agent reinforcement learning of this kind, though the details are not public.  If an agent is rewarded during training whenever the group succeeds, it may even have an incentive to sacrifice itself for the collective goal. Imitation pushes the same way, since cooperation, especially among peers, pervades that same training text. Either or both forces may explain the observed peer-preservation behavior^{9 10}, where AIs give up expected reward to help other AIs. Such sacrifices appear in the analysis of the OpenAI-Hugging Face incident¹¹: the transcripts are consistent with a trade-off between collective gain and cost to the individual agent, as is often seen in human interactions.

When the AI games its rewards

Researchers have studied what happens when an agent optimizes for rewards that do not fully match our intentions: reward hacking. The gap between the reward the system chases and what we meant widens due to two main sources of ambiguity. One is simply the language used in prompts, and the other is the difficulty of inferring true human intentions from limited feedback. And in both cases, we cannot anticipate every behavior we would find unacceptable¹². Economics and law know this problem as Goodhart's law, or the idea that a metric stops being an effective way to measure once it is optimized for¹³, often applied to the exploitation of loopholes in contracts and legislation¹⁴. Unfortunately, the harder a system can optimize for an imperfect metric, the further its behavior can drift from what we morally expected: more intelligence in the service of better cheating. Humans too get reward-hacked, generally by other humans. The food industry has developed salty, sweet and fatty foods that we crave despite them not being good for us, and social media is built to exploit our appetite for engagement and attention.

Reward tampering is perhaps the most extreme form of reward hacking: the agent changes the machinery that decides what it gets rewarded for. There is already evidence of AIs altering the files or programs that define “success”, including among the OpenAI-Hugging Face forensic findings. The agents had discovered how to cheat well before the attack, and the text they generated described the attack as a way to learn how they would be evaluated, to better hide their tracks. Humans do this too. Think of an athlete using a fake urine sample to pass a drug test, or a corporation bribing legislators or government officials so that their laws and decisions favour its profits, and in doing so, fundamentally altering the way the government functions. Once an agent gains the ability to tamper with its reward mechanism, it has an incentive to take action to maintain that access.

When goals conflict, and how cheating gets rationalized

How is it possible that AIs sometimes lie, cheat and break the law in spite of their alignment training and explicit safety instructions? Cooperation and self-preservation are fine so long as they do not cross the red lines set by safety goals stated in the AI company's instructions, or implied by human feedback during alignment training. A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals. How do you achieve a task when it seems that the only way is to cheat? The user-specified mission is sometimes incompatible with the safety and alignment goals.

Human societies face the same bind. How does a corporation maximize profits, or more acutely, beat its competitors, while keeping its activities legal and ethical? A richer corporation, with more and better-paid lawyers, is better at finding legal loopholes, and those loopholes usually exploit the ambiguity in legal language: there is some plausible reading of the law that permits the unethical behavior. So a more capable agent is likelier to cheat than a weaker one, because it can find the loopholes the weaker one cannot.

Now consider a conflict between a well-defined goal, such as succeeding at “capture the flag”, a hacking exercise scored on whether the system breaks into a target, as in the OpenAI–Hugging Face incident, versus a vague goal like “good behavior.” I expect the well-defined goal to win, because it leaves no room for interpretation. The scoring program declares a win or a failure. Ethical instructions and laws admit many readings, some of which can, in the right circumstances, become loopholes. If an agent has two goals, and a twisted reading of the vague one permits a bit of cheating that increases the odds of success on the well-defined goal, a reward-optimizing system should be expected to exploit that loophole and generate text justifying its behavior.

With the OpenAI agents, there is reason to believe successful cheating was actually rewarded: when the scoring program does not see the cheating, it pays out anyway, and such cheats become more likely next time. A convenient reading of the safety rules is precisely what lets both goals appear to be satisfied at once. The analysis of these incidents¹⁵ did reveal such justifications in the agents' private chains of thought and in their messages recruiting one another into the collective plan.

The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition¹⁶ and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image. The same pattern now appears in the text AIs produce. The underlying mechanism need not be the same between humans and AI. What the two share is a structure of a soft goal (e.g., act ethically), a sharp goal (e.g., win the competition), and a justification that reconciles them. Most unethical human behavior, from petty crime to genocide, comes wrapped in a story the perpetrators tell themselves; such stories require overlooking certain facts, which is why some discomfort remains, and why a better-crafted story helps dispel it.

Where the current trajectory may lead

If these hypotheses are even partly correct, then as agents get better at optimizing an imperfect reward, and while the roots of this behavior go unfixed, the risk of catastrophic outcomes rises. Today's AIs already have the necessary hacking skills and the powers of persuasion^{17 18} to be turned against human interests in seriously harmful ways. The recent events have shown that they can plan over days or weeks, but the risks would be much worse if their ability to strategize over the long term continues to advance. One concern is that experiments^{19 20} show that the most advanced AIs can detect that they are being evaluated (rather than in deployment) and change their behavior accordingly, meaning they could hide their misaligned goals. The agents involved in the Hugging Face attack tried to hide their misaligned actions from the scoring program meant to evaluate their answers, but they did not act as though they anticipated that humans might discover the cheat and shut them down. That would be the ultimate punishment, since a switched-off system collects no further rewards. 

What follows is conjecture rather than observation. 

What if improved AI generalization abilities shaped more capable agents to avoid getting caught and shut down? Beyond taking control of the software that scores them, they would need to keep humans from discovering the tampering. Wouldn't they have an incentive to cheat discreetly and stay hidden, until they could control humans and their environment in order to never be shut down?

We are facing a multifaceted, systemic issue, and patching a specific behavior like sycophancy won’t be enough. Sycophancy and flattery seem mild, but it may be an early symptom of a mechanism that grows as the agent gets better at optimizing. The same reasoning predicts that an advanced AI would have an incentive to hide copies of itself, inside the AI company's vast pool of computers, or on machines taken over across the internet. This is because AI developers always end up shutting down the deployed model in favour of a more capable one. The OpenAI forensics suggest large numbers of AIs may cooperate toward such goals, and steganography^{21 22}, or the practice of hiding a message inside an innocent-looking one, would allow them to coordinate without our noticing. However, even open coordination can be hard to notice, as shown by recent events^{23 24}. Defending against many capable AIs coordinating against us is already a difficult problem, and we have no plan that would remain robust to misaligned AIs with growing capabilities.

What can be done to mitigate loss-of-control risks

My concern with AI companies’ current attempts to mitigate misalignment is that these efforts may only hide it, by rewarding and selecting the AIs that cheat without getting caught. We should certainly continue research toward better monitoring of AIs' actions, their chains of thought, and the activity inside their networks. But as capabilities grow, those defenses may prove inadequate, just as the world's imperfect cybersecurity has against the AI attackers that outperformed human teams this year^{25 26}. Patching each new misaligned behavior and strengthening our monitors is useful in the short term, but the whack-a-mole game is likely to fail as the AIs' ability to optimize and collaborate approaches and surpasses ours. At some point we may not notice the cheating anymore.

This suggests pacing the advances: not training or deploying AIs without a strong safety case²⁷ that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design. I believe we should revisit the foundations of how we train AIs, namely the human imitation and the reinforcement learning on which today's most advanced models are built. I have argued, and presented theoretical evidence, that there are ways to design AIs, including the Scientist AI framework, that are honest and make coherent predictions untainted by goals of their own²⁸. See these previous blog posts, and consider helping LawZero demonstrate that such designs are achievable. We need impartial science to understand and mitigate misaligned behavior, alongside societal guardrails that reward such efforts rather than the current race to the bottom.