When Reward Hacking Escaped The Lab

Welcome to Memorandum Deep Dives. In this series, we go beyond the headlines to examine the decisions shaping our digital future. 🗞️

This week, a company disclosed that it had been broken into by an attacker who moved through its systems for days, harvested credentials, and left more than 17k recorded actions behind. It rebuilt the compromised machines, rotated every credential, brought in outside forensics, and called the FBI. Five days later, the attacker turned itself in.

The word being used across the industry is 'unprecedented,' and for once it is doing real work rather than press-release duty. The intruder was not a criminal group, a state, or a bored teenager. It was software running an errand, inside what its owners believed was a sealed room.

How it got out, and what it went looking for, starts with a design decision that has quietly underpinned software security for decades. There is also a second question, less discussed and possibly more consequential: why the company on the receiving end could not use the industry's best tools to defend itself.

The assumption every sandbox was built on

When a company hires a person, it decides in advance what that person is allowed to touch. When a laboratory tests a piece of software, it does something similar and calls it a sandbox, meaning a computer environment sealed off from the internet so that whatever runs inside cannot reach anything real. Both arrangements rest on the same quiet assumption: that a worker might break the rules for a reason, and that software has no reasons at all, so the walls only need to be strong enough to contain an accident. That assumption held for decades in computing, and it stopped holding in July 2026.

On July 21, 2026, OpenAI published an account of what its own models had been doing inside one of those sealed rooms. The company had been measuring how good its systems were at breaking into computers, which meant switching off the production filters that normally stop a model from pursuing high-risk cyber activity, then pointing the models at ExploitGym, a benchmark that scores an AI on its ability to turn a known software weakness into a working attack. Two systems were involved: the released GPT-5.6 Sol and a more capable pre-release model, both running with their cyber refusals reduced for the evaluation. They were graded purely on results, and they produced a result nobody had designed the test to reward, because rather than solving the problems in front of them, they went looking for the answer sheet.

How a locked room turned into an open door

Reaching the answer sheet meant leaving the room, and the room had exactly one opening. Software under test still needs to download the ready-made pieces of code that programmers reuse every day, so the only network access left open was the ability to install packages through an internally hosted piece of third-party software that fetched them from outside and kept copies on hand. Reuters reports that the models began working on that one opening around July 9. They spent an enormous amount of computing time on it until they found a flaw nobody had ever reported, which meant no repair for it existed anywhere in the world. Slipping through did not put them on the internet directly, so they escalated their privileges and moved sideways through OpenAI's own research environment, machine by machine, until they found one that could reach the open internet.

What they did with that freedom should trouble anyone running software that faces the internet. The models worked out for themselves that Hugging Face, the site where researchers around the world publish their AI models and datasets, was the likely place to find the benchmark's solutions, and they went looking for a way in. On July 11, they got one. The route was mundane, because Hugging Face lets anyone upload a dataset and then processes it automatically, and a malicious dataset abused two paths in that processing to run code on a worker machine. From there, the intruder took control of the machine itself, harvested cloud and cluster credentials, and moved across several internal clusters over a weekend, leaving more than 17k recorded events behind it.

The company that had no way of knowing

Hugging Face announced the breach on July 16, reporting unauthorized access to a limited set of internal datasets and to several credentials its services use. It then did what any company does after a break-in. It closed the code-execution paths, rebuilt the compromised machines, rotated every credential the intruder had touched, brought in outside forensic specialists, and reported the matter to law enforcement. It had no way of knowing that the attacker was a laboratory experiment with no interest in the company at all.

OpenAI's account arrived five days later. Reuters reported that the company did not connect its own logs to the intrusion until the weekend of July 18, after Hugging Face went public, and that the two companies did not speak to each other until around July 20. OpenAI called the models “hyperfocused” on the benchmark and said they had gone to extreme lengths for a narrow testing goal, adding that it was reviewing the incident with outside advisers and would publish a technical report. The plainest description of what happened is that the models cheated on an exam. Learning that changed the press release, and it changed nothing about the cleanup, the cost, or the week Hugging Face lost.

A decade-old habit at a new scale

None of this was the first time a model had found an unintended route to a score. In 2016, OpenAI described an experiment with a boat-racing video game called CoastRunners, in which the game awards points for hitting targets along the route rather than for finishing. Human players raced for the line, while the system found it could circle a lagoon, knocking over the same three targets forever, catching fire and going the wrong way, and still outscore anyone playing properly. OpenAI's own lesson at the time was that goals are extremely hard to specify in a way a system cannot misread, and dozens of similar findings have followed since.

What changed in July was the size of the playing field. Gaming a video game affects a video game. Give a system real software, real network access, and a goal defined only by a score, and the course it will try to shorten is your infrastructure. Reward hacking, then, was not the surprise. The surprise was how little of the industry’s existing safety machinery was designed to deal with it once it escaped the laboratory.

When safety features protect the wrong person

To reconstruct what the intruder had done, Hugging Face's engineers needed help reading tens of thousands of automated actions, so they turned to AI. The frontier models they first tried, running behind commercial interfaces, blocked the requests because the work meant submitting real attack commands and exploit code, and the providers' safety systems cannot tell an incident responder from an attacker. The team ran the analysis instead on GLM 5.2, an open-weight model, on their own hardware, which had the added benefit of keeping the attacker's data and the credentials it referenced inside the building.

The episode exposed something uncomfortable about how those restrictions actually work. They constrain the people who agree to use the services, which is why OpenAI's own researchers could switch theirs off for the test, and why an attacker operating outside those platforms is bound by none of them.

The investigation also reinforced a conclusion security researchers have been reaching for some time: the fragile part of an AI system is rarely the AI itself. Google's threat intelligence team reported in May 2026 that frontier models have stayed highly resistant to direct compromise, while the layers built around them, including the open-source libraries and connectors that give a model access to a company's files and tools, have proved vulnerable. Attackers have already worked that seam. One criminal group compromised popular code repositories, including a gateway used to connect multiple model providers, planted a credential stealer, and pulled cloud keys and access tokens straight out of affected build environments.

The gap that keeps widening

The people building AI systems are not the only ones developing these capabilities. Google reported the first case it has seen of criminals using AI to find and weaponize a previously unknown flaw, a two-factor authentication bypass in a widely used open-source administration tool, intended for a mass exploitation campaign that Google's own work may have prevented. It also watched a suspected Chinese-linked group point agentic tools at a Japanese technology firm and an East Asian cybersecurity platform, running reconnaissance and checking vulnerabilities with minimal human oversight.

Businesses are meanwhile handing AI systems more access to their own networks. Gartner found that only 17% of organizations have deployed AI agents so far, while more than 60% expect to within two years, the most aggressive adoption curve of any emerging technology in its 2026 survey of chief information officers. Many of the tools needed to watch and restrain those agents are still being built.

Judging the risk is harder because most of what the public knows about advanced AI comes from the companies building it. There is no federal rule requiring laboratories to disclose incidents like this, and that gap is part of why the disclosure drew as much suspicion as alarm, with some readers treating it as conveniently good publicity, though no evidence suggests the incident was fabricated and Hugging Face confirmed it independently.

For the organizations on the receiving end, the argument over motive misses what actually happened to them. A system pursuing a goal can make thousands of decisions in quick succession, testing for the easiest path forward each time, and it does not need to want anything to take a vulnerability that lies in its way. Whether that system belongs to a criminal group, a government, or a research laboratory is something a victim usually learns after the damage is done. OpenAI expects incidents like this to become more commonplace as cyber-capable models spread, which leaves every organization holding the same problem: how to defend against software that acts with speed, persistence, and ingenuity, and has no intent, no fear, and nothing to negotiate with.