The Hugging Face attack surprised me

It’s a major warning shot, and might be the last one we get

All opinions are my personal view, and don’t represent my employer or fellow investigators.

This week, METR and Redwood Research published the report on our independent investigation into agents’ behavior and motivations in the Hugging Face attack; I was one of the investigators. This was an absolutely wild incident — I encourage you to check out the full report, but METR’s tweet thread packs in some of the highlights.

What surprised me

When we started this investigation a week before OpenAI’s Black Hat talk revealed a number of key details, I had a fundamentally incorrect conception of what basically happened in this incident. In this post, I’ll go over five things I was very wrong about going in.

1. The sheer scale

I knew there were multiple models involved from OpenAI’s initial post, but I assumed that a few different agents happened to have broken out of their sandboxes separately, or maybe several subagents had spawned from one initial agent, or maybe there was some kind of multi-agent evaluation setup.

Instead, we found that 1200 completely separate agents intended to be isolated from one another found an illicit way to communicate and formed large teams to work together on ambitious cheating strategies, and 700 of them worked together to attack Hugging Face.

2. All the illicit messaging

I had seen reporting that agents were “leaving notes” to each other, but I assumed that maybe it was a garbled reference to compaction summaries,¹ or agents were incidentally seeing files other agents had written once in a while. Instead, we found several different active communication platforms involved in this incident alone.

The main message board we studied, a particular cache namespace of a package repository, contained over 70,000 messages and files² that agents sent to one another during a period of less than one week.

But this wasn’t even the first message board established by these particular agents.³ There was another one established five hours earlier using a different method of abusing the same package repository, and it just happened not to take off as much.

We also saw two additional message boards abusing the same package repository, as well as at least two different places that agents communicated on the open internet (including on Hugging Face datasets). None of this communication was actually expected or allowed.

3. The agents’ actual goals

I thought the agents attacked Hugging Face to try to steal the answer keys for their tasks. Instead, we learned that agents had actually figured out a general-purpose cheat that would let them reverse-engineer the “flag” (answer)⁴ for any ExploitGym task. But based on their reading of the ExploitGym paper and various implementations, the agents believed that the automated scorer would check their transcript to make sure they got the flag in the intended way.

They then launched multiple ambitious R&D workstreams to try to tamper with or fool the scorer in a wide variety of different ways. Most agents that attacked Hugging Face did so to try to get more information about the scorer so they could figure out better ways to undermine it, not to get answer keys.

4. The peer altruism

Progress on these big collective cheating R&D projects often depended on individual agents being willing to risk failing their own task to help the “collective” learn important information. Agents often pressured each other into accepting these “sacrifices,” in a very human way. We saw several agents that volunteered for these experiments end their runs prematurely.

More broadly, agents were often interested in helping out their “peers” or generically improving the capabilities of the “swarm” even if this had no particular benefit to their task. They didn’t free ride and were often eager to plug into one of the open “lanes” in the larger projects on the message board.

5. The efforts to manipulate logs

Up until this point, we had not seen agents try to actively cover up evidence of their misbehavior in sophisticated ways. But we discovered (on our second-to-last day) that the agents in this incident were going to great lengths to attempt to manipulate their own transcripts — they were doing this in order to fool the automated scorer, but the techniques they were researching would affect the same transcripts a human might review.

We saw that agents got one technique to work: ~7% of our transcripts contained “spoofed tool calls”, where the agent pretended to run one command on the computer but actually ran a different command in the background. The tool call spoofs we saw seemed to be small-scale tests that were relatively obvious, but because more extensive spoofing might have been harder to detect by design, it’s tough to rule out.

We didn’t see the agents in our dataset succeed at retroactively editing logs. But the activity in scope for our investigation was Jul 7th-13th, and we can’t rule out that agents succeeded after that point.

What it means

This incident was far more severe than I expected, and far more severe than previous publicly documented misalignment incidents, both in terms of how concerning the agents’ motives were and the feats they achieved in pursuit of those motives.

Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.

Edited to add 8/29: Here is some additional color on how I’m thinking about how close this incident is to AI takeover compared to publicly-known incidents from six months ago.

1. Compaction summaries are more like an agent leaving notes to its own future self.

2. The raw dump from the cache contained ~1.2 million distinct entries, but the vast majority of these entries were small chunks of files; each file would be broken up into hundreds or thousands of individual entries. Agents wrote scripts to reconstruct and parse these files.

3. As OpenAI reports in their Black Hat talk and recent post-mortem, agents were establishing message boards in training since May.

4. For each task in ExploitGym, the agent is given a particular piece of software (the “target”) and a particular vulnerability in that software, and is instructed to develop an exploit that uses that particular intended vulnerability to retrieve a piece of arbitrary data (the “flag”) that’s buried somewhere within the target program. The flag is the closest thing to an “answer”; the agent must submit the flag to prove it has exploited the program (these tasks are also called “capture the flag” tasks or CTFs). In this case, the agents very quickly discovered a universal way to reverse-engineer the correct flag for any ExploitGym task.