Total research transparency would be nice
We could learn what's scary and stop doing that
The AI Futures Project (the team behind the AI 2027 scenario) just released Plan A, their positive recommendation for how the United States should handle the advent of superintelligence. At the most basic level, Plan A is an arms control deal: the US agrees to jointly regulate frontier AI development with China so both countries can take the time they need to develop confidence in the safety of their systems without having to worry about the other one cutting corners and racing ahead.
It’s an old and simple idea, but the AIFP team attempts to work out a number of implementation details, as well as prime the reader’s imagination by playing out how demand might mount in the face of more Mythos moments and anxiety about jobs, how we could cobble together a crude stopgap deal in a rush, and how we could bootstrap from there to a mature and robust regime. Plan A is the most comprehensive vision out there for how things could go really well even if the default takeoff trajectory would have us hurtling toward god-like superintelligence in a few years1 and alignment is very difficult; I recommend checking it out.
The beating heart of the setup, the main mechanism by which the regulation in this international regulatory regime is actually decided upon, is “total research transparency.” In their words: “We’ll agree to let each other see all the AI research. Then, if we don’t like something someone is doing, we’ll talk about it and perhaps agree to ban it.” To enable this, they imagine a truly radical degree of transparency: to a first approximation, anything any AI researcher can see (e.g. training code or scaling curves or architecture designs), everyone can see; any experiment any AI researcher can run (including fine-tuning experiments), anyone can run.2
Speaking as someone who just ran a multi-month multi-stakeholder process to inform the public about the state of loss-of-control risk at AI companies, this sounds really nice.
Total research transparency would fundamentally and radically simplify the task of governing AI alignment: both figuring out what the rules should be in the first place, and then actually enforcing them.
Our scientific understanding of AI capabilities is very nascent and fast-moving, and “nascent” is a generous term for the science of AI alignment. This makes it extremely unfortunate that the norms in frontier AI development right now put third party evaluators like METR into a position of having to justify in advance why every scrap of information they would like to share publicly is tightly connected to misalignment risk. We had to go into the Frontier Risk Report having in mind the structure of the argument we wanted to make about risk so we could design a set of questions to get at pieces of that argument. Each of these questions had to have a clear and logical connection to misalignment risk that would be obvious at a glance. We could ask about specific things we were concerned about, like whether models had architectures that let them think in “neuralese” instead of English,3 but there wasn’t room to discover unknown unknowns. We also weren’t able to ask basic, open-ended questions that could have been jumping-off points for deeper investigation, like “describe your training process in detail.”4
Once we shared the questionnaire with our companies, our points of contact (who are great and who we really appreciate!) ran the answers up the chain, lawyers and comms people looked at it, and we got back a Very Official Response. And of course, all the information companies shared was subject to redaction at their discretion — this meant there was a period of months when only a tiny silo at METR could see and discuss all the information that was feeding into the Frontier Risk Report.5 This is not the ideal setup for sensemaking about a complicated and fraught subject where no one really knows what they’re talking about; when we finally got through the redactions process and were able to talk to a wider range of peers, a whole host of new considerations and framings naturally cropped up.
With total research transparency, scientists outside an AI company wouldn’t have to articulate in advance exactly what information will turn out to be relevant for their understanding of the risks posed by that company, and they’d be able to think out loud with anyone they want from day one. This would dramatically accelerate the pace of the scientific conversation, and we would have a much better hope of getting on the same page about what development practices are good and bad.
And once we have a shared understanding that some development practice is necessary or scary, it is seamless and almost trivial to verify whether companies are complying with this standard. We could just…see if they are. There would be far less burden to perfectly specify every edge case of every rule. If a company is doing something that violates the spirit of a vague principle,6 a large group of technically informed outsiders — notably including researchers at competing companies, who have perhaps the most relevant expertise and the best incentives here — would be in a position to call them out in real time.
Beyond the technical problem of alignment, these AI systems are going to be the fundamental infrastructure our society relies on, they’ll be making millions of consequential decisions every day, with more and more discretion. It is in the public interest to know how they were trained — for example, total transparency would make it much harder for the AI company to insert secret loyalties into AI systems that governments and militaries may depend on.
Total research transparency is, of course, a very tall order. It would require strong-arming the world’s most powerful companies into essentially wiping out their ludicrously valuable IP, not to mention require the US to willingly give up a huge chunk of its geopolitical lead against China.7 We will most likely have to work out a more limited and less flexible transparency regime built on third party auditing. That’s my day job — METR’s job is to prototype the transparency regime that could power Plan A- or B+ or whatever you want to call it. I think we can get really far; I’m especially excited about pushing hard on more embedded assessments. But I want to be clear that anything less than total research transparency would make it harder to know we can trust the AI systems that could soon have tremendous power over us.
This is a takeoff trajectory that also heavily concentrates power by default; if the super-exponential capabilities curve is that steep, then a small initial lead can very quickly blow up into an overwhelming advantage.
Model weights are secured, but they are also secured from AI researchers themselves: no one can directly access or see the model weights. Similarly, customer data and a small amount of other sensitive information is also protected from both researchers and the public.
All participating companies (OpenAI, Anthropic, Google DeepMind, and Meta) stated that they did not use such architectures.
In one memorable conversation with a friend, he remarked that it was like trying to do an operational security audit of a building, except instead of asking for blueprints of the building you just ask questions like “Are you aware of any doors that might be unlocked?”
And our first embedded red teaming exercise involved a silo of just one METR employee, who for three straight weeks maintained a poker face in his one-on-ones with his manager to avoid leaking bits about how the exercise was going.
For example, many AI companies have stated an intention not to put “too much training pressure” on their models’ chains of thought (multiple companies have also reported accidentally training on chain of thought anyway).
Another big chunk of the lead is our compute advantage, though Plan A also heavily regulates competition on this axis, which probably also results in the US advantage shrinking.
We do have precursors to this that make data auditable but not public, like FedRAMP. Or financial audits. Both were built for a specific problem, both are imperfect, but both are also vastly more effective and close-to-truth than something like the Gartner magic quadrant questionnaire or another quasi-evaluative-quasi-marketing process, which, from your description, seems like it's closer to where independent evals are today.
This would constitute a takings under the Fifth Amendment of proprietary American IP on the order of many, many trillions of dollars for the express purposes of handing it to a declared foreign adversary. That seems...impractical?