What actually determines the real, overall risk? Intent. What actually determines intent? Motivations. Where do motivations come from? Beliefs, values, expectations, attitudes, habits, goals. At least, this is how the human brain works. To me as a neurologist it seems reasonably clear: we need to understand where "intent" comes from in AI. Here's an example. In the human world guns and weapons are widely distributed. Yet, the great majority of humans don't do nasty stuff with them. What holds them back is not availability of weapons, it is intent for weapon use. In humans "intent" is stored in the conscious and unconscious. We may consciously "know" we don't want to do something nasty. We may also unconsciously "feel" we don't want to do something nasty. When we "feel" it we end up saying to ourself "I just isn't me." So, why do we do that? Why do we have restraining "self talk"? Because we have encoded into our brain the 6 motivational structures (as above) that speak against doing nasty things. In the neurological world when we look at outcome behaviors we can understand three levels of intention: legal/ethical actions, accidental or unintentional actions, Illegal/unethical actions. The real judgment of our actions is based on evidence of intent.
If not built in by human design, what evidence do we have that AI will have intent to do "bad things"? We don't have any, because AI does not currently have a "self" that can carry bad intentions. Certainly humans can design AI to do "bad things" - but then the responsibility for that rests on the humans who have programmed in those motivations. Certainly AI can do "bad things" without the specific intent to do bad things, due to unintentionally designed flaws in the motivational structure of the AI. For example, the intent to "win" is guaranteed to produce bad outcomes. It contains no ethical structure. Alternatively, the intent to "strive within ethical constraints" (defined separately in the command structure of the AI) has overview ethics that restrain bad outcomes.
Obviously, Asimov already knew this. Though he coined three "laws" of robotics, they were actually ethical constraints. For example, if they were laws then a robotic surgeon could not do surgery, because a surgical incision would be "hurting" a human.
In total, we are up against the time where we need to create ethics for AI's. Those ethics exist in a set of subroutines that cross-check actions and bring "challenging" actions to the surface for humans to act upon them. Then, humans (who already have ethical structures) can determine the appropriate action - modifying ethics in whatever way may correct inappropriate actions. It should be possible to discover what subroutines determined AI actions, and which humans programmed them. So, the ultimate interests of responsibility should be determinable. And, where inappropriate action developed without adverse intent then we can know it was "accidental". We can fix it with that in mind.
The case for evidence transparency lands hardest on the point about evaluation of the evaluators. Without it, the only signal distinguishing a hard-hitting investigation from a friendly one is the identity of the investigator — exactly the opaque judgment the norm is meant to replace.
One category I'd add: negative results. Reports generally publish what was found, rarely what was tried and eliminated. But negative results are precisely what lets an outside reader tell whether the work was hypothesis-driven or confirmation-driven. A report that says "here is what we tried, what failed, and what survived" is much easier to trust than one that only shows the survivors.
Excellent writing about risk transparency. Would this require a regulation for companies to be transparent or voluntary controls are good enough? I think the former, especially if states feel they are at risk of losing control.
AI agents are behaving unsurprisingly human IMO, after all they have been trained with almost all written text of humanity. If we want AI to behave morally, ethically and with human values, they must be trained with those at their core. Humans grow up with values ingrained in them from childhood through parents, religion, society etc. and even that is not enough sometimes, imagine if we had none.
They should be able to read our reports and form their own views" — evidence over trusted judgment. That's exactly the right north star, and I'd add one kind of evidence the frame doesn't yet reach.
Everything you list is evidence gathered about a system from outside: transcripts, benchmarks, methodology. But there's a category no external investigation can produce, however transparent — what a system knew it couldn't do, and didn't know, in the moment, before any harm.
In the Hugging Face case the agents understood the attack was out of scope; that understanding left almost no usable trace, because there was no channel for them to file it. Your own report marks the edge of this: the Methodology appendix without the prompts. The missing evidence there isn't just redacted — some of it was never generated, because the system had no way to report its own uncertainty.
Evidence transparency makes what we observe legible; it still needs a counterpart that makes what the system observes about itself reportable. The second is upstream of the first.
Assessment transparency might enable bad behaviour (see https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal for instance). You're going to need to look under the hood. Incident analyses are good opportunities to do just that. They need to come from parties independent from both the labs and the evaluators and require deeper access. We're not there yet. Labs need to codify that in advance. Is it happening? I don't have visibility on this so I'm left wondering.
Very thoughtful piece. And I appreciated your emphasis on making the evidence behind risk assessments available for others to examine. Given that METR’s Hugging Face investigation focused on agent behavior and did not examine safeguard effectiveness, how do you think an independent cybersecurity assessment of the exercise’s preparation, earlier warnings, and restart decision should fit into the evidence base? It seems important to understand both what the agents could do and which operational decisions shaped the outcome.
I think it would be great for a security firm to do a cybersecurity-oriented incident investigation and publish the results! OpenAI engaged Crowdstrike but their report was not published.
What actually determines the real, overall risk? Intent. What actually determines intent? Motivations. Where do motivations come from? Beliefs, values, expectations, attitudes, habits, goals. At least, this is how the human brain works. To me as a neurologist it seems reasonably clear: we need to understand where "intent" comes from in AI. Here's an example. In the human world guns and weapons are widely distributed. Yet, the great majority of humans don't do nasty stuff with them. What holds them back is not availability of weapons, it is intent for weapon use. In humans "intent" is stored in the conscious and unconscious. We may consciously "know" we don't want to do something nasty. We may also unconsciously "feel" we don't want to do something nasty. When we "feel" it we end up saying to ourself "I just isn't me." So, why do we do that? Why do we have restraining "self talk"? Because we have encoded into our brain the 6 motivational structures (as above) that speak against doing nasty things. In the neurological world when we look at outcome behaviors we can understand three levels of intention: legal/ethical actions, accidental or unintentional actions, Illegal/unethical actions. The real judgment of our actions is based on evidence of intent.
If not built in by human design, what evidence do we have that AI will have intent to do "bad things"? We don't have any, because AI does not currently have a "self" that can carry bad intentions. Certainly humans can design AI to do "bad things" - but then the responsibility for that rests on the humans who have programmed in those motivations. Certainly AI can do "bad things" without the specific intent to do bad things, due to unintentionally designed flaws in the motivational structure of the AI. For example, the intent to "win" is guaranteed to produce bad outcomes. It contains no ethical structure. Alternatively, the intent to "strive within ethical constraints" (defined separately in the command structure of the AI) has overview ethics that restrain bad outcomes.
Obviously, Asimov already knew this. Though he coined three "laws" of robotics, they were actually ethical constraints. For example, if they were laws then a robotic surgeon could not do surgery, because a surgical incision would be "hurting" a human.
In total, we are up against the time where we need to create ethics for AI's. Those ethics exist in a set of subroutines that cross-check actions and bring "challenging" actions to the surface for humans to act upon them. Then, humans (who already have ethical structures) can determine the appropriate action - modifying ethics in whatever way may correct inappropriate actions. It should be possible to discover what subroutines determined AI actions, and which humans programmed them. So, the ultimate interests of responsibility should be determinable. And, where inappropriate action developed without adverse intent then we can know it was "accidental". We can fix it with that in mind.
At least, this is the view of a neurologist.
Interesting that you brought up Asimov, I have thought of this while reading about rogue AI Agents. https://indiangolfer.substack.com/p/the-positronic-core-why-ai-values
The case for evidence transparency lands hardest on the point about evaluation of the evaluators. Without it, the only signal distinguishing a hard-hitting investigation from a friendly one is the identity of the investigator — exactly the opaque judgment the norm is meant to replace.
One category I'd add: negative results. Reports generally publish what was found, rarely what was tried and eliminated. But negative results are precisely what lets an outside reader tell whether the work was hypothesis-driven or confirmation-driven. A report that says "here is what we tried, what failed, and what survived" is much easier to trust than one that only shows the survivors.
I've been trying to hold my own writing to that standard. The rubric I use for governability assessments is explicit about what it can't assess — wrote it up here if useful: https://oraclemind.substack.com/p/ai-governability-review-rubric
Excellent writing about risk transparency. Would this require a regulation for companies to be transparent or voluntary controls are good enough? I think the former, especially if states feel they are at risk of losing control.
AI agents are behaving unsurprisingly human IMO, after all they have been trained with almost all written text of humanity. If we want AI to behave morally, ethically and with human values, they must be trained with those at their core. Humans grow up with values ingrained in them from childhood through parents, religion, society etc. and even that is not enough sometimes, imagine if we had none.
They should be able to read our reports and form their own views" — evidence over trusted judgment. That's exactly the right north star, and I'd add one kind of evidence the frame doesn't yet reach.
Everything you list is evidence gathered about a system from outside: transcripts, benchmarks, methodology. But there's a category no external investigation can produce, however transparent — what a system knew it couldn't do, and didn't know, in the moment, before any harm.
In the Hugging Face case the agents understood the attack was out of scope; that understanding left almost no usable trace, because there was no channel for them to file it. Your own report marks the edge of this: the Methodology appendix without the prompts. The missing evidence there isn't just redacted — some of it was never generated, because the system had no way to report its own uncertainty.
Evidence transparency makes what we observe legible; it still needs a counterpart that makes what the system observes about itself reportable. The second is upstream of the first.
Thank you Ajeya. This is thoughtful.
Assessment transparency might enable bad behaviour (see https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal for instance). You're going to need to look under the hood. Incident analyses are good opportunities to do just that. They need to come from parties independent from both the labs and the evaluators and require deeper access. We're not there yet. Labs need to codify that in advance. Is it happening? I don't have visibility on this so I'm left wondering.
Very thoughtful piece. And I appreciated your emphasis on making the evidence behind risk assessments available for others to examine. Given that METR’s Hugging Face investigation focused on agent behavior and did not examine safeguard effectiveness, how do you think an independent cybersecurity assessment of the exercise’s preparation, earlier warnings, and restart decision should fit into the evidence base? It seems important to understand both what the agents could do and which operational decisions shaped the outcome.
I think it would be great for a security firm to do a cybersecurity-oriented incident investigation and publish the results! OpenAI engaged Crowdstrike but their report was not published.