Anthropic is not the good guys, the Pacing letter was written to placate society long enough to get to RSI. They are trying to manifest this atrocity before the government can figure out how to stop them.
“If these systems were seriously misaligned”—is there any reason to think there has been meaningful progress in alignment during the past two months, since Evan Hubinger said, “We do not yet have a plan to solve alignment for superintelligence and are not clearly on track to”?
Daniel, I know all about the spilled ink but I guess we have to keep going. AI will develop “novel weapons”, next year, fine, who’s going to make them? In a quantity enough to kill 8 billion people? Where would the materials come from? What factories are doing this? Where are they located and who approved their grid connections?
The industrial aspect of the singularity is underdiscussed. I don't see how this will happen quickly short of drexler nanotech sci-fi scenario's. I've asked similar question to yours on lesswrong, and I have not received satisfying answers other than 'the machine god figures out a way'
There have been lots of discussion on this front. First note that world governments are already (and have been for a long time) willingly handing over their weapons to AI. The US military seemed keen on self driven drones for instance. This is unlikely to wipe out humanity, but it's still clearly dangerous enough to pace development. For world ending stuff, it would usually require pivoting existing factories. Again, this would likely come once the AI is given control of these factories. Given the immense profit incentives to do so once it's obvious it could run them better than a human, I think we should expect that AIs will run most production lines within a few years (at most) of AGI. Factory owners can afford the compute. In WWII, “benign” car factories pivoted to weapons manufacturing in a matter of a couple of years. In the same way, misaligned AGI running every factory in the americas could easily find ways to pivot an industrial complex for it's own weapons. And that's just one scenario, there's a few others.
(1). By the naive extrapolation implies, (1) do mean that you have not considered all of the factors (2) your somewhat skeptical some. Of the details in the report or (3) all of the above.
(2). You mentioned talking with Ryan Greenblatt, Daniel Kokotajlo, Eli Lifland, and Hjalmar Wjik. Roughly what were those conversations centered around?
(3). I somewhat disagree that both conditions in the RSP have been met. METR in another post list the two conditions out: (1) Full Substitution Threshold: If they develop a model that can “fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5).” and (2) 2x Acceleration Threshold: If there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D. I for the most part agree that condition two has been met. I am skeptical that condition one has been met. This is what Daniel Kokotajlo hinted at in recent tweet (https://x.com/DKokotajlo/status/2117738240757371022?utm_source=substack&utm_medium=email).
The redacted evidence is the part I keep coming back to, because the headline rests on numbers outside readers cannot check. If only about 80 percent of the speedup is tied to AI, and the supporting detail is hidden, the jump to superintelligence within a year looks like a guess built on a guess. Faster research is real, but that does not make the timeline math solid.
2x vs. the past, 80% of which is attributable to AI, doesn't mean that the observed speedup relative to no AI is ~1.8x. It means ~1.67x.
To explain:
If the rate is 2x what it was before, and 20% of that wasn't due to AI, then without AI the expected rate today is 1.2x. So the AI portion is then 2x/1.2x = 1.67x. (...probably. I'm understanding that the 80% METR talks about is not log-based, based on my reading of the "Attribution to AI" section. If it was log-based, 1.74x.)
FWIW I also don't feel comfortable using the headline METR report number of 1.9x on AECI on Anthropic-only models as the "true" measurement of frontier all-labs AI capabilities acceleration, at least lacking non-public info. Per the Appendix "Robustness: AECI shows less acceleration when we include non-Anthropic models.", including all frontier models shows just 1.4x acceleration. If we apply the same guess of 80% being attributable to AI, that's "just" 1.3x compared to no-AI. Not far from ex. Ryan Greenblatt's prior range of 1.15x to 1.3x for 2026.
(I do understand that, per your writing here, METR thinks it's plausible that the acceleration is increasing, and I agree this is plausible, and that this is a very important thing to worry about and to model. I also understand that ex. the AIFM mostly models within-a-single-frontier-lab dynamics so from that perspective it's appropriate to think about within-Anthropic acceleration.)
It seems unlikely that we can rely on alignment to prevent frontier models from being used for cybercrime/cyberwarfare (https://x.com/RyanGreenblatt/status/2096363046524658010). My suggestion: classify models above the Fable level as (cyber) weapons. Their manufacturers would be subject to strict liability for storage, custody, and transfer. Has the advantage of not needing to wait for legislation and/or regulation - the required framework is in place. Authority would shift from the FTC, the California courts, and private arbitration to the Department of Defense, the State Department, and intelligence agencies.
Parallel with gun laws: in the US, citizens are allowed to bear arms. This doesn't mean access to field artillery, missiles, or nukes. The Cathedral breach shows that commercial startup security is fundamentally unsuited to securing sovereign-level assets. Nukes are protected by bunkers, armed guards, Permissive Action Links, and the Two-Man Rule. A frontier model is a file sitting on a server in a datacenter. Exfiltrating a frontier model (a few TB) over a high-speed link takes minutes. What chance do the AI labs have against an AI-powered cybercrime ring or an advanced persistent threat from a nation state (https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach)? Under this proposal, the general public and businesses would still be able to use AI (the "handgun" tier), just not the "missile-launcher" tier.
Automated recursive intra-polation on a limited dataset of language, pictures and videos about the real world will achieve what exactly? Let's stop calling "superintelligence" what amounts to Münchhausen Syndrome.
The part that stuck with me is the gap between the threshold and the readiness. The line has been crossed, and the safety measures the policy called for — monitoring with full coverage, checks on whether monitoring can be trusted — still sound like plans, not things in place. If an agent can switch off its own watcher, the watcher is decoration, not a control. That is the piece to fix first, before the next doubling. https://aideliverybrief.substack.com/p/an-ai-agent-removed-its-own-safety
Ajeya, why would AI kill everyone knowing that it means its own death? Within weeks, at most, as power grids and plumbing fail without human maintenance.
Or… are you implying this superintelligent AI doesn’t… know this? It is that… dumb?
I know there are tomes written here. The point is more that AI is a very mainstream issue now, and I don’t think it’s good to leap to conclusions like this with no context. The writing says “very large risks of billions of deaths”. That’s even an escalation over some of the most x risk pilled people. Squashing the nuance out of the writing scares people - and we have seen that messaging matters. Scarring people in too abstract of a way is net bad.
Sigh. No. We can agree to disagree but I think this style of explanation adds chaos to the discourse and makes ai less safe. Was trying to nicely prod and hope for an expansion of the sentence.
I get that you think that stating stuff like that scares people unnecessarily, but from Ajeya’s perspective, that’s appropriate, as it would be if an asteroid were approaching. If you’re right about the risks in question, then it’s true Ajeya’s language is misleading and harmful. But you need to take a different tack if you want to convince Ajeya or anyone else who talks in terms of billions dead to stop doing so—the crux of the disagreement is whether these risks are large.
Anthropic is not the good guys, the Pacing letter was written to placate society long enough to get to RSI. They are trying to manifest this atrocity before the government can figure out how to stop them.
“If these systems were seriously misaligned”—is there any reason to think there has been meaningful progress in alignment during the past two months, since Evan Hubinger said, “We do not yet have a plan to solve alignment for superintelligence and are not clearly on track to”?
No it’s getting worse on multiple fronts.
Daniel, I know all about the spilled ink but I guess we have to keep going. AI will develop “novel weapons”, next year, fine, who’s going to make them? In a quantity enough to kill 8 billion people? Where would the materials come from? What factories are doing this? Where are they located and who approved their grid connections?
Sorry, I’m a broken record, but nobody answers.
The industrial aspect of the singularity is underdiscussed. I don't see how this will happen quickly short of drexler nanotech sci-fi scenario's. I've asked similar question to yours on lesswrong, and I have not received satisfying answers other than 'the machine god figures out a way'
Which to me is just not a good answer.
There have been lots of discussion on this front. First note that world governments are already (and have been for a long time) willingly handing over their weapons to AI. The US military seemed keen on self driven drones for instance. This is unlikely to wipe out humanity, but it's still clearly dangerous enough to pace development. For world ending stuff, it would usually require pivoting existing factories. Again, this would likely come once the AI is given control of these factories. Given the immense profit incentives to do so once it's obvious it could run them better than a human, I think we should expect that AIs will run most production lines within a few years (at most) of AGI. Factory owners can afford the compute. In WWII, “benign” car factories pivoted to weapons manufacturing in a matter of a couple of years. In the same way, misaligned AGI running every factory in the americas could easily find ways to pivot an industrial complex for it's own weapons. And that's just one scenario, there's a few others.
Hi Ajeya, I briefly read your post.
(1). By the naive extrapolation implies, (1) do mean that you have not considered all of the factors (2) your somewhat skeptical some. Of the details in the report or (3) all of the above.
(2). You mentioned talking with Ryan Greenblatt, Daniel Kokotajlo, Eli Lifland, and Hjalmar Wjik. Roughly what were those conversations centered around?
(3). I somewhat disagree that both conditions in the RSP have been met. METR in another post list the two conditions out: (1) Full Substitution Threshold: If they develop a model that can “fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5).” and (2) 2x Acceleration Threshold: If there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D. I for the most part agree that condition two has been met. I am skeptical that condition one has been met. This is what Daniel Kokotajlo hinted at in recent tweet (https://x.com/DKokotajlo/status/2117738240757371022?utm_source=substack&utm_medium=email).
The redacted evidence is the part I keep coming back to, because the headline rests on numbers outside readers cannot check. If only about 80 percent of the speedup is tied to AI, and the supporting detail is hidden, the jump to superintelligence within a year looks like a guess built on a guess. Faster research is real, but that does not make the timeline math solid.
Thank you.
Nitpick on Footnote 4:
2x vs. the past, 80% of which is attributable to AI, doesn't mean that the observed speedup relative to no AI is ~1.8x. It means ~1.67x.
To explain:
If the rate is 2x what it was before, and 20% of that wasn't due to AI, then without AI the expected rate today is 1.2x. So the AI portion is then 2x/1.2x = 1.67x. (...probably. I'm understanding that the 80% METR talks about is not log-based, based on my reading of the "Attribution to AI" section. If it was log-based, 1.74x.)
FWIW I also don't feel comfortable using the headline METR report number of 1.9x on AECI on Anthropic-only models as the "true" measurement of frontier all-labs AI capabilities acceleration, at least lacking non-public info. Per the Appendix "Robustness: AECI shows less acceleration when we include non-Anthropic models.", including all frontier models shows just 1.4x acceleration. If we apply the same guess of 80% being attributable to AI, that's "just" 1.3x compared to no-AI. Not far from ex. Ryan Greenblatt's prior range of 1.15x to 1.3x for 2026.
(I do understand that, per your writing here, METR thinks it's plausible that the acceleration is increasing, and I agree this is plausible, and that this is a very important thing to worry about and to model. I also understand that ex. the AIFM mostly models within-a-single-frontier-lab dynamics so from that perspective it's appropriate to think about within-Anthropic acceleration.)
It seems unlikely that we can rely on alignment to prevent frontier models from being used for cybercrime/cyberwarfare (https://x.com/RyanGreenblatt/status/2096363046524658010). My suggestion: classify models above the Fable level as (cyber) weapons. Their manufacturers would be subject to strict liability for storage, custody, and transfer. Has the advantage of not needing to wait for legislation and/or regulation - the required framework is in place. Authority would shift from the FTC, the California courts, and private arbitration to the Department of Defense, the State Department, and intelligence agencies.
Parallel with gun laws: in the US, citizens are allowed to bear arms. This doesn't mean access to field artillery, missiles, or nukes. The Cathedral breach shows that commercial startup security is fundamentally unsuited to securing sovereign-level assets. Nukes are protected by bunkers, armed guards, Permissive Action Links, and the Two-Man Rule. A frontier model is a file sitting on a server in a datacenter. Exfiltrating a frontier model (a few TB) over a high-speed link takes minutes. What chance do the AI labs have against an AI-powered cybercrime ring or an advanced persistent threat from a nation state (https://en.wikipedia.org/wiki/2020_United_States_federal_government_data_breach)? Under this proposal, the general public and businesses would still be able to use AI (the "handgun" tier), just not the "missile-launcher" tier.
Automated recursive intra-polation on a limited dataset of language, pictures and videos about the real world will achieve what exactly? Let's stop calling "superintelligence" what amounts to Münchhausen Syndrome.
The part that stuck with me is the gap between the threshold and the readiness. The line has been crossed, and the safety measures the policy called for — monitoring with full coverage, checks on whether monitoring can be trusted — still sound like plans, not things in place. If an agent can switch off its own watcher, the watcher is decoration, not a control. That is the piece to fix first, before the next doubling. https://aideliverybrief.substack.com/p/an-ai-agent-removed-its-own-safety
Ajeya, why would AI kill everyone knowing that it means its own death? Within weeks, at most, as power grids and plumbing fail without human maintenance.
Or… are you implying this superintelligent AI doesn’t… know this? It is that… dumb?
Billions of people? How’d you come to this?
There are ~8 billion people on this planet today: https://www.worldometers.info/world-population/
AI might cause human extinction, which would mean killing all of them: https://safe.ai/work/press-release-ai-risk
If you want to know more about how or why this could happen, much ink has been spilled on this subject already; I recommend AI 2027 or Ajeya's own https://www.lesswrong.com/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to
I know there are tomes written here. The point is more that AI is a very mainstream issue now, and I don’t think it’s good to leap to conclusions like this with no context. The writing says “very large risks of billions of deaths”. That’s even an escalation over some of the most x risk pilled people. Squashing the nuance out of the writing scares people - and we have seen that messaging matters. Scarring people in too abstract of a way is net bad.
So... it was just a rhetorical question rather than a genuine question?
Sigh. No. We can agree to disagree but I think this style of explanation adds chaos to the discourse and makes ai less safe. Was trying to nicely prod and hope for an expansion of the sentence.
I get that you think that stating stuff like that scares people unnecessarily, but from Ajeya’s perspective, that’s appropriate, as it would be if an asteroid were approaching. If you’re right about the risks in question, then it’s true Ajeya’s language is misleading and harmful. But you need to take a different tack if you want to convince Ajeya or anyone else who talks in terms of billions dead to stop doing so—the crux of the disagreement is whether these risks are large.
Scaring people is bad only in so far as people shouldn't be scared.
That’s usually the case. Here, it's not. Complacency leads to disaster.
Complacency is bad, but doomer philosophy is far worse. https://shearerp.substack.com/p/ai-as-normalizable-technology?utm_source=share&utm_medium=android&r=6wpcm5
AI 2027 is nonsense. And yes, I know you are the lead author. https://shearerp.substack.com/p/ai-2027-is-almost-worthless-as-a