Intuitively, and from an outside perspective, your AI Research Parity timeline seems too long, and your AI Production Parity timeline is implausibly short.
If progress from November 2025 to date is maintained, I would guess research parity sometime in 2027. But I don't know the details, and as the saying goes: to pick the expert, choose the person who says it will take the longest and cost the most.
In the world of atoms and environmental impact reports, though, automating the whole stack of the most fantastically complicated manufacturing process we have come up with, in less than 25 years, let alone 7 years, seems wildly optimistic. Just today I am reading that only a third of data centers planned for 2027 are being built, in part because of a five-year backlog in the production of the large transformers needed for the generation capacity needed to power those data centers. 2032 as a prediction ignores supply chain realities (fantastically interconnected supply web, really) and the realities of capital stock turnover and investment.
(I don't know if you remember that in the covid era, the supply of compute was heavily restricted for about a year, because of a fire in the one factory in Japan that produces a component of the epoxy resin essential to packaging computer chips. The whole production process is full of single points of failure and capacity constraints like this.)
Are you saying that you expect chip making to be fully automated, from trucks carrying sand to fully tested functional chips, by the end of 2032? Am I understanding this correctly?
This will not happen. You can round probability to 100% that it will not happen in 2032. Or 2042. Or 2052. I’ve talked to AI2027 people and they are profoundly ignorant about the world of manufacturing.
We don’t have the robots that can do plumbing. We don’t have the robots that can use a screwdriver ffs. Or wire cables. Or replace filters. Or do thousands of small tasks required to run a basic machine shop, never mind a semiconductor fab, the pinnacle of our technological civilization.
These robots are possible, but they will take decades to develop. There are no shortcuts, no intelligence explosion, no “singularity” that can speed up industrial development. You can’t cut metal faster than than the safe rate of speeds and feeds. You can’t deposit material, or etch away material, faster than the chemistry allows. You can’t stress-test products faster than the quality requirements specify, 1000-hr HTOL test takes 1000 hours to run no matter how “smart” your AI is.
The whole field of making these grandiose forecasts of “AI taking over the economy” is a laughing stock, filled with people who apparently never set foot into a factory or a mine. Or replaced a gasket in an oven.
You said a 1000h cost to test hardware couldn't be avoided.
I'm not in your industry, but in the context we are talking about of rapid development, why does the hardware need to last ten years?
Or if tests at higher stresses are shorter but risk only failing in cases they are unlikely to see in practice, run shorter tests on a much larger set of cases and accept you are holding them to a higher quality bar than necessary?
In new contexts old practices and constraints can matter less.
This is a helpful clarification, thank you. So when you write "it keeps stress low to avoid false negatives" you are not actually speaking from deep knowledge or experience. These tests fail components all the time, it's just you don't see the failures because we don't ship them. Which is the point of doing these tests.
The reason these tests are kind of non-negotiable, really, is that they are written in blood, sweat, and tears. Even for hardware that is expected to last 2 to 3 years, like phones, the standard reliability tests are used unless you want to have a reputation for unreliable crappy products that fail and set customers' pants on fire. And no, you can't "run shorter tests on a much larger set of cases", shorter runs don't expose failure modes that take longer runs to develop. Really, we have looked, everybody would love to shorten the cycles, but you can't, not if you want the chips to actually work.
This is the thing that drives me nuts about the AI prognosticators, their utter ignorance of how physical world works, and their refusal to learn anything about it. "AI can code, therefore AI can make robots". Well, epoxy curing rates don't care about how powerful your AI is. Thermal expansion coefficients don't know anything about Claude Code. Crack propagation physics remain as hard-to-model as ever. Solder diffusion-related failures aren't going away because AI designed the chip. New fab process development will proceed one wafer batch at a time next year, and next decade, and the one after that.
I've been thinking about this post for over an hour now and I must ask: Are your best guesses really unconditional? Or are they conditioned on AI progress continuing on the current trajectory without governments clamping down on AI companies or the AI companies themselves saying "This is crazy, we need to stop" and then voluntarily stopping?
I ask because imagining AI production supremacy six-and-a-half years from now feels so crazy that I would think even the most reckless AI companies would go "yeah, this is crazy; let's halt" at some point before AI production supremacy if they were able to. (And if humans have already lost control (or are killed?) in the months leading up to supremacy, then my question is what is your vision for how humans could lose control before things are changing fast enough that everyone is alarmed enough to want to halt? Are you imagining humans lose control before parity?)
And relatedly, is your best guess 'locked-in future' by end of 2032? If not, how can it still be an open question if AI production supremacy has been reached? It seems to me like once that milestone is reached then we'll either be locked into 'utopia or doom' or maybe something in between, but regardless it will no longer be something humans can influence.
>Are your best guesses really unconditional? Or are they conditioned on AI progress continuing on the current trajectory without governments clamping down on AI companies or the AI companies themselves saying "This is crazy, we need to stop" and then voluntarily stopping?
It's always hard to account for this sort of thing, so in practice I haven't, though I don't think there's a huge impact on my medians from this because I think there's >50% chance of no giant slowdown.
>I ask because imagining AI production supremacy six-and-a-half years from now feels so crazy that I would think even the most reckless AI companies would go "yeah, this is crazy; let's halt" at some point before AI production supremacy if they were able to.
I disagree with this, I don't think it would feel scary and awful to the AI companies, I think it would feel exciting, like civilization is finally building.
>if humans have already lost control (or are killed?) in the months leading up to supremacy, then my question is what is your vision for how humans could lose control before things are changing fast enough that everyone is alarmed enough to want to halt? Are you imagining humans lose control before parity?
I think the probability of loss-of-control rapidly climbs from parity through supremacy, and I think it's >50% AIs could take over if they wanted to before AI production supremacy (e.g. in AI 2027 the takeover sort of starts around AI research supremacy and then is cemented around AI production supremacy: https://ai-2027.com/)
>And relatedly, is your best guess 'locked-in future' by end of 2032? If not, how can it still be an open question if AI production supremacy has been reached? It seems to me like once that milestone is reached then we'll either be locked into 'utopia or doom' or maybe something in between, but regardless it will no longer be something humans can influence.
Yes, I do think that by the time of AI production supremacy, the AIs (almost by definition) are going to be making the decisions impacting the future and the humans' influence over it is going to be minimal and flow through the AIs protecting their interests (the way childrens' influence over the world flows through adults protecting their interests now).
> I don't think there's a huge impact on my medians from this because I think there's >50% chance of no giant slowdown.
I want to examine the question of the size of the impact of a moderate probability of pause/slowdown on unconditional medians for a second.
My cached answer to p(slowdown) question is a ~20-30% chance of a significant pause or slowdown before transformative AI (so I'm on the same side of maybe as you). I've found it hard to reflect this in my unconditional forecasting using intuition alone, so I've done explicit math before to see how much my intuitive conditional forecasts change when I add in the 20-30% chance of an X-year pause. And IIRC what I found is that even a 20-30% chance of pause actually can have a significant effect on my unconditional medians due to my conditional PDF having high uncertainty (low probability density) around my median.
I just asked Gemini 3.1 Pro to do the math for me and it said that if my conditional median is mid-2032 (2032.5), and my conditional 60th percentile is 2034 (1.5 years later), then a 25% chance of a 5-year pause before supremacy is enough to cause my unconditional median to move 1.5 years to 2034 as well.
Quoting Gemini for the math explanation:
> To find the unconditional median, we need to find the year where the combined probability mass crosses 50%. The formula for the probability of reaching supremacy by any given year (Y) is a weighted average of the two scenarios: Unconditional P(Y) = [0.75 × Conditional P(Y)] + [0.25 × Conditional P(Y - 5)].
> If we test 2034 as the new unconditional median, we plug in my 60th percentile (0.60) for the no-pause scenario:
> The first part of that equation (0.75 × 0.60) equals 0.45. To reach the 50% threshold to be the new median, the delayed-timeline scenario only needs to provide the remaining 5% of probability mass. Mathematically, 0.05 / 0.25 = 0.20. Therefore, as long as my conditional 20th percentile is the year 2029—which fits perfectly on a curve where the median is 2032.5—the math balances perfectly.
> Because my probability density is relatively flat around the median (it takes 1.5 years just to gain 10% of probability mass, going from 50% to 60%), a 25% chance of a 5-year pause mathematically "steals" enough probability mass to drag the median back an entire year and a half.
This is obviously a simplified toy model that doesn't capture everything perfectly, and ~1.5 years may not be a "huge impact" on the median, but the point is that it seems a moderate probability of a pause or slowdown can have a nontrivial impact on one's unconditional median.
> I think the probability of loss-of-control rapidly climbs from parity through supremacy, and I think it's >50% AIs could take over if they wanted to before AI production supremacy (e.g. in AI 2027 the takeover sort of starts around AI research supremacy and then is cemented around AI production supremacy: https://ai-2027.com/)
I'm interpreting your "the probability of loss-of-control rapidly climbs from parity through supremacy" as implying that your probability of the-AIs-could-take-over-if-they-wanted-to at parity is very significantly lower at parity than at supremacy, like maybe 30% at parity versus 80% at supremacy to put some very rough numbers in your mouth, with the upshot being that you think that at parity humans are probably still in control even if the AIs are misaligned enough that they want to take over, meaning that at parity it *probably* wouldn't be too late for humans to halt AI progress (e.g. turn off the data centers and robots) and buy more time, even though it probably *would* be too late to do this at supremacy. Is this accurate for what you believe (the exact numbers obviously aside)?
These definitions implicitly involve drawing somewhat arbitrary boundaries in the supply chain though. For example, in AI R&D you need to exclude datacenter staff to make it make sense, even though at some point the operations of AI alone would constrained without datacenter staff since they debug large training runs (if you included them, you would require AGI-complete humanoid robots to reach adequacy). And maybe you also exclude your data contractors, even though you require fresh data to keep making research progress and it's purely a financial decision whether data contractors are within or without the company...
Similar issues arise if you try to generalize this definition to pharmaceuticals or any other area of the economy.
Based on this, it seems to me that we should only expect "adequacy" at some post-TAI stage, and it should happen roughly concurrently with parity and supremacy.
I’d argue the AI research boundary is arbitrary but the AI production boundary / self-sufficiency is not; it’s just the point where AIs can keep going even if literally all humans died. That’s the ultimate endpoint but I think it’s worth accepting a bit of arbitrariness to define earlier waypoints.
Also, the problem of having to draw an arbitrary boundary is inherent to any operationalization of “full automation of AI R&D” or full automation of any sector short of the whole economy.
So you basically don't separate the AI production boundary from a "whole industrial base" production definition? Interesting
Regardless, I don't really see how you could possibly get to 2032 if that's the definition. I would expect AI research adequacy in the early 2030s, at a stretch.
Well you can strip parts of the industrial base that aren't necessary for AI production, like e.g. all those machines that make taffy or bicycles, right?
I'm not saying it necessarily includes 100% of what we call "industry," I'm just saying it's more crisply defined than other thresholds (because you can talk about *all* humans disappearing rather than having to write down some subset of tasks humans are no longer allowed to do).
>Regardless, I don't really see how you could possibly get to 2032 if that's the definition
>I would expect AI research adequacy in the early 2030s, at a stretch.
I don't see why you're confident in this, it seems like we could be much closer. If we got AI research adequacy next year, would you agree that AI production supremacy in the early 2030s is plausible?
A few comments. #1: I feel both the blue and orange lines may be at very near 100% for a good while. So much that maybe what we'd need is a log scale view of 'how much slower' it would be. Right now it is already much slower to work without AI. Eventually it will be 10x slower, and then 100x times lower. I expect this to happen before 'adequacy' begins (e.g. currently removing humans slows things down by >100x).
#2: the parity notion is interesting, but I expect it to happen at maybe 100x levels (so at 99% in your chart), maybe higher, so it would in my view probably happen at around the same time as 'adequacy', and maybe earlier! E.g. if parity happens at a point where humans and AIs being removed reduces productivity by 500x... then can you say that 'adequacy' has happened? Unless you compare 'adequacy' to the 'baseline' of humans-without-AIs instead.
#3: the 'supremacy' notion. You invoke 'aliens' to 'adjucate' an objective notion of productivity here. And it does make some sense, due to instrumental convergence. But the term 'supremacy' usually means 'power', not 'instrumental productivity'. The AIs achieving superiority of productivity is not the same notion as the AIs achieving superiority of world-power. Between humans, power and productivity are quite distinct. Of course higher productivity normally leads to gains in power over time, but this also depends on many other assumptions. My expectation is that most likely than not, there will be, at some point, a stage where AIs have >50% of all power, but my expectation is that this can happen much later than your 'supremacy' point.
#4: "At this point, every country on Earth is wholly reliant on its entirely-automated military, and it would be trivial for AI systems to take over the world if they wanted to." -- AI systems at this point will even more heterogeneous than they are now. While I agree that some ASIs will be 'real' agents that maximize some utility function, there will be likely a wide range of other AI systems that are functionally very important and that may not behave in the same way, and some of these AI systems, in a world where ASIs exist, maybe 'hardened/secure' to perform their 'functions' and not capable of being 'negotiated with'. They may as parts of supersystems (like cryptography systems, computer security systems, political systems, financial systems, biodefense systems) that are secure but not utility-maximizing, and that stabilize the existing world power-structure.
>Right now it is already much slower to work without AI. Eventually it will be 10x slower, and then 100x times lower. I expect this to happen before 'adequacy' begins (e.g. currently removing humans slows things down by >100x).
In your framework (talking about factor slowdowns instead of percentage of output), I'm saying removing humans would slow things down by a factor of *infinity*, because literally no output is made without any human involvement.
I think adequacy (AIs being able to make progress entirely on their own) will probably happen somewhere around the time that removing the AIs would slow things down 10x. This isn't a general thing about technology, it's just a prediction about AIs' autonomous capabilities.
>the parity notion is interesting, but I expect it to happen at maybe 100x levels (so at 99% in your chart), maybe higher, so it would in my view probably happen at around the same time as 'adequacy', and maybe earlier!
I agree that parity should happened at a higher level than shown, but by definition I don't think parity can happen before adequacy. The AI systems have to be able to make progress *entirely by themselves*, otherwise removing them will always be worse than removing the humans (because removing humans will take output down to *actually zero* while removing the AIs will take output down to some small non-zero value). This is just like farm equipment — they probably make output >100x higher compared to farming by hand, but they are not at parity, because removing humans would still be even worse.
>The AIs achieving superiority of productivity is not the same notion as the AIs achieving superiority of world-power. Between humans, power and productivity are quite distinct.
Agree these are somewhat distinct, though I think they're much more closely coupled than you're talking about. I think the right comparison is not between human *people* but between human *civilizations*, where productivity and military power are extremely closely linked.
> "At this point, every country on Earth is wholly reliant on its entirely-automated military, and it would be trivial for AI systems to take over the world if they wanted to." -- AI systems at this point will even more heterogeneous than they are now. While I agree that some ASIs will be 'real' agents that maximize some utility function, there will be likely a wide range of other AI systems that are functionally very important and that may not behave in the same way, and some of these AI systems, in a world where ASIs exist, maybe 'hardened/secure' to perform their 'functions' and not capable of being 'negotiated with'. They may as parts of supersystems (like cryptography systems, computer security systems, political systems, financial systems, biodefense systems) that are secure but not utility-maximizing, and that stabilize the existing world power-structure.
Yeah, this post isn't arguing about whether the AIs would coordinate with one another or be opposed to one another, that's a different piece of the story, and I do think that using AIs to control / check other AIs is a major source of hope: https://www.cold-takes.com/high-level-hopes-for-ai-alignment/#ai-checks-and-balances That said I think that it's pretty likely that they would coordinate with one another, since it's likely that they're misaligned in correlated ways even if trained by different developers, and it's likely that they find it rational to cooperate with one another than to cooperate with humans.
These are great terms for advanced AI milestones: AI 'adequacy' when a process doesn't completely grind to a halt without humans, 'parity' when it'd proceed roughly equally with only humans or only AI, and 'supremacy' when humans are actively holding it back.
It's a bit tricky carving up particular processes. Which inputs and context are we assuming? Arguably what we might call the *sufficiency* milestone is 'somewhere' after the adequacy milestones across a swath of supply chain: not only could AI proceed alone, but in principle indefinitely, accounting for depreciation and replacement, sourcing inputs, raw materials, energy, etc. That strikes me as needing a really high bar of physical generality at the extreme, really much better than existing robotic hardware or software, and likely much tacit contextual knowledge at the extremities as well.
Possibly those things could be substituted across the board for less demanding (but perhaps more brute force or scaled) alternatives in short order.
And possibly really advanced AI could accelerate development of robotics enough to bring those extremes into reach quickly.
Honestly I'm fairly unsure how dextrous and general and tacit the extremes of the AI supply chain are, how close those are on current R&D trends, and how substitutable.
I appreciate your strong (?) position, and I'd even more appreciate some pointers or gestures at the grounding for them! As for me, I think you're directionally right, but I don't have enough contact with the range of evidence to feel certain. It might be that with more pointers, I'd be able to be more confident.
Quick summary: my strong position comes from the standpoint of knowing, as opposed to guessing. Notice your own language: "possibly", "could", "perhaps". These are not words of confident knowledge.
This is not a dig at you personally, nobody can know everything. What is really crazy though is that there's this whole filed of AI "research" that completely ignores the reality of atoms, and thinks that the rapid advances in the world of bits are going to be naturally replicated. Nobody personalizes this better than the intellectual idiot Yudkowsky, who knows a lot about nothing and makes regular pronouncements of cosmic scale and cosmic stupidity.
But let me give you a couple of examples of what I'm driving at.
LLMs (that's what we're talking about when we say "AI" now) are language machines. So anything that is a "language" is a fair game. Let's apply this to chip design because I know chip design:
I have little doubt that AI will be able to design, verify, and layout a chip relatively soon (say, 10 years). Today this design cycle takes many months. With AI running it, the cycle may shrink to, say 1 month (why not 10 minutes? simulations are incredibly compute-intense, and you need a lot of simulations to design and verify a chip). Next, the layout is sent to a fab ("tapeout").
Well. The fab cycle takes, say, a day per layer, and say there are 45 layers, here's your 6 weeks of fab time. This cannot be shortened, AI or no AI, it's ultimately governed by chemistry. Then the wafers have to be tested, diced, packaged, validated, characterized, and stress-tested. None of these steps can be meaningfully accelerated, we're still looking at months.
So you see now how new chip development can be made 50% faster by compressing the design cycle, but not fabrication cycle. This means that the idea of a "takeoff" where we wake up tomorrow to fully autonomous robots is a fantasy, because these robots need new chips (among other things), and those new chips cannot appear tomorrow.
And then: the dicing machine has a clog in the slurry line. We're done. Need a human to fix it. There are no robots that can do it, not even on the horizon. They are decades away. The current prototypes can awkwardly fold laundry (freaking amazing!!!), and finding and replacing a blocked hose in a complex machine is a several orders of magnitude more difficult.
First, that's really helpful textural detail on fabrication, I learned something - details at least, and your experienced perspective on where bottlenecks lie. To be clear, that's not an area I'm totally clueless in, and I absolutely respect the reality of experimentation and validation there and elsewhere: see my post, You Can't Skip Exploration. https://oliversourbut.substack.com/p/you-cant-skip-exploration?utm_source=share&utm_medium=android&r=lwkfb
I respect also the limits of chemistry and other constraints to construction and fabrication. To me, it's a decent bet that these can't be blown open in a few short years (though there are many exponentials at work even with merely human efforts).
Briefly, though, taking your example.
It sounds like you're less intimately acquainted with robotics than fabs. So we're both 'guessing' when we expect real dexterity to be years away or more. It remains plausible to me though that current progress can accelerate, perhaps even without substantial AI cognitive input - compare autonomous driving which has benefitted quite quickly from massive data capture and moved from stumbling to commercially viable in a few years. AI speeding research design could push that along even faster.
Alternatively, suppose we could (wastefully!) circumvent the fiddliest cloggages by simply building the same old machines and replacing them far more willingly. Or perhaps we could produce much more modular, swappable designs. Or perhaps machines could be constructed (more expensively on current margins, but efficiently if human un-clogging labour becomes scarce) to clog less frequently. Or perhaps reams of humans could be recruited and retrained as de-cloggers - on some hypotheses this might become highly valuable labour. etc.
I don't necessarily need to give much weight to any of these to think it overall conceivable that these are overcomable, even if I'm not banking on it being likely on Ajeya's timelines.
Everything is indeed overcomable, it is the timelines that are important to comprehend properly. Ajeya’s “months” are obviously impossible. We are talking decades for true manufacturing autonomy, from mines to shelves.
I don’t design robots, correct. I have/have been designing sensors that go into robots (among other things), and I have done quite a bit of intricate mechanical design for test systems to test those sensors. So I’m quite familiar with the mechanical end of things, and closely follow advances in robotics.
Self-driving cars are a great illustration of my point! They have a well-defined mission profile, which is rather narrow compared to a robot plumber or electrician. The cars have the mechanical parts related to their mission all figured out, we just need to add a bunch of sensors and start training. It’s going great! Exactly the kind of automation that benefits hugely from all the advances in machine learning.
And yet! It’s going much slower than many, including me, expected. We’re like 90% there, but the next 9% will be much harder, and the next 0.9% will be really-really hard, and we may be many decades away from closing the last 0.1% on our way to a fully autonomous car that I want: “car, take me to terminal C, then go home and come back on Friday to pick me up, AA flight such-and-such”. Maybe 20 years?
But I bet the mechanic will still be human even though he’d need a degree in computer science to service this thing.
Your illustration shows "parity" as happening at roughly 60% productivity loss, but as others have commented that seems like it could be way too low. What is your median and 80% confidence interval for what productivity loss would actually be at parity?
Intended to be medians, modes would be somewhat shorter. Though this is all very volatile and I'd probably move around by multiple years if I thought about it longer and made the definitions more precise and tried to make my views consistent with all my other views
I like this framing and post, but I worry that it fails because we haven't defined "sectors" precisely enough.
If the sector is "Send information back to earth from one light-day away as soon as possible", the Voyager 1 probe, launched in 1977, has achieved adequacy, parity, and sufficiency.
More seriously, the possibility that we've already achieved adequacy or even supremacy today seems extremely remote, but it could be true if you defined the sector sufficiently narrowly? Is AI R&D a "natural" sector? Does it include decisions about buying and purchasing hardware? Does it include the humans who go into the data centers and replace bad hardware, or who make sysadmin-style choices about compute style allocations? I suspect there are a lot of edge cases that undermine the clean definition that make the questions hard to answer.
Yeah, this kind of thing is why I hedged a bit and said "major" sector. I think machines haven't reached adequacy at any ongoing field of activity that's currently a non-trivial share of GDP (e.g. that you can look up in the BLS or whatever they use to classify sectors in the stock market).
Agreed, but I think the choice of what you define as sector is load-bearing. AI R&D isn't something we can look up in the BLS either, is it? Claude suggests that AI R&D could be classified under "NAICS 5417 — Scientific Research and Development Services". So maybe we're imagining that everyone responsible for keeping all the compute and systems available and running is in some other sector, so the computers stay on if we fire all the humans?
I take seriously your point that "changing the particular definition of powerful AI can easily double or triple your median forecast," and I'm adding that your new metrics, while adding useful clarity by distinguishing adequacy from parity from supremacy, rely on a definition of "AI R&D" that is sufficiently unclear that I don't know how to adjudicate "adequacy on AI Research by 2028."
Agree the boundaries of AI R&D are fuzzy, though you could expand to eg the information technology sector and I think it’d be pretty much the same timelines.
I think the boundary around whole AI production stack is less fuzzy and ultimately what matters most. In some sense it’s the privileged end point and we’re trying to define way points that will necessarily be more tenuous.
Where is any proof symbols sustain automation? This is like using dandelions to gin cotton. A total lack of understanding what syntax in the physical biological realm is evident.
(1). Would you say that AI research adequacy is the equivalent to the “drop in remote worker”?
(2). In the graph about adequacy, parity and supremacy as it relates to AI research and AI production, what would you consider the equivalent is to TED-AI or AGI?
This is likely a big part of why AI agents fail now, but at some point agents' self-correction ability will match humans, allowing agents to autonomously operate for many years the way humans do (and allowing agent "civilization" to operate for many centuries the way human civilization does)
But, generally speaking, the overall outlook is certainly bleak. If, however, we view AI as part of the economic system, then who will buy the goods or services produced by AI? An economy isn't just about production.
I admit I have a hard time understanding why the "consumer argument" would be a strong constraint. If we stop paying wages, it is because what those humans produce is irrelevant, when compared to a machine. So, I don't need to trade with him anymore to get what I want, I either own the machine or trade with it (offering land or the services of my own machines). So yes, I don't produce anything for the human consumer anymore, diminishing my total output, but it does not matter, because I still keep getting everything I was getting *as a physical entity*. I may end up losing money in paper, but deflation more than counterbalances that.
Of course, there are second-order effects, a world where machines are producing everything is not a world where your capital ownership rights are secure at all, starving hunans because they are economically obsolete is monstrous, and so on, but from a pure "losing your costumer base" point-of-view, this is a non-problem.
That is a strong argument. But what about the "civilizational shell"-
within which your property is actually considered "your"? The rest of the population- the part deemed "useless"- might not agree with your definition of property. I can cite a couple of historical precedents that ended in rivers of blood and chaos.
Yes, I completely agree that, by default, such a thing ends in anarchy. However, the problem is that if full automation happens, it includes the automation of security forces themselves, not only by physical robots imposing violence, but also by the whole surveillance apparatus behind them. So, in this case, just letting the "useless" population be discontent is not unfeasible.
There is a report that touches on this better than I could do here, I recommend:
Intuitively, and from an outside perspective, your AI Research Parity timeline seems too long, and your AI Production Parity timeline is implausibly short.
If progress from November 2025 to date is maintained, I would guess research parity sometime in 2027. But I don't know the details, and as the saying goes: to pick the expert, choose the person who says it will take the longest and cost the most.
In the world of atoms and environmental impact reports, though, automating the whole stack of the most fantastically complicated manufacturing process we have come up with, in less than 25 years, let alone 7 years, seems wildly optimistic. Just today I am reading that only a third of data centers planned for 2027 are being built, in part because of a five-year backlog in the production of the large transformers needed for the generation capacity needed to power those data centers. 2032 as a prediction ignores supply chain realities (fantastically interconnected supply web, really) and the realities of capital stock turnover and investment.
(I don't know if you remember that in the covid era, the supply of compute was heavily restricted for about a year, because of a fire in the one factory in Japan that produces a component of the epoxy resin essential to packaging computer chips. The whole production process is full of single points of failure and capacity constraints like this.)
Are you saying that you expect chip making to be fully automated, from trucks carrying sand to fully tested functional chips, by the end of 2032? Am I understanding this correctly?
Yes, through the mechanism of superintelligent AI systems kicking off a rapid industrial explosion: https://www.forethought.org/research/the-industrial-explosion
This is depicted as happening in 2029 in the AI 2027 scenario: https://ai-2027.com/race
Ok. I work in semis.
This will not happen. You can round probability to 100% that it will not happen in 2032. Or 2042. Or 2052. I’ve talked to AI2027 people and they are profoundly ignorant about the world of manufacturing.
We don’t have the robots that can do plumbing. We don’t have the robots that can use a screwdriver ffs. Or wire cables. Or replace filters. Or do thousands of small tasks required to run a basic machine shop, never mind a semiconductor fab, the pinnacle of our technological civilization.
These robots are possible, but they will take decades to develop. There are no shortcuts, no intelligence explosion, no “singularity” that can speed up industrial development. You can’t cut metal faster than than the safe rate of speeds and feeds. You can’t deposit material, or etch away material, faster than the chemistry allows. You can’t stress-test products faster than the quality requirements specify, 1000-hr HTOL test takes 1000 hours to run no matter how “smart” your AI is.
The whole field of making these grandiose forecasts of “AI taking over the economy” is a laughing stock, filled with people who apparently never set foot into a factory or a mine. Or replaced a gasket in an oven.
1,000 hours of HTOL is to simulate ten years of being in the field. And it keeps stress low to avoid false negatives.
There seems to be a failure of imagination here?
Elaborate?
This isn't xitter, you can write down what "failure of imagination" means.
Oh, sorry.
You said a 1000h cost to test hardware couldn't be avoided.
I'm not in your industry, but in the context we are talking about of rapid development, why does the hardware need to last ten years?
Or if tests at higher stresses are shorter but risk only failing in cases they are unlikely to see in practice, run shorter tests on a much larger set of cases and accept you are holding them to a higher quality bar than necessary?
In new contexts old practices and constraints can matter less.
"I'm not in your industry"
This is a helpful clarification, thank you. So when you write "it keeps stress low to avoid false negatives" you are not actually speaking from deep knowledge or experience. These tests fail components all the time, it's just you don't see the failures because we don't ship them. Which is the point of doing these tests.
The reason these tests are kind of non-negotiable, really, is that they are written in blood, sweat, and tears. Even for hardware that is expected to last 2 to 3 years, like phones, the standard reliability tests are used unless you want to have a reputation for unreliable crappy products that fail and set customers' pants on fire. And no, you can't "run shorter tests on a much larger set of cases", shorter runs don't expose failure modes that take longer runs to develop. Really, we have looked, everybody would love to shorten the cycles, but you can't, not if you want the chips to actually work.
For high-stress environments like data centers or robotics, industrial reliability stresses may not be enough, reports of GPUs failing are all over the place (example: https://techblog.comsoc.org/2024/11/25/superclusters-of-nvidia-gpu-ai-chips-combined-with-end-to-end-network-platforms-to-create-next-generation-data-centers/, scroll down to the bullet points past the image). We may have to use automotive qualifications, which are even harder.
This is the thing that drives me nuts about the AI prognosticators, their utter ignorance of how physical world works, and their refusal to learn anything about it. "AI can code, therefore AI can make robots". Well, epoxy curing rates don't care about how powerful your AI is. Thermal expansion coefficients don't know anything about Claude Code. Crack propagation physics remain as hard-to-model as ever. Solder diffusion-related failures aren't going away because AI designed the chip. New fab process development will proceed one wafer batch at a time next year, and next decade, and the one after that.
I've been thinking about this post for over an hour now and I must ask: Are your best guesses really unconditional? Or are they conditioned on AI progress continuing on the current trajectory without governments clamping down on AI companies or the AI companies themselves saying "This is crazy, we need to stop" and then voluntarily stopping?
I ask because imagining AI production supremacy six-and-a-half years from now feels so crazy that I would think even the most reckless AI companies would go "yeah, this is crazy; let's halt" at some point before AI production supremacy if they were able to. (And if humans have already lost control (or are killed?) in the months leading up to supremacy, then my question is what is your vision for how humans could lose control before things are changing fast enough that everyone is alarmed enough to want to halt? Are you imagining humans lose control before parity?)
And relatedly, is your best guess 'locked-in future' by end of 2032? If not, how can it still be an open question if AI production supremacy has been reached? It seems to me like once that milestone is reached then we'll either be locked into 'utopia or doom' or maybe something in between, but regardless it will no longer be something humans can influence.
>Are your best guesses really unconditional? Or are they conditioned on AI progress continuing on the current trajectory without governments clamping down on AI companies or the AI companies themselves saying "This is crazy, we need to stop" and then voluntarily stopping?
It's always hard to account for this sort of thing, so in practice I haven't, though I don't think there's a huge impact on my medians from this because I think there's >50% chance of no giant slowdown.
>I ask because imagining AI production supremacy six-and-a-half years from now feels so crazy that I would think even the most reckless AI companies would go "yeah, this is crazy; let's halt" at some point before AI production supremacy if they were able to.
I disagree with this, I don't think it would feel scary and awful to the AI companies, I think it would feel exciting, like civilization is finally building.
>if humans have already lost control (or are killed?) in the months leading up to supremacy, then my question is what is your vision for how humans could lose control before things are changing fast enough that everyone is alarmed enough to want to halt? Are you imagining humans lose control before parity?
I think the probability of loss-of-control rapidly climbs from parity through supremacy, and I think it's >50% AIs could take over if they wanted to before AI production supremacy (e.g. in AI 2027 the takeover sort of starts around AI research supremacy and then is cemented around AI production supremacy: https://ai-2027.com/)
>And relatedly, is your best guess 'locked-in future' by end of 2032? If not, how can it still be an open question if AI production supremacy has been reached? It seems to me like once that milestone is reached then we'll either be locked into 'utopia or doom' or maybe something in between, but regardless it will no longer be something humans can influence.
Yes, I do think that by the time of AI production supremacy, the AIs (almost by definition) are going to be making the decisions impacting the future and the humans' influence over it is going to be minimal and flow through the AIs protecting their interests (the way childrens' influence over the world flows through adults protecting their interests now).
> I don't think there's a huge impact on my medians from this because I think there's >50% chance of no giant slowdown.
I want to examine the question of the size of the impact of a moderate probability of pause/slowdown on unconditional medians for a second.
My cached answer to p(slowdown) question is a ~20-30% chance of a significant pause or slowdown before transformative AI (so I'm on the same side of maybe as you). I've found it hard to reflect this in my unconditional forecasting using intuition alone, so I've done explicit math before to see how much my intuitive conditional forecasts change when I add in the 20-30% chance of an X-year pause. And IIRC what I found is that even a 20-30% chance of pause actually can have a significant effect on my unconditional medians due to my conditional PDF having high uncertainty (low probability density) around my median.
I just asked Gemini 3.1 Pro to do the math for me and it said that if my conditional median is mid-2032 (2032.5), and my conditional 60th percentile is 2034 (1.5 years later), then a 25% chance of a 5-year pause before supremacy is enough to cause my unconditional median to move 1.5 years to 2034 as well.
Quoting Gemini for the math explanation:
> To find the unconditional median, we need to find the year where the combined probability mass crosses 50%. The formula for the probability of reaching supremacy by any given year (Y) is a weighted average of the two scenarios: Unconditional P(Y) = [0.75 × Conditional P(Y)] + [0.25 × Conditional P(Y - 5)].
> If we test 2034 as the new unconditional median, we plug in my 60th percentile (0.60) for the no-pause scenario:
> Unconditional P(2034) = [0.75 × 0.60] + [0.25 × Conditional P(2029)].
> The first part of that equation (0.75 × 0.60) equals 0.45. To reach the 50% threshold to be the new median, the delayed-timeline scenario only needs to provide the remaining 5% of probability mass. Mathematically, 0.05 / 0.25 = 0.20. Therefore, as long as my conditional 20th percentile is the year 2029—which fits perfectly on a curve where the median is 2032.5—the math balances perfectly.
> Because my probability density is relatively flat around the median (it takes 1.5 years just to gain 10% of probability mass, going from 50% to 60%), a 25% chance of a 5-year pause mathematically "steals" enough probability mass to drag the median back an entire year and a half.
This is obviously a simplified toy model that doesn't capture everything perfectly, and ~1.5 years may not be a "huge impact" on the median, but the point is that it seems a moderate probability of a pause or slowdown can have a nontrivial impact on one's unconditional median.
> I think the probability of loss-of-control rapidly climbs from parity through supremacy, and I think it's >50% AIs could take over if they wanted to before AI production supremacy (e.g. in AI 2027 the takeover sort of starts around AI research supremacy and then is cemented around AI production supremacy: https://ai-2027.com/)
I'm interpreting your "the probability of loss-of-control rapidly climbs from parity through supremacy" as implying that your probability of the-AIs-could-take-over-if-they-wanted-to at parity is very significantly lower at parity than at supremacy, like maybe 30% at parity versus 80% at supremacy to put some very rough numbers in your mouth, with the upshot being that you think that at parity humans are probably still in control even if the AIs are misaligned enough that they want to take over, meaning that at parity it *probably* wouldn't be too late for humans to halt AI progress (e.g. turn off the data centers and robots) and buy more time, even though it probably *would* be too late to do this at supremacy. Is this accurate for what you believe (the exact numbers obviously aside)?
Yeah I think this is basically right!
These definitions implicitly involve drawing somewhat arbitrary boundaries in the supply chain though. For example, in AI R&D you need to exclude datacenter staff to make it make sense, even though at some point the operations of AI alone would constrained without datacenter staff since they debug large training runs (if you included them, you would require AGI-complete humanoid robots to reach adequacy). And maybe you also exclude your data contractors, even though you require fresh data to keep making research progress and it's purely a financial decision whether data contractors are within or without the company...
Similar issues arise if you try to generalize this definition to pharmaceuticals or any other area of the economy.
Based on this, it seems to me that we should only expect "adequacy" at some post-TAI stage, and it should happen roughly concurrently with parity and supremacy.
I’d argue the AI research boundary is arbitrary but the AI production boundary / self-sufficiency is not; it’s just the point where AIs can keep going even if literally all humans died. That’s the ultimate endpoint but I think it’s worth accepting a bit of arbitrariness to define earlier waypoints.
Also, the problem of having to draw an arbitrary boundary is inherent to any operationalization of “full automation of AI R&D” or full automation of any sector short of the whole economy.
So you basically don't separate the AI production boundary from a "whole industrial base" production definition? Interesting
Regardless, I don't really see how you could possibly get to 2032 if that's the definition. I would expect AI research adequacy in the early 2030s, at a stretch.
Well you can strip parts of the industrial base that aren't necessary for AI production, like e.g. all those machines that make taffy or bicycles, right?
I'm not saying it necessarily includes 100% of what we call "industry," I'm just saying it's more crisply defined than other thresholds (because you can talk about *all* humans disappearing rather than having to write down some subset of tasks humans are no longer allowed to do).
>Regardless, I don't really see how you could possibly get to 2032 if that's the definition
The basic path is AI research supremacy (which I think corresponds to cognitively ~superhuman AI) --> rapid industrial explosion: https://www.forethought.org/research/the-industrial-explosion
>I would expect AI research adequacy in the early 2030s, at a stretch.
I don't see why you're confident in this, it seems like we could be much closer. If we got AI research adequacy next year, would you agree that AI production supremacy in the early 2030s is plausible?
A few comments. #1: I feel both the blue and orange lines may be at very near 100% for a good while. So much that maybe what we'd need is a log scale view of 'how much slower' it would be. Right now it is already much slower to work without AI. Eventually it will be 10x slower, and then 100x times lower. I expect this to happen before 'adequacy' begins (e.g. currently removing humans slows things down by >100x).
#2: the parity notion is interesting, but I expect it to happen at maybe 100x levels (so at 99% in your chart), maybe higher, so it would in my view probably happen at around the same time as 'adequacy', and maybe earlier! E.g. if parity happens at a point where humans and AIs being removed reduces productivity by 500x... then can you say that 'adequacy' has happened? Unless you compare 'adequacy' to the 'baseline' of humans-without-AIs instead.
#3: the 'supremacy' notion. You invoke 'aliens' to 'adjucate' an objective notion of productivity here. And it does make some sense, due to instrumental convergence. But the term 'supremacy' usually means 'power', not 'instrumental productivity'. The AIs achieving superiority of productivity is not the same notion as the AIs achieving superiority of world-power. Between humans, power and productivity are quite distinct. Of course higher productivity normally leads to gains in power over time, but this also depends on many other assumptions. My expectation is that most likely than not, there will be, at some point, a stage where AIs have >50% of all power, but my expectation is that this can happen much later than your 'supremacy' point.
#4: "At this point, every country on Earth is wholly reliant on its entirely-automated military, and it would be trivial for AI systems to take over the world if they wanted to." -- AI systems at this point will even more heterogeneous than they are now. While I agree that some ASIs will be 'real' agents that maximize some utility function, there will be likely a wide range of other AI systems that are functionally very important and that may not behave in the same way, and some of these AI systems, in a world where ASIs exist, maybe 'hardened/secure' to perform their 'functions' and not capable of being 'negotiated with'. They may as parts of supersystems (like cryptography systems, computer security systems, political systems, financial systems, biodefense systems) that are secure but not utility-maximizing, and that stabilize the existing world power-structure.
>Right now it is already much slower to work without AI. Eventually it will be 10x slower, and then 100x times lower. I expect this to happen before 'adequacy' begins (e.g. currently removing humans slows things down by >100x).
In your framework (talking about factor slowdowns instead of percentage of output), I'm saying removing humans would slow things down by a factor of *infinity*, because literally no output is made without any human involvement.
I think adequacy (AIs being able to make progress entirely on their own) will probably happen somewhere around the time that removing the AIs would slow things down 10x. This isn't a general thing about technology, it's just a prediction about AIs' autonomous capabilities.
>the parity notion is interesting, but I expect it to happen at maybe 100x levels (so at 99% in your chart), maybe higher, so it would in my view probably happen at around the same time as 'adequacy', and maybe earlier!
I agree that parity should happened at a higher level than shown, but by definition I don't think parity can happen before adequacy. The AI systems have to be able to make progress *entirely by themselves*, otherwise removing them will always be worse than removing the humans (because removing humans will take output down to *actually zero* while removing the AIs will take output down to some small non-zero value). This is just like farm equipment — they probably make output >100x higher compared to farming by hand, but they are not at parity, because removing humans would still be even worse.
>The AIs achieving superiority of productivity is not the same notion as the AIs achieving superiority of world-power. Between humans, power and productivity are quite distinct.
Agree these are somewhat distinct, though I think they're much more closely coupled than you're talking about. I think the right comparison is not between human *people* but between human *civilizations*, where productivity and military power are extremely closely linked.
> "At this point, every country on Earth is wholly reliant on its entirely-automated military, and it would be trivial for AI systems to take over the world if they wanted to." -- AI systems at this point will even more heterogeneous than they are now. While I agree that some ASIs will be 'real' agents that maximize some utility function, there will be likely a wide range of other AI systems that are functionally very important and that may not behave in the same way, and some of these AI systems, in a world where ASIs exist, maybe 'hardened/secure' to perform their 'functions' and not capable of being 'negotiated with'. They may as parts of supersystems (like cryptography systems, computer security systems, political systems, financial systems, biodefense systems) that are secure but not utility-maximizing, and that stabilize the existing world power-structure.
Yeah, this post isn't arguing about whether the AIs would coordinate with one another or be opposed to one another, that's a different piece of the story, and I do think that using AIs to control / check other AIs is a major source of hope: https://www.cold-takes.com/high-level-hopes-for-ai-alignment/#ai-checks-and-balances That said I think that it's pretty likely that they would coordinate with one another, since it's likely that they're misaligned in correlated ways even if trained by different developers, and it's likely that they find it rational to cooperate with one another than to cooperate with humans.
These are great terms for advanced AI milestones: AI 'adequacy' when a process doesn't completely grind to a halt without humans, 'parity' when it'd proceed roughly equally with only humans or only AI, and 'supremacy' when humans are actively holding it back.
It's a bit tricky carving up particular processes. Which inputs and context are we assuming? Arguably what we might call the *sufficiency* milestone is 'somewhere' after the adequacy milestones across a swath of supply chain: not only could AI proceed alone, but in principle indefinitely, accounting for depreciation and replacement, sourcing inputs, raw materials, energy, etc. That strikes me as needing a really high bar of physical generality at the extreme, really much better than existing robotic hardware or software, and likely much tacit contextual knowledge at the extremities as well.
Possibly those things could be substituted across the board for less demanding (but perhaps more brute force or scaled) alternatives in short order.
And possibly really advanced AI could accelerate development of robotics enough to bring those extremes into reach quickly.
Honestly I'm fairly unsure how dextrous and general and tacit the extremes of the AI supply chain are, how close those are on current R&D trends, and how substitutable.
Timelines here look plausible, if quite bullish.
“Possibly those things could be substituted across the board for less demanding (but perhaps more brute force or scaled) alternatives in short order.”
No they couldn’t.
“And possibly really advanced AI could accelerate development of robotics enough to bring those extremes into reach quickly.”
No it cannot.
“ Timelines here look plausible”
No they don’t. They’re impossible.
I appreciate your strong (?) position, and I'd even more appreciate some pointers or gestures at the grounding for them! As for me, I think you're directionally right, but I don't have enough contact with the range of evidence to feel certain. It might be that with more pointers, I'd be able to be more confident.
Quick summary: my strong position comes from the standpoint of knowing, as opposed to guessing. Notice your own language: "possibly", "could", "perhaps". These are not words of confident knowledge.
This is not a dig at you personally, nobody can know everything. What is really crazy though is that there's this whole filed of AI "research" that completely ignores the reality of atoms, and thinks that the rapid advances in the world of bits are going to be naturally replicated. Nobody personalizes this better than the intellectual idiot Yudkowsky, who knows a lot about nothing and makes regular pronouncements of cosmic scale and cosmic stupidity.
But let me give you a couple of examples of what I'm driving at.
LLMs (that's what we're talking about when we say "AI" now) are language machines. So anything that is a "language" is a fair game. Let's apply this to chip design because I know chip design:
I have little doubt that AI will be able to design, verify, and layout a chip relatively soon (say, 10 years). Today this design cycle takes many months. With AI running it, the cycle may shrink to, say 1 month (why not 10 minutes? simulations are incredibly compute-intense, and you need a lot of simulations to design and verify a chip). Next, the layout is sent to a fab ("tapeout").
Well. The fab cycle takes, say, a day per layer, and say there are 45 layers, here's your 6 weeks of fab time. This cannot be shortened, AI or no AI, it's ultimately governed by chemistry. Then the wafers have to be tested, diced, packaged, validated, characterized, and stress-tested. None of these steps can be meaningfully accelerated, we're still looking at months.
So you see now how new chip development can be made 50% faster by compressing the design cycle, but not fabrication cycle. This means that the idea of a "takeoff" where we wake up tomorrow to fully autonomous robots is a fantasy, because these robots need new chips (among other things), and those new chips cannot appear tomorrow.
And then: the dicing machine has a clog in the slurry line. We're done. Need a human to fix it. There are no robots that can do it, not even on the horizon. They are decades away. The current prototypes can awkwardly fold laundry (freaking amazing!!!), and finding and replacing a blocked hose in a complex machine is a several orders of magnitude more difficult.
First, that's really helpful textural detail on fabrication, I learned something - details at least, and your experienced perspective on where bottlenecks lie. To be clear, that's not an area I'm totally clueless in, and I absolutely respect the reality of experimentation and validation there and elsewhere: see my post, You Can't Skip Exploration. https://oliversourbut.substack.com/p/you-cant-skip-exploration?utm_source=share&utm_medium=android&r=lwkfb
I respect also the limits of chemistry and other constraints to construction and fabrication. To me, it's a decent bet that these can't be blown open in a few short years (though there are many exponentials at work even with merely human efforts).
Briefly, though, taking your example.
It sounds like you're less intimately acquainted with robotics than fabs. So we're both 'guessing' when we expect real dexterity to be years away or more. It remains plausible to me though that current progress can accelerate, perhaps even without substantial AI cognitive input - compare autonomous driving which has benefitted quite quickly from massive data capture and moved from stumbling to commercially viable in a few years. AI speeding research design could push that along even faster.
Alternatively, suppose we could (wastefully!) circumvent the fiddliest cloggages by simply building the same old machines and replacing them far more willingly. Or perhaps we could produce much more modular, swappable designs. Or perhaps machines could be constructed (more expensively on current margins, but efficiently if human un-clogging labour becomes scarce) to clog less frequently. Or perhaps reams of humans could be recruited and retrained as de-cloggers - on some hypotheses this might become highly valuable labour. etc.
I don't necessarily need to give much weight to any of these to think it overall conceivable that these are overcomable, even if I'm not banking on it being likely on Ajeya's timelines.
Everything is indeed overcomable, it is the timelines that are important to comprehend properly. Ajeya’s “months” are obviously impossible. We are talking decades for true manufacturing autonomy, from mines to shelves.
I don’t design robots, correct. I have/have been designing sensors that go into robots (among other things), and I have done quite a bit of intricate mechanical design for test systems to test those sensors. So I’m quite familiar with the mechanical end of things, and closely follow advances in robotics.
Self-driving cars are a great illustration of my point! They have a well-defined mission profile, which is rather narrow compared to a robot plumber or electrician. The cars have the mechanical parts related to their mission all figured out, we just need to add a bunch of sensors and start training. It’s going great! Exactly the kind of automation that benefits hugely from all the advances in machine learning.
And yet! It’s going much slower than many, including me, expected. We’re like 90% there, but the next 9% will be much harder, and the next 0.9% will be really-really hard, and we may be many decades away from closing the last 0.1% on our way to a fully autonomous car that I want: “car, take me to terminal C, then go home and come back on Friday to pick me up, AA flight such-and-such”. Maybe 20 years?
But I bet the mechanic will still be human even though he’d need a degree in computer science to service this thing.
Your illustration shows "parity" as happening at roughly 60% productivity loss, but as others have commented that seems like it could be way too low. What is your median and 80% confidence interval for what productivity loss would actually be at parity?
Yeah the numbers were not really trying to be accurate. I guess it'd be something like ~80-95% productivity loss (from removing the AI) at parity?
Are your "best guesses" about the six milestones your modal predictions, median predictions, or something else (e.g. just vague point estimates)?
Intended to be medians, modes would be somewhat shorter. Though this is all very volatile and I'd probably move around by multiple years if I thought about it longer and made the definitions more precise and tried to make my views consistent with all my other views
the adequacy/parity/supremacy framework is useful. we're probably closer to adequacy in ai r&d than most people want to admit
I like this framing and post, but I worry that it fails because we haven't defined "sectors" precisely enough.
If the sector is "Send information back to earth from one light-day away as soon as possible", the Voyager 1 probe, launched in 1977, has achieved adequacy, parity, and sufficiency.
More seriously, the possibility that we've already achieved adequacy or even supremacy today seems extremely remote, but it could be true if you defined the sector sufficiently narrowly? Is AI R&D a "natural" sector? Does it include decisions about buying and purchasing hardware? Does it include the humans who go into the data centers and replace bad hardware, or who make sysadmin-style choices about compute style allocations? I suspect there are a lot of edge cases that undermine the clean definition that make the questions hard to answer.
Yeah, this kind of thing is why I hedged a bit and said "major" sector. I think machines haven't reached adequacy at any ongoing field of activity that's currently a non-trivial share of GDP (e.g. that you can look up in the BLS or whatever they use to classify sectors in the stock market).
Agreed, but I think the choice of what you define as sector is load-bearing. AI R&D isn't something we can look up in the BLS either, is it? Claude suggests that AI R&D could be classified under "NAICS 5417 — Scientific Research and Development Services". So maybe we're imagining that everyone responsible for keeping all the compute and systems available and running is in some other sector, so the computers stay on if we fire all the humans?
I take seriously your point that "changing the particular definition of powerful AI can easily double or triple your median forecast," and I'm adding that your new metrics, while adding useful clarity by distinguishing adequacy from parity from supremacy, rely on a definition of "AI R&D" that is sufficiently unclear that I don't know how to adjudicate "adequacy on AI Research by 2028."
Agree the boundaries of AI R&D are fuzzy, though you could expand to eg the information technology sector and I think it’d be pretty much the same timelines.
I think the boundary around whole AI production stack is less fuzzy and ultimately what matters most. In some sense it’s the privileged end point and we’re trying to define way points that will necessarily be more tenuous.
Where is any proof symbols sustain automation? This is like using dandelions to gin cotton. A total lack of understanding what syntax in the physical biological realm is evident.
I'm curious whether and how your predictions for these milestones have shifted with the details of the Claude Mythos Preview release.
(1). Would you say that AI research adequacy is the equivalent to the “drop in remote worker”?
(2). In the graph about adequacy, parity and supremacy as it relates to AI research and AI production, what would you consider the equivalent is to TED-AI or AGI?
It seems to me that the problem of error accumulation is being overlooked.
Autonomous systems typically do not fail immediately, but rather through a cascade of minor errors.
This is likely a big part of why AI agents fail now, but at some point agents' self-correction ability will match humans, allowing agents to autonomously operate for many years the way humans do (and allowing agent "civilization" to operate for many centuries the way human civilization does)
Hallucinations aren't going anywhere, ever.
But, generally speaking, the overall outlook is certainly bleak. If, however, we view AI as part of the economic system, then who will buy the goods or services produced by AI? An economy isn't just about production.
I admit I have a hard time understanding why the "consumer argument" would be a strong constraint. If we stop paying wages, it is because what those humans produce is irrelevant, when compared to a machine. So, I don't need to trade with him anymore to get what I want, I either own the machine or trade with it (offering land or the services of my own machines). So yes, I don't produce anything for the human consumer anymore, diminishing my total output, but it does not matter, because I still keep getting everything I was getting *as a physical entity*. I may end up losing money in paper, but deflation more than counterbalances that.
Of course, there are second-order effects, a world where machines are producing everything is not a world where your capital ownership rights are secure at all, starving hunans because they are economically obsolete is monstrous, and so on, but from a pure "losing your costumer base" point-of-view, this is a non-problem.
That is a strong argument. But what about the "civilizational shell"-
within which your property is actually considered "your"? The rest of the population- the part deemed "useless"- might not agree with your definition of property. I can cite a couple of historical precedents that ended in rivers of blood and chaos.
Yes, I completely agree that, by default, such a thing ends in anarchy. However, the problem is that if full automation happens, it includes the automation of security forces themselves, not only by physical robots imposing violence, but also by the whole surveillance apparatus behind them. So, in this case, just letting the "useless" population be discontent is not unfeasible.
There is a report that touches on this better than I could do here, I recommend:
https://intelligence-curse.ai/