Really great essay, appreciate you writing it. There's a nice self-containedness too that it's about whether AI can do a set of tasks, without needing to think through how humans respond.
That's in contrast to considering, say, "when is AI capable enough to meaningfully threaten people's livelihoods," which requires a bunch of economic theorizing and considering other dynamic human choices (what is our preference for labor from other humans, etc).
Agreed that this is a useful (and helpfully precise) concept! I had some thoughts along these lines a while back I never got around to posting anywhere, so I figured I'd write them up as a post of my own, in case you (or anyone else reading this) is interested: https://philiptrammell.substack.com/p/what-and-how-far-off-is-self-replicating
By self-sufficient I presume you mean "Earth absent of humans". But otherwise similar to the Earth now. Not "a box with gpu, vram, power, sensors, actuators floating in interstellar space surrounded by darkness and cold". 😊 To replicate, make copies of itself (maybe imperfect ones like we do), it would need to be alive in a way life forms of carbon (and water?) are: low power, hardware and software one and the same, or at least entangled. Don't see that particularly more advantageous (from AI-s PoV) compared to now. Where us HI-s, that are analogue and mortal but low power, bootstrap and boot-up AI-s, that are digital and immortal, but use much more power. There is no escaping dependence on someone or something from the environment, outside self.
I meant current Earth (including the existing infrastructure meant to support computers), but I do expect that over time AI systems would develop more biologically-inspired physical technology that can replicate more cheaply and quickly.
I think we will continue to be inspired by nature, and we'll add AI-s help to our chest of tools. We will use all that to change both (us) HI-s and (them) AI-s, to be 'more alive'. In the sense of: to better predict the future, and change both ourselves, and the environment, to reduce the discrepancy. That will make us and them more intelligent than we are now. I see no point of us and them becoming the same. That serves no purpose. I see us and them cohabitating, like the cells and the mitochondria. Like birds and planes: both fly, but achieve very different goals at differing tradeoffs.
But in that vein, something that might be helpful is more concrete examples of what would most narrowly *not* qualify under your definition of self-sufficient. For example, "AI that can earn money in the human economy to pay for itself" and "AI that can coerce humans into doing tasks to be self-sufficient" do not qualify.
I'm less clear about AI that can continue running on extant hardware and maintain and run power systems, but cannot currently produce more chips; does an AI with the ability to survive and run for a year cross the bar of self-sufficiency? What about if it can do that maintenance for a decade? Does it matter that we'd expect further progress in self-sufficiency afterwards? If so, at what point is its ability sufficient to qualify?
But surely it's plausible (and unfortunate!) that there exist several humans who are willing to help a rogue AGI/ASI take over? Suppose it were impossible for that AI to take over without those humans' help. Then surely crossing *that threshold* of AI is what we should already be concerned about, with the self-sufficient AI population being too high a bar?
Agree that this is more extreme than the minimum capability level needed for AI takeover to be plausible, so it's not the only milestone we should be watching. It just has the benefit of being unusually crisp. But we can generate other similar milestones (essentially, could the AI survive with the help of only N humans).
Really great essay, appreciate you writing it. There's a nice self-containedness too that it's about whether AI can do a set of tasks, without needing to think through how humans respond.
That's in contrast to considering, say, "when is AI capable enough to meaningfully threaten people's livelihoods," which requires a bunch of economic theorizing and considering other dynamic human choices (what is our preference for labor from other humans, etc).
Agreed that this is a useful (and helpfully precise) concept! I had some thoughts along these lines a while back I never got around to posting anywhere, so I figured I'd write them up as a post of my own, in case you (or anyone else reading this) is interested: https://philiptrammell.substack.com/p/what-and-how-far-off-is-self-replicating
By self-sufficient I presume you mean "Earth absent of humans". But otherwise similar to the Earth now. Not "a box with gpu, vram, power, sensors, actuators floating in interstellar space surrounded by darkness and cold". 😊 To replicate, make copies of itself (maybe imperfect ones like we do), it would need to be alive in a way life forms of carbon (and water?) are: low power, hardware and software one and the same, or at least entangled. Don't see that particularly more advantageous (from AI-s PoV) compared to now. Where us HI-s, that are analogue and mortal but low power, bootstrap and boot-up AI-s, that are digital and immortal, but use much more power. There is no escaping dependence on someone or something from the environment, outside self.
I meant current Earth (including the existing infrastructure meant to support computers), but I do expect that over time AI systems would develop more biologically-inspired physical technology that can replicate more cheaply and quickly.
I think we will continue to be inspired by nature, and we'll add AI-s help to our chest of tools. We will use all that to change both (us) HI-s and (them) AI-s, to be 'more alive'. In the sense of: to better predict the future, and change both ourselves, and the environment, to reduce the discrepancy. That will make us and them more intelligent than we are now. I see no point of us and them becoming the same. That serves no purpose. I see us and them cohabitating, like the cells and the mitochondria. Like birds and planes: both fly, but achieve very different goals at differing tradeoffs.
Good to have a clear and more concrete definition - and I'd strongly agree that AGI is underspecified, as we've argued for years - see this - https://parallel-forecast.github.io/AI-dict/docs/otherterms.html#artificial-general-intelligence - from the AI forecasting dictionary, ca. 2019.
But in that vein, something that might be helpful is more concrete examples of what would most narrowly *not* qualify under your definition of self-sufficient. For example, "AI that can earn money in the human economy to pay for itself" and "AI that can coerce humans into doing tasks to be self-sufficient" do not qualify.
I'm less clear about AI that can continue running on extant hardware and maintain and run power systems, but cannot currently produce more chips; does an AI with the ability to survive and run for a year cross the bar of self-sufficiency? What about if it can do that maintenance for a decade? Does it matter that we'd expect further progress in self-sufficiency afterwards? If so, at what point is its ability sufficient to qualify?
But surely it's plausible (and unfortunate!) that there exist several humans who are willing to help a rogue AGI/ASI take over? Suppose it were impossible for that AI to take over without those humans' help. Then surely crossing *that threshold* of AI is what we should already be concerned about, with the self-sufficient AI population being too high a bar?
Agree that this is more extreme than the minimum capability level needed for AI takeover to be plausible, so it's not the only milestone we should be watching. It just has the benefit of being unusually crisp. But we can generate other similar milestones (essentially, could the AI survive with the help of only N humans).