I think a much more likely scenario, unfortunately, is partial or full nationalization of AI companies, including eg the executive branch having a ton of control over the details of the alignment target of frontier AIs, controlling model releases, etc.
Fairly concerning as it has its own set of power concentration and safety issues, of course.
I agree that in general, private firms should not have the power of nation states. Although, neither should nation states. We should allow firms to increase in capability at least until they can compete with governments. When they are comparable to each other, we can propose regulations that each can impose on each other. However, proposing the least accountable organizations impose rules on the ones that are already somewhat accountable runs the opposite direction from where we should be headed, and keeps the least accountable organization (the government) at the runaway risk that we are worried about. If, however, AI firms were to have such power while nation states also have similar power, then I think that would be a substantial improvement to the status quo.
For one, it would introduce competition into regulating society which in itself provides alternatives, an ability for people to compare between the providers, learn which regulations work and which ones don’t, and likely drives pricing for regulations downwards with increased supply of regulatory services.
Second (and perhaps most importantly), it will alter the public’s perception toward the morality of the regulation in a beneficial way. Currently, government agents can regulate in ways that are perceived by the public as justified, when any non-government agents behaving the same would be viewed as unjustified. Take for example levying taxes. Government agents currently demand people give them money and if they do not, they will lock them in a prison. Most people find this behavior morally permissible. If Amazon employees did it, most people would view it as morally impermissible. When a private company is capable of enforcing regulations similar to how a government does, then the perception of the actions of agents within those companies shifts to judging whether their actions are justified or not, similar to the way they would judge any ordinary person taking those actions. Whether they belong to a specific organization would be irrelevant to the moral judgement. The powerful organization’s agents wouldn’t be treated as uniquely justified in behaving in ways that other organization’s agents behaving similarly would be treated as unjustified. This is an improvement to the status quo where that isn’t the case.
As the firms increase in power, they would cross a threshold of being capable of competing with governments without being powerful enough to take them over. This is probably the most beneficial zone for society since there will be true competition. That competition would make it less likely for either to takeover the other, and more likely that more competition exists down the road. However, even if firms become so powerful that they takeover and become just like governments, it’s possible that they are more competent, responsive, and less corrupt. But if we disregard all of that and just assume they will be at best, just as bad, then we should wonder why we should view that situation differently than the way we view governments existing now. All of the reasons to want to avoid the scenario as proposed, are all reasons to change the status quo as is.
“If this AI is trained to be loyal to the company’s values or the CEO’s orders rather than the ordinary chain of command, it would be the whole US government was run entirely by foreign double agents.” Seems like this should have ”as if” inserted after “it would be…”
Great quick overview! Isn’t another option to try to prevent reaching AI research parity in the first place to avoid the trade offs downstream of that?
These scenarios generally hinge on rapid recursive self improvement, but I've yet to see a particularly good argument as to why we should expect that to be the case. Yes, AI researchers provide more labour, and maybe even qualitatively better labour, but basically all other optimization processes get much more difficult as your underlying metric improves.
For rsi to happen you need to believe that each incremental IQ point is about as difficult to achieve than the one before. I would find that very surprising, given the low hanging fruit dynamic in almost every other endeavour.
RSI doesn't require the difficulty of making improvements to stay constant; a low hanging fruit dynamic is expected. It just requires that further improvements continue to be tractable if you apply enough effort and compute at the (steadily increasing) state of the art intelligence level. Think of it like this: each time it gains an IQ point, that might make it better at discovering new improvements, in a way that counteracts the increasing difficulty of the problem. It also might not, if the difficulty increases too fast, and we won't know for sure until afterward. So far, though, it seems like there's still a lot of low hanging fruit.
The dangerous version of AI power is not a company declaring sovereignty.
It is the state quietly needing the company to remain sovereign.
If the courts, military, intelligence services, cyber defense, finance, and information systems all depend on one frontier model, then public authority becomes downstream of private infrastructure.
We have had intelligences as powerful as nation-states before. Think of Napoleon: "I saw the crown of France lying on the ground, and I picked it up on the point of my sword." And his general, Bernadotte, who became King of Sweden, at the time still something of a power.
The real problem is that AI will not stop there. It will get more powerful.
AI companies would likely be de facto nationalized long before this situation would actually occur. The government is not a toothless, static force—companies operate at the pleasure of their respective governments, not vice versa.
If you're looking for precedents, consider: the entire field of nuclear fission technology in America was nationalized in 1946 by the McMahon Act. For a decade, it was illegal to own fissionable material. Is it really feasible to say that superintelligence is so much less likely to result in government nationalization than nuclear weapons so as to not even need to consider the possibility in thought exercises?
"Long before" is no longer an option. The government may well nationalize the AI companies this or next year, but any later than that might be too late.
I think a much more likely scenario, unfortunately, is partial or full nationalization of AI companies, including eg the executive branch having a ton of control over the details of the alignment target of frontier AIs, controlling model releases, etc.
Fairly concerning as it has its own set of power concentration and safety issues, of course.
I agree that in general, private firms should not have the power of nation states. Although, neither should nation states. We should allow firms to increase in capability at least until they can compete with governments. When they are comparable to each other, we can propose regulations that each can impose on each other. However, proposing the least accountable organizations impose rules on the ones that are already somewhat accountable runs the opposite direction from where we should be headed, and keeps the least accountable organization (the government) at the runaway risk that we are worried about. If, however, AI firms were to have such power while nation states also have similar power, then I think that would be a substantial improvement to the status quo.
For one, it would introduce competition into regulating society which in itself provides alternatives, an ability for people to compare between the providers, learn which regulations work and which ones don’t, and likely drives pricing for regulations downwards with increased supply of regulatory services.
Second (and perhaps most importantly), it will alter the public’s perception toward the morality of the regulation in a beneficial way. Currently, government agents can regulate in ways that are perceived by the public as justified, when any non-government agents behaving the same would be viewed as unjustified. Take for example levying taxes. Government agents currently demand people give them money and if they do not, they will lock them in a prison. Most people find this behavior morally permissible. If Amazon employees did it, most people would view it as morally impermissible. When a private company is capable of enforcing regulations similar to how a government does, then the perception of the actions of agents within those companies shifts to judging whether their actions are justified or not, similar to the way they would judge any ordinary person taking those actions. Whether they belong to a specific organization would be irrelevant to the moral judgement. The powerful organization’s agents wouldn’t be treated as uniquely justified in behaving in ways that other organization’s agents behaving similarly would be treated as unjustified. This is an improvement to the status quo where that isn’t the case.
As the firms increase in power, they would cross a threshold of being capable of competing with governments without being powerful enough to take them over. This is probably the most beneficial zone for society since there will be true competition. That competition would make it less likely for either to takeover the other, and more likely that more competition exists down the road. However, even if firms become so powerful that they takeover and become just like governments, it’s possible that they are more competent, responsive, and less corrupt. But if we disregard all of that and just assume they will be at best, just as bad, then we should wonder why we should view that situation differently than the way we view governments existing now. All of the reasons to want to avoid the scenario as proposed, are all reasons to change the status quo as is.
“If this AI is trained to be loyal to the company’s values or the CEO’s orders rather than the ordinary chain of command, it would be the whole US government was run entirely by foreign double agents.” Seems like this should have ”as if” inserted after “it would be…”
Great quick overview! Isn’t another option to try to prevent reaching AI research parity in the first place to avoid the trade offs downstream of that?
What is “superintelligent labor” and where does a “vast army” of it (them? Is this a category or a unit?) come from?
These scenarios generally hinge on rapid recursive self improvement, but I've yet to see a particularly good argument as to why we should expect that to be the case. Yes, AI researchers provide more labour, and maybe even qualitatively better labour, but basically all other optimization processes get much more difficult as your underlying metric improves.
For rsi to happen you need to believe that each incremental IQ point is about as difficult to achieve than the one before. I would find that very surprising, given the low hanging fruit dynamic in almost every other endeavour.
RSI doesn't require the difficulty of making improvements to stay constant; a low hanging fruit dynamic is expected. It just requires that further improvements continue to be tractable if you apply enough effort and compute at the (steadily increasing) state of the art intelligence level. Think of it like this: each time it gains an IQ point, that might make it better at discovering new improvements, in a way that counteracts the increasing difficulty of the problem. It also might not, if the difficulty increases too fast, and we won't know for sure until afterward. So far, though, it seems like there's still a lot of low hanging fruit.
The dangerous version of AI power is not a company declaring sovereignty.
It is the state quietly needing the company to remain sovereign.
If the courts, military, intelligence services, cyber defense, finance, and information systems all depend on one frontier model, then public authority becomes downstream of private infrastructure.
That is not regulation failure.
That is sovereignty inversion.
imo undisclosed AI slop on substack should be heavily muted.
We have had intelligences as powerful as nation-states before. Think of Napoleon: "I saw the crown of France lying on the ground, and I picked it up on the point of my sword." And his general, Bernadotte, who became King of Sweden, at the time still something of a power.
The real problem is that AI will not stop there. It will get more powerful.
AI companies would likely be de facto nationalized long before this situation would actually occur. The government is not a toothless, static force—companies operate at the pleasure of their respective governments, not vice versa.
If you're looking for precedents, consider: the entire field of nuclear fission technology in America was nationalized in 1946 by the McMahon Act. For a decade, it was illegal to own fissionable material. Is it really feasible to say that superintelligence is so much less likely to result in government nationalization than nuclear weapons so as to not even need to consider the possibility in thought exercises?
"Long before" is no longer an option. The government may well nationalize the AI companies this or next year, but any later than that might be too late.
This scenario explicitly has the military using this model.