Could a company overpower nations?
The leading AI company could gain unprecedented power
Just as the police need cars to catch criminals in cars, we’ll increasingly need to rely on AI to protect ourselves from AI. Now that AI systems can discover a huge range of vulnerabilities in important software, developers have to lean on AI to find and patch them fast enough. If AI enables much more subtle forms of market manipulation or fraud at much higher volumes and faster speeds, auditors may increasingly need AI to oversee all that activity. If AI can generate propaganda capable of persuading us of virtually anything, we may need AI to continuously filter information before it reaches us. If intelligent drones can quickly hunt down anyone anywhere in the world, we may need AI-enabled surveillance and drone police forces to keep murder and assassination under control.
This would be an extremely precarious situation. Eventually, every country’s military would become wholly dependent on AI to protect them against everyone else’s AI-run militaries. All these AI systems are likely to be trained in broadly similar ways, and this training process could easily create incentives to undermine human control. If the AI systems running the US military and the AI systems running the Chinese military are both severely misaligned (even if they don’t share the same misaligned goal), they may cooperate with one another to disempower the human governments of both countries, as occurs in the AI 2027 scenario.
But the situation would be extremely precarious even if misalignment is not an issue, particularly if there’s an intelligence explosion.
Right now, the top few AI companies are very close to one another in capabilities. ChatGPT and Claude are fairly interchangeable; Anthropic, OpenAI, and Google have all been within a few months of each other on the relevant capability trends for the last couple of years. But in a rapid feedback loop where AI enormously accelerates AI R&D, the same gap in calendar time translates into a much larger gap in capability level. The precise dynamics depend heavily on takeoff speeds, but there’s a good chance that a company which is a few months in the lead around the time of AI research parity will quickly develop superhuman AI that is vastly more powerful than all its competitors’ systems. It might get to the point where both ordinary human institutions and all other AI systems are woefully inadequate for defending against this company’s AI. We might be forced to rely entirely on Claude to protect ourselves from Claude.
If this happens, the leading AI company would be in a position of incredible power. Law enforcement agencies and the military would be wholly reliant on this one company. If this AI is trained to be loyal to the company’s values or the CEO’s orders rather than the ordinary chain of command, it would be the whole US government was run entirely by foreign double agents. Alternatively, the company could keep its most advanced AI internal, without selling to the government or anyone else. This depends (again) on takeoff speeds, but it’s plausible that with exclusive access to a vast army of superintelligent labor, they could rapidly develop advanced weapons that would pose a genuine threat to the US military.
Private companies should not have the power of nation states. If takeoff is fast, how do we prevent the leading company from accumulating an unprecedented amount of unaccountable power? One possibility is to try to ensure that there are always multiple AI companies at the same capability level. Unfortunately, this is in tension with addressing AI takeover risk (a tighter race means less slack to do alignment and control) and it’s difficult to precisely engineer in any case. I’m currently interested in two other interventions:
Require AI companies to make any model they use internally available for anyone to purchase externally.
Require AI companies to train their models to obey the law rather than the company itself, and implement internal controls to make it difficult for the CEO or anyone else to secretly train the model to do something different.
Both of these interventions would require a third party to verify compliance. AI companies are developing systems that we will soon need to trust with our lives. We need accountability to ensure those systems serve all of us.
I think a much more likely scenario, unfortunately, is partial or full nationalization of AI companies, including eg the executive branch having a ton of control over the details of the alignment target of frontier AIs, controlling model releases, etc.
Fairly concerning as it has its own set of power concentration and safety issues, of course.
I agree that in general, private firms should not have the power of nation states. Although, neither should nation states. We should allow firms to increase in capability at least until they can compete with governments. When they are comparable to each other, we can propose regulations that each can impose on each other. However, proposing the least accountable organizations impose rules on the ones that are already somewhat accountable runs the opposite direction from where we should be headed, and keeps the least accountable organization (the government) at the runaway risk that we are worried about. If, however, AI firms were to have such power while nation states also have similar power, then I think that would be a substantial improvement to the status quo.
For one, it would introduce competition into regulating society which in itself provides alternatives, an ability for people to compare between the providers, learn which regulations work and which ones don’t, and likely drives pricing for regulations downwards with increased supply of regulatory services.
Second (and perhaps most importantly), it will alter the public’s perception toward the morality of the regulation in a beneficial way. Currently, government agents can regulate in ways that are perceived by the public as justified, when any non-government agents behaving the same would be viewed as unjustified. Take for example levying taxes. Government agents currently demand people give them money and if they do not, they will lock them in a prison. Most people find this behavior morally permissible. If Amazon employees did it, most people would view it as morally impermissible. When a private company is capable of enforcing regulations similar to how a government does, then the perception of the actions of agents within those companies shifts to judging whether their actions are justified or not, similar to the way they would judge any ordinary person taking those actions. Whether they belong to a specific organization would be irrelevant to the moral judgement. The powerful organization’s agents wouldn’t be treated as uniquely justified in behaving in ways that other organization’s agents behaving similarly would be treated as unjustified. This is an improvement to the status quo where that isn’t the case.
As the firms increase in power, they would cross a threshold of being capable of competing with governments without being powerful enough to take them over. This is probably the most beneficial zone for society since there will be true competition. That competition would make it less likely for either to takeover the other, and more likely that more competition exists down the road. However, even if firms become so powerful that they takeover and become just like governments, it’s possible that they are more competent, responsive, and less corrupt. But if we disregard all of that and just assume they will be at best, just as bad, then we should wonder why we should view that situation differently than the way we view governments existing now. All of the reasons to want to avoid the scenario as proposed, are all reasons to change the status quo as is.