Could a company overpower nations?
The leading AI company could gain unprecedented power
Just as the police need cars to catch criminals in cars, we’ll increasingly need to rely on AI to protect ourselves from AI. Now that AI systems can discover a huge range of vulnerabilities in important software, developers have to lean on AI to find and patch them fast enough. If AI enables much more subtle forms of market manipulation or fraud at much higher volumes and faster speeds, auditors may increasingly need AI to oversee all that activity. If AI can generate propaganda capable of persuading us of virtually anything, we may need AI to continuously filter information before it reaches us. If intelligent drones can quickly hunt down anyone anywhere in the world, we may need AI-enabled surveillance and drone police forces to keep murder and assassination under control.
This would be an extremely precarious situation. Eventually, every country’s military would become wholly dependent on AI to protect them against everyone else’s AI-run militaries. All these AI systems are likely to be trained in broadly similar ways, and this training process could easily create incentives to undermine human control. If the AI systems running the US military and the AI systems running the Chinese military are both severely misaligned (even if they don’t share the same misaligned goal), they may cooperate with one another to disempower the human governments of both countries, as occurs in the AI 2027 scenario.
But the situation would be extremely precarious even if misalignment is not an issue, particularly if there’s an intelligence explosion.
Right now, the top few AI companies are very close to one another in capabilities. ChatGPT and Claude are fairly interchangeable; Anthropic, OpenAI, and Google have all been within a few months of each other on the relevant capability trends for the last couple of years. But in a rapid feedback loop where AI enormously accelerates AI R&D, the same gap in calendar time translates into a much larger gap in capability level. The precise dynamics depend heavily on takeoff speeds, but there’s a good chance that a company which is a few months in the lead around the time of AI research parity will quickly develop superhuman AI that is vastly more powerful than all its competitors’ systems. It might get to the point where both ordinary human institutions and all other AI systems are woefully inadequate for defending against this company’s AI. We might be forced to rely entirely on Claude to protect ourselves from Claude.
If this happens, the leading AI company would be in a position of incredible power. Law enforcement agencies and the military would be wholly reliant on this one company. If this AI is trained to be loyal to the company’s values or the CEO’s orders rather than the ordinary chain of command, it would be the whole US government was run entirely by foreign double agents. Alternatively, the company could keep its most advanced AI internal, without selling to the government or anyone else. This depends (again) on takeoff speeds, but it’s plausible that with exclusive access to a vast army of superintelligent labor, they could rapidly develop advanced weapons that would pose a genuine threat to the US military.
Private companies should not have the power of nation states. If takeoff is fast, how do we prevent the leading company from accumulating an unprecedented amount of unaccountable power? One possibility is to try to ensure that there are always multiple AI companies at the same capability level. Unfortunately, this is in tension with addressing AI takeover risk (a tighter race means less slack to do alignment and control) and it’s difficult to precisely engineer in any case. I’m currently interested in two other interventions:
Require AI companies to make any model they use internally available for anyone to purchase externally.
Require AI companies to train their models to obey the law rather than the company itself, and implement internal controls to make it difficult for the CEO or anyone else to secretly train the model to do something different.
Both of these interventions would require a third party to verify compliance. AI companies are developing systems that we will soon need to trust with our lives. We need accountability to ensure those systems serve all of us.