Just ask Moloch nicely
Or why an AI pause could be actively harmful
Should we try to pause AI development? Surely the answer is no, right? That could not possibly work.
But others disagree. (And appear quite frustrated about it!)
So, I’ll pose a set of questions, for which I’m open to answers from pro-pause people, because I don’t have good answers to them.
1. How do you appease Moloch?
Moloch, the ruthless, cunning, ever-bargaining God of Defection? Can you placate him? Saying this time — this time! — we will finally defeat him. All at once, say it: “Cooperate!”, all 100 power-seeking entities around the world with sufficient resources to advance AI, in unison. If we all do it at once, then AI will pause, and we will have time. This is just asking Moloch nicely if he will — this time, the first time in history — sit this one out and let us be. Never mind that Moloch is hungry and AI, the feast of all feasts, sits before him — buffets, endless tables, of succulent power and delectable capability, for good or evil or just fun, for whatever ends he wants, waiting to be dined upon. Will his appetite disappear suddenly when presented with the all-capable, all-knowing AI-god, if we just ask politely enough?
2. Are you sure you got them all though?
Well, you could enforce the pause, attempt to eradicate or commandeer data centers, militantly, terroristically, that do not comply. Not appease Moloch, or cave to him, but defeat him in battle. But who is the victor in this scenario? The most militant? What data centers and labs continue advancing AI, if any? The most well-defended militarily? The best hidden? Are you sure you want those selection effects on AI development? The most militant and most secretive? Are you sure the victors will pause? Won’t the victors be the most vicious and strategic and militarily capable? Are they the type to ask Moloch nicely? Is this not the darkest timeline possible? One where the most militant, strategically ruthless, and secretive AI efforts are the ones that race ahead?
3. What, exactly, is a pause?
Presumably it means pausing AI advancement but not AI safety research (boy that seems hard to enforce). Do we keep selling AI as it is today but making no product improvements? Or just no model improvements? Just no more compute? Or no more algorithmic improvements? (boy that all sounds hard to enforce!) What is the legal framework for this? What is the market and economic framework for this? That it is hard doesn’t mean we shouldn’t try, but perhaps the complexity should make us wonder if any democratic or even autocratic political process could possibly pull it off in a sane way? Is it more likely, given the complexity, any actual pause movement or legislation gets hijacked by interested parties for their own ends?
4. Are you sure a pause decreases x-risk?
Does AI safety research ahead-of-capabilities actually decrease risks? Is it all a theoretical and philosophical exercise since we have no empirical framework for x-risk? (how could we?) What is the historical track record of theoretical and philosophical moral efforts when deployed into the real world? Often catastrophic? Of course, the answer can’t be to do nothing about AI risk, and all exercises are theoretical or philosophical to start, but actual AI safety comes out of the interplay between researchers, new capabilities, deployed technology, and society at large. LLMs are safe today because they were trained on human data, by humans, for humans, in a human world, with the vast majority of tokens going to constructive, pro-social, market-accountable actors. Not because of AI safety work from 2019. In reality, safety research is an empirical and practical science and engineering challenge, and actual safety outcomes are mediated by political and social forces.
This is the steelman for pausing: if we pause at just the right moment(s), when capabilities emerge but are not yet widely deployed, we can give AI safety teams ample time to red team and guardrail and inoculate and debug and neuter. This seems like an unambiguous good, if you ignore the competitive and selection effects above. Maybe we’re at such a point right now — agentic AI exists but is still sub-human on most things. So say we pause right now. Does the research continue? Well you can’t stop people from thinking! So yes. Does the compute continue to build? Maybe not if you enforce it (good luck!) but the compute research and optimizations continue. How do we force research minds to work on safety vs capabilities, when all their incentives are for the latter? Just ask Moloch nicely?
What happens when we unpause? Isn’t a pause a self-inflicted compute overhang? Isn’t a pause a potential self-inflicted algorithmic overhang? Have we not just created a moment of unpausing that everyone builds up to in anticipation? A sudden unleash of capabilities on the world? Are we in a better place then? Or a worse place, just later in time?
5. Why aren’t we working on putting resources towards defense instead?
Stopping AI, neutering AI, curtailing AI, pausing AI — they all will fail. There is no question. They are broken ideas. Moloch will feast. He cannot be stopped. All we can do is prepare defenses now ahead of him.
AI is the most powerful general purpose technology of all time. Every entity on the planet, all 8 billion humans individually, all million governmental entities, all 100 million companies, all will be better off personally from harnessing the power of AI vs not, even if on a whole it will be net-negative for many or most of them. How do you stand up to that force? How do you limit that force? You can’t possibly. It will carve its course through civilization, as assuredly as agriculture and sedentism, and all its goods and ills, carved its way through every arable piece of land on the planet, with no respect for the net effect on individual humans, broken by malnutrition and back-breaking field work, epidemic and famine. Moloch must eat.
Constructively, then, what are we to do? We must put enormous resources towards the defense of the Lifeforce against advanced machine intelligence. We cannot know all the risks ahead of time, but we can guess many of them, and we can prepare defenses now so that we are ready, when Moloch comes, to mitigate the harm. Importantly, this does NOT require perfect coordination among all AI actors. It can be done by a few motivated and well-funded organizations, and it prepares us for the inevitable —unaligned AI — instead of waiting for the fantasy universally coordinated alignment. Malevolent AI is inevitable. We can’t prevent it, we can just prepare for it, by putting as much funding and support behind benevolent AI, as quickly as possible.
Defense means defending all Orders of the Lifeforce, from First Order organic beings — you and I, the trees and the bees, to Second Order social beings — cultures, social and government organizations, to Third Order experimental beings — companies, academies. Each are vulnerable in their own way to AI, each essential to the thriving of civilization and the Lifeforce, each a load-bearing pillar of everything we care about, each a part of the Lifeforce. If any one of these fall to AI quickly, the entirety of progress, beauty, life, order, love, consciousness — everything we care about, could be lost or set back millions or billions of years, engulfed by an expanding sun before the spark of intelligence could ever rise again on Earth.
We have much work to do. The variable we should fight for is percent of tokens allocated to the defense of the Lifeforce. Not length of pause.