The last week has seen some of the most important people in the artificial intelligence (AI) industry declare to the world that the technology has become so far advanced that there is a 10 percent chance we may actually experience Terminator 2: Judgment Day — in short, there’s a one-in-ten shot that Skynet is self-aware.
This was the hype that permeated mainstream media, Silicon Valley industry outlets, Capitol Hill, and even the White House. The thrust of the argument is that we must slow down and regulate AI to avert disaster. However, the reaction from the AI industry and lawmakers in Washington, D.C., raises the question of whether to take the claims of Dario Amodei of Anthropic, Sam Altman of OpenAI, and even Elon Musk at face value. And yes, maybe we should. But then again, maybe AI and the imminent death of humanity is really just one of the oldest scams in the book.
At the core, AI and the future of humanity could — at a high level — take four paths. The first is one in which much ink has been spilled. Namely, if we continue to train and develop what is believed to be artificial general intelligence (AGI) agents at a rampant pace, with few safeguards, and a lack of focus on alignment — think ethical and moral frameworks — then we’ll end up with something like Skynet or HAL 9000, and it will kill us all. And certainly, if Dario, Altman, and the rest have achieved AGI and built what Dune author Frank Herbert termed “thinking machines,” then alignment and regulation should be our top priorities.
This position has inspired, among some, a bleak view of the future should humanity fail to properly regulate what — I’m told — they believe to be either already conscious or near-conscious technology. To that end, I spoke with Samuel Hammond — the director of Artificial Intelligence Policy and chief economist at the Foundation for American Innovation (FAI) — who warned me that we have about 18 months to figure this out. Hammond left me with an impression — echoed by other experts — that there are developments in the AI industry being kept behind closed doors, unknown to the public, that point to an imminent technological leap.
Still, others in the technology space have stressed that the imminent demise of humanity at the hands of our robot overlords is greatly exaggerated. Speaking on background, a national academic who works on ethics and technology emphasized that alignment and safety protocols are advancing just as fast as AI itself. This perspective — while not as prevalent as AI doomerism — still finds a number of standard-bearers in Silicon Valley, who see the future as something far more akin to Star Trek. AI, in their view, will usher in an era of exponential advances in science, human health, resource management, and technology, catapulting humanity into a future where resources are limitless.
Outside the AGI-believing community, however, there is far more skepticism toward both the technology itself and the claims made by Anthropic and OpenAI. The skeptics — myself included — make two arguments as to what is really going on with the claim that AI development needs to be slowed down and either regulated by the federal government or by some international body.
The first argument is that what Altman, Dario, Musk, and others are doing is feigning fear to cajole lawmakers into enacting extensive rules that will grant OpenAI and Anthropic regulatory capture. In essence, the regulations will prevent new competition for Anthropic and OpenAI, as each company is established and has sufficient capital, making compliance less burdensome. However, for new companies, the regulation makes entry into the AI industry prohibitive. In many ways, this boils down to a handful of well-established AI companies — and adjacent players like Musk — setting aside their business rivalries and personal beefs to form a cartel. This move, while perhaps making the companies a modicum less profitable, has the long-term effect of ensuring market capture and very little serious competition.
But why would a handful of well-established AI companies seek to cartelize the industry? The answer is simple. They haven’t achieved AGI — let alone even what we might call dumb AI. What they claim is the next leap isn’t happening and won’t happen — and may actually be unachievable. The “thinking machines” and the Butlerian Jihad to destroy them that Herbert wrote of in Dune may remain in the realm of science fiction and fantasy.
Creating a cartel or regulatory moat then makes sense. If what Anthropic, OpenAI, and others have actually developed is just flashy, souped-up Large Language Models (LLMs), then seeking regulatory capture is arguably the smart move. Unlike what we’d recognize as AGI, LLMs have become relatively easy — albeit still energy-intensive — to build. Without regulations preventing market entry, OpenAI and Anthropic would face a flood of competitors and serious threats to their long-term profitability.
All of this brings us to the fourth possibility for why the AI doomers went public when they did: Initial Public Offerings (IPOs). Anthropic could go public as early as next month, and the stock sale promises to be one of the biggest financial events in human history. OpenAI is likely to go public in the near future as well, though Altman has insisted it won’t happen this year.
If Anthropic has failed to develop, or believes it never will, AGI, that could crater its IPO stock value. However, government regulation to stop killer robots is a pretty good excuse to point to to assuage angry investors! In this case, the major AI companies wouldn’t have to admit — at least not yet — that the push for AGI has failed, is too far off, or is unattainable. They can just blame Uncle Sam, the United Nations, or whatever regulatory body they convince global leaders to impose on them.
It is a clever move, and one that could help them avoid a Theranos situation — but it also means that the whole AI frenzy is based on snake oil. And most troubling, if investors get wise to it, that could impact NVIDIA — which supplies many of the GPU clusters used for AI and makes up nearly 10 percent of the S&P 500’s market capitalization. If NVIDIA’s value were to collapse, the ensuing financial crisis might make 2008 look like a walk in the park.
While I’m a skeptic, I don’t always like to assume the worst in what people claim. So I’m still willing to give Altman and Dario the benefit of the doubt. And several very close friends who work in the AI policy space, especially the brilliant minds at FAI, insist that AI is real and a future of “thinking machines” is coming whether humanity likes it or not. Which leaves me with one final concern … paper clips.
Philosopher Nick Bostrom — to illustrate the real problem of improperly aligned AGI, or even a dumb AI — posed a hypothetical scenario in which an AI is tasked with manufacturing paper clips. The AI is given the relatively mundane task of maximizing the production of aluminum paper clips with little else to guide it. In such a situation, it should be expected that the AI might quickly exhaust its supply of raw materials and, without constraint, begin ripping up cars and other objects with aluminum components. But when that avenue to resources expires, the AI may even realize it can harvest aluminum directly from the human body or even rearrange atoms to create more aluminum. Bostrom’s thought experiment ends with the AI essentially reducing everything in all of existence to paper clips.
It is a scary thought; after all, Russia and Ukraine are both using autonomous drones with algorithmic targeting on the battlefield. Algorithms can have sizable error rates, and according to some, we’ve already seen the results of AI error when the United States likely — by mistake — bombed a school in the Iranian city of Minab, killing 156 people, including at least 120 children and 26 teachers. Just hours later, a second American strike hit a sports and recreation center in a residential area of the city of Lamerd, killing 22 civilians. It is alleged that both strikes resulted from errors made by AI systems using algorithmic targeting.
Maybe this is what Dario and Altman fear. Not Skynet, not Hal 900, but rather paperclips and dead kids.
Like what you’re reading? If so, please consider upgrading to a paid State of the Day subscription or sharing this with a friend. You’d be supporting this newsletter and helping keep independent journalism alive.
If you are already a paid subscriber, make sure to join the conversation in our subscribers-only chat below.
Again, I believe you fail to mention the fact that it is not necessarily a question of will humans survive the development of machine intelligence??? What you and others are forgetting is that there it is not a two choice situation. It is not either we or the machine's win, our tiny inefficient brains are used to thinking it an on or off position (similar to how must computers think). The computers were and are being made in OUR IMAGE, JUST like God made us in his or her image. This third alternative was NOT my idea. Many many computer experts believe this and talk about it yet somehow in the general public discourse it is omitted, why when this is the most likely situation to occur it is never discussed???? I do not know, but it is something to think about. What I am talking about is the SINGULARITY!!! WHAT IS THE SINGULARITY??? In the most simple terms it is the "MERGER OF MAN AND MACHINE!". Machines will not "take us over and dominate us", that is not the most likely scenario. The most likely scenario which most computer experts agree will happen is the Singularity. In fact IT IS ALREADY HAPPENING RIGHT NOW. Computers have been developed and are being developed right now that can be inserted into our human brains to help stop the symptoms of Parkinson's disease, and also some types of Epilepsy. It is only a matter of time before other brain disorders will be able to be treated with what are basically mini computers that can be inserted into the human brain. THAT IS A SINGULARITY WHETHER WE LIKE IT OR NOT! This whole discussion about "computers" so called taking over IGNORES THE POSSIBLITY OF THE SINGULARITY WHICH IS THE MOST LIKELY EVENT TO ACTUALLY HAPPEN. We need less conversation about computers taking over and much more conversation about what we should do and how human activities will be changed and should be conducted under the SINGULARITY, because is exactly what is going to happen whether we want it to happen or not. Elena