Silicon Valley is Suffering From Model Collapse
For anyone living outside Northern California, a lot of the messages coming out of Silicon Valley feel like they’ve been sent from another planet. Last week, many of the industry’s top minds engaged in Olympic-level mental gymnastics as they defended the former Nvidia CEO Jensen Huang’s affairs: We must excuse his dalliances with two employees and an alleged Chinese spy, they argued, not just because Huang is absolutely necessary to the American economy, which is being propped up by AI investment, but because we are in a geopolitical superintelligence race with China.
Then this week, Anthropic, the company most vocally obsessed with the existential risk of AI, went public in an effort to raise $90 billion so that it could speed up construction of the very same systems it’s warning us about.
How do we explain this pattern of illogical behavior? Maybe it’s standard avarice: Shareholders stand to make unfathomable amounts of money from Nvidia and Anthropic stock, and, ultimately, cash rules everything around me. But many Anthropic employees seem uninterested in the money. Maybe it’s groupthink. Maybe it’s a religious-level belief in the abundance of a future in which humans have ushered in superintelligence. Or maybe it’s the cult of the founder and a messianic sense of grandeur—the conviction that only those who see the future first in San Francisco can help protect us from the scourge of disease, from the Chinese, or from ourselves.
Read: A $5.7 trillion face-plant
But watching the AI doomerism on X, the lead-up to Anthropic’s IPO, and the fallout from Huang’s resignation, I think there’s a better framework for understanding what we’re seeing. The degradation of logic, the inscrutable decision making, and the bizarre detachment from popular opinion all point to a dynamic that the AI industry calls model collapse. The term describes a type of digital inbreeding that can theoretically occur when AI models train off synthetic AI output. The most influential figures in AI now sound as if they, having marinated in the insular worlds of the Bay Area and X, are suffering from their own version of the phenomenon.
Model collapse isn’t a perfect analogy. AI companies actually take pains to filter out AI slop when training their language models. But humans? Not so much. You might call it circular reasoning, but the overall takeaway is the same: The corporate leaders shaping our future seem stuck in their own world. A recent post on a popular AI blog captures this spirit well, comparing AI-company employees to people who are microdosing psychedelics. These individuals appear as if they are “in a chronically mildly altered state of mind,” the post argues. Maybe, the post suggests, this state of mind allows them to “see some things more clearly” about AI models and their progress and capabilities than those of us on the outside. The downside, of course, is some form of inebriation. This is a helpful framework through which to view a lot of what’s coming out of the Bay Area. But there’s also a compounding effect: All day, every day, this culture talks with itself, its participants training on a recursive loop of information about the world and what they are building. This information bubble is self-reinforcing, frequently illogical, and also alienating.
The Silicon Valley discourse about recursive self improvement in AI offers the best example of this degraded, circular reasoning. RSI refers to the concept of AI models improving themselves without human intervention or supervision, which would improve AI capabilities exponentially, accelerating discoveries in different fields and increasing the likelihood that models show signs of superintelligence and become far better than humans at any task they pursue. The downside of RSI, according to individuals including the Anthropic CEO Dario Amodei, is that models “could outrun our ability to understand and control these systems.” Humans would be flying blind and thus would likely rely more heavily on AI systems to interpret what is happening inside other AI systems. This is already happening to varying degrees when AI companies train and test new models, but RSI would speed it up to the point that humans would have fully relinquished control. This prospect seems to terrify many of the people who are closest to the technology. “There is not yet a viable scientific plan to solve risks from recursively self-improving AI. Please look up!” Anna Wang, an alignment researcher at Anthropic, wrote this fall.
Yet companies keep racing toward RSI out of fear that someone less principled will get there first. Anthropic argues that pushing forward with an eye toward AI safety will create a virtuous cycle of innovation—a “race to the top.” But you could just as easily argue that these companies are plowing ahead to justify and pay for hundreds of billions of dollars in infrastructure investments. Or that they are so deeply locked in competition with other AI companies and open models in China that they’re behaving irrationally.
This September, Amodei wrote an essay arguing for a multilateral slowdown of AI development, a process he dubbed “pacing the frontier.” Ten days later, Anthropic released its newest model. Roughly a month later, in its IPO filing, the company confessed that its call to pace the frontier “does not describe our own model development or release timelines.”
Sometimes, you can see the model collapse happening at an individual level—and it looks like naked hypocrisy. In September, an Anthropic researcher named Jacob Coxon quit the company in order to argue that AI companies might kill us all. He offered a heartfelt and convincing case in favor of taking the risk of human extinction seriously. But then, when asked in an interview that surfaced this week whether he would donate his OpenAI shares, he seemed to laugh off the suggestion that doing so would make any difference. “I like the money, bro,” he told the audience. He has since agreed to donate his equity.
Sigal Samuel, a writer who covers AI at Vox, has argued that at least some of this dynamic can be explained by a Silicon Valley culture steeped in effective altruism and rationalism. In this sphere, it’s not uncommon for adherents to justify huge risks using counterintuitive and dispassionate rationale. Some see this as a sophisticated form of argumentation, one that can eschew conventional morality (by risking rogue-AI swarm hacks or worse) for a greater purpose (achieving a technological utopia). Samuel argues that the hyperrational thinking deployed by these leaders is overly clever. “Commonsense morality is actually often commonsense for a reason,” she told me recently. “And that’s because it tracks intuitions. Usually, reasoning like ‘If you don’t want a doom machine, don’t build a doom machine’ tends to be good reasoning.”
Some who have quit the AI companies in protest describe the internal cultures as nonsensical, even schizophrenic, in nature. David Robinson, who wrote OpenAI’s transparency reports as a safety lead, argued in this publication that the company, which speaks obsessively about safety, approached AI development “with unimpeded optimism about being able to solve problems as they arise,” leading it to adopt a trial-and-error approach that is no longer responsible, given the power of its models. Relentless optimism is a necessary component of technological achievement but, as Robinson suggests, it can veer into hubris.
And so the Gordian knot grows ever tighter: Founders argue that humanity must accept enormous trade-offs for the benefit of a technological revolution. In exchange for curing cancer, one must be willing to risk the many-sided threat of gradual disempowerment, in the form of either mass job loss or a world where much of the cognitive work is performed by machines. Many of the people grinding away to usher in this future seem to lack humility or be unable to articulate whether the society they’re building toward is one that’s worth living in at all. At some point, their plainly stated claims can end up sounding, frankly, insane. Recently on X, an AI enthusiast argued that one possible positive end state for humans in an AI utopia would be that “we become the happy dogs” living in a world that “does not belong to them.” Musk has said that we might end up like a “pet Labrador if we’re lucky.” He has also argued that it is “unlikely” that humans will be in control in 10 years time. It all signals a certain irrational confidence in where the future is headed, and a disregard for how off-putting it all sounds.
To prevent model collapse, AI companies don’t just add more data to training sets; they add higher-quality information from a diverse array of sources. The dynamic, according to some research, is not an evenly distributed phenomenon, meaning that it appears in specific areas of a model where there are weak points in the training data. A natural fix is to identify the blind spots or shallow areas and bolster them with better data. If Silicon Valley followed this example, perhaps what they would find is an enormous political blind spot.
A June survey of Americans by the Pew Research Center showed that just 9 percent of respondents were more excited than concerned about AI, while 52 percent were more concerned than excited. In an exit poll from the recent midterm elections, 56 percent of voters responded that AI and data centers were a “crucial or major factor” in how they had voted. A recent survey of 1,000 young Americans suggested that few of them trust tech CEOs to act responsibly with AI. Yet many voices in Silicon Valley seem either disconnected from this reality or spiteful of it, accusing anti–data center voters of destroying America’s path to prosperity. As the fallout from the Nvidia scandal shows, Silicon Valley’s disconnect goes beyond just the general public; the venture capitalists arguing for Huang’s reinstatement seem to miss or simply not care about the bipartisan anger in Washington over Huang’s clearly reckless behavior. In an industry that has thoroughly embraced civilizational risk, the notion of the replaceable founder seems to be the true existential threat to the tech elite.
Tech executives are failing at politics for the same reason they’re struggling among regular Americans: They’ve been so monomaniacally focused on winning the AI race that they’ve ignored everything else, including the need to build consensus or even a baseline of enthusiasm for their plans. That may be why the midterm shift away from MAGA took them by surprise. Maybe it’s because they believe that nobody, outside of the AI companies, is qualified or intelligent enough to judge their work, making them both perpetually unaccountable and correct in every argument. Perhaps it is because they believe that the technology is so self-evidently powerful that a public consensus is irrelevant. That could prove to be true, but in the interim, their rhetoric comes off as incomprehensible, frequently condescending, and too often frantic.
At some point, the behaviors that signal a model collapse can also resemble those of a gambler on a bender. One day’s ecstasy over abundant superintelligence turns into the next day’s dire warnings about the future. Half of Silicon Valley appears to be arguing that AI progress is unstoppable—that human loss of control (or life) within the next year seems inevitable; the other half seems to suggest that the entire fate of both AI progress and the economy hinges on one man in a leather jacket who sells chips. Regardless, the only way out is through, so you go harder and faster—anything to increase your odds.
The AI industry, after all, is built on a historic bet: of marshaling as much human knowledge and capital as possible to conjure an alien intelligence greater than our own. Everything we’ve seen from Silicon Valley’s leaders in recent months—the legitimate safety fears, the people willing to cast all morality aside to defend Huang—is the result of this bet. The model-collapse behavior is its own form of recursive self-destruction: AI companies are building the very thing they fear; safety obsessives are alienating themselves from the public who could help them tackle an extraordinary collective-action problem.
This is what happens when the bet becomes so big—when the industry is so full of financial, geopolitical, and existential risk—that it creates its own warped logic. It transforms human innovation from a worthy ambition to an inexorable force and, finally, into something akin to a hostage situation. To stop now, Silcon Valley’s leaders argue, would mean losing not merely the future, but also the present. We’re committed to whatever happens next; nothing is more important than driving forward, and no price is too steep.
This is the lie the gambler tells himself. But how can we accept the premise when the logic behind it is degraded? I don’t doubt the power or even the potential of what many in Silicon Valley are building. But this is an industry that is asking for an unprecedented level of trust—trust it has neither earned nor seems all that interested in cultivating. At least for now, humans still have a right to self-determination and are under no obligation to submit to an ideology that might remove our agency without our consent. No vision of the future, no matter how well capitalized it is, is inevitable. Silicon Valley must widen the aperture. Its leaders must broaden the conversation around what this future might look like, what we will value, and how to keep it human.