Overview
Casey Newton reports from the Curve, a conference focused on AI policy, safety, and the prospect of increasingly capable systems. The most consequential idea he heard was that AI labs and governments may eventually need to set an upper limit on model intelligence, potentially blocking systems from reaching superhuman capability.
The episode also covers Newton's decision to return to posting on X after a three-year boycott. He argues that, despite the platform's deterioration under Elon Musk, much of the AI product and policy debate still happens there.
Key Takeaways
Discussion of AI safety is moving beyond calls to slow development. Some people at the Curve are considering whether certain levels of intelligence should simply be off-limits. That could mean a practical ban on superhuman systems, depending on how intelligence and dangerous capability are defined.
The concern is tied to recursive self-improvement: AI systems that can help research, train, and improve their own successors. If that process accelerates, labs could release increasingly capable models faster than governments and safety institutions can respond.
Possible limits include restricting frontier models from conducting AI research, capping available compute, limiting the number of copies a system can run, or barring deployment once a model reaches defined capability thresholds. Anthropic's responsible scaling policy and its calls for a "speed limit" on recursive self-improvement offer an early version of this approach.
The main obstacle is enforcement. A single company cannot impose global intelligence limits if competitors can continue building more powerful systems. Individual governments face the same problem, particularly when major states see AI progress as an economic and geopolitical priority.
Newton sees a widening disagreement between AI lab leaders and the U.S. government. Some industry figures warn that severe risks could arrive as soon as next year, while the administration has alternated between discussing controls and encouraging American firms to accelerate.
On social media, Newton concludes that alternatives to X have not displaced it as the main meeting place for AI researchers, founders, investors, and policy participants. His return is a professional calculation rather than an endorsement of the platform or its ownership.
Practical Steps
Follow AI policy debates with attention to specific proposed controls. Ask whether a proposal limits training, deployment, compute, model autonomy, or use in AI research. These approaches address different risks and require different enforcement methods.
When evaluating claims about superintelligence, separate three questions: whether recursive self-improvement is technically plausible, how dangerous a given capability may be, and who has authority to restrict it.
Pay attention to responsible scaling policies from major labs. Look for concrete triggers, evaluation methods, deployment restrictions, and commitments to pause or alter development when models cross defined thresholds.
If your work depends on public technical or policy discussion, identify where the relevant community actually gathers. Support alternatives where possible, but account for the costs of disengaging from the venue where decisions and debates are taking place.
Notable Quotes
Casey Newton: "We may want to limit how intelligent a future system is allowed to become."
Casey Newton: "The community of people working in and around artificial intelligence product and policy simply refuses to move away from X, no matter how bad it gets."
Casey Newton: "As the stakes of the AI conversation have risen, boycotting X over my disagreements with management has grown both more costly and less effective."
Full Transcript
This is Platformer Plus. I'm Casey Newton. The following column was created using a synthetic voice clone made by ElevenLabs. In today's episode, does intelligence need a hard cap? Calls for a new kind of slowdown were the talk of the curve. Plus, slinking back to X. This is a column about AI. My fiancée works at Anthropic. See my full ethics disclosure at platformer.news/ethics. Today, let's talk about a new front in the conversation about whether to slow the pace of AI development, an emerging push to consider limits on just how smart a large language model can be. I spent the past weekend at the Curve, an annual conference that brings together executives from top AI labs, leaders of nonprofit organizations, government officials, and a small contingent of journalists. Inside a converted hotel in Berkeley, the group spent a few days loudly debating questions of economics, politics, and safety. And while the conference has had one eye on existential risk for each of the three years it has existed, the questions felt notably more urgent this year. I see two explanations. One is the ongoing fallout of the OpenAI hugging face incident, whose implications have alarmed people across the industry. The other is recent blog posts from OpenAI and Anthropic outlining their progress toward recursive self-improvement, AI systems that can research and train their successors, leading to ever faster releases and increased risk that the systems will escape their control. At the curve, I heard multiple speakers say we may want to limit how intelligent a future system is allowed to become. Depending on how far labs take any restrictions, the result could be a de facto ban on systems reaching superhuman intelligence. Whether that seems meaningful to you depends on whether you believe recursive self-improvement or superintelligence is even possible, given the architecture of the current models. And a lot depends on how you would define and assess intelligence here. But it was the most striking idea I heard at a conference full of noteworthy discussions, and its airing in the relatively friendly confines of the curve may be a preview of a more public discussion sometime soon. Over the weekend, though, these comments were made at sessions being conducted under the Chatham House rule. Accordingly, I can't identify the speakers. But the fact that there appears to be agreement on this subject among speakers at the curve struck me as newsworthy. So what does it mean to place limits on intelligence? The speakers I heard did not offer many details, though we can gather clues from other remarks made recently by AI executives. For one, Anthropic CEO Dario Amodei has called for some kind of speed limit on recursive self-improvement. For another, the company's responsible scaling policy, which has been copied in some form by most of its leading rivals, seeks to impose limits on the training and deployment of more powerful systems as they develop new capabilities. Other approaches could involve restrictions on using frontier models to conduct AI research, limits on how much compute is available to systems and how many copies of itself a system is allowed to run, or preventing labs from deploying models past a certain level of capability. That last one is arguably already in action. Notably, though, any effort to restrict models' advancement in this way would require enforcement capabilities that don't exist yet. Individual labs can't impose restrictions like this, nor can individual countries, and the current U.S. government actively opposes them. There are other, easier methods to manage AI's development. Anthropic has already adopted the idea of embedded evaluators, and OpenAI has said it will follow. An antitrust waiver that explicitly allows labs to collaborate on safety questions might be useful enough that it could overcome fears that it would allow leading companies to consolidate their power even further. Still, speakers' comments over the weekend suggested to me that nothing proposed so far meets their own definition of enough, including the morally binding accord AI leaders signed last week with the president. For the moment, lab leaders and the U.S. government differ sharply on one all-important question: how close are we to actual danger? The former say we may see a catastrophe as soon as next year. The latter has vacillated between creating a de facto licensing regime for frontier models and publicly encouraging U.S. companies to go even faster. But the Trump administration remains frustratingly obtuse on the question of where all this is headed, even as more warning signs blink red. A few days before speakers mused about the need to limit intelligence, the president was insisting that everyone start calling it super intelligence now. But super intelligence is already worth more than a branding exercise, and the limits that matter most today are among the officials at the White House. Now let's talk about something truly unpleasant: slinking back to X. About three years ago, I quit posting on X in protest of the new ownership. For a while there, every day brought some fresh outrage from Elon Musk, inciting dangerous harassment of his former employees, rigging the platform against journalists and their work, posting support for anti-Semitic conspiracy theories, dismantling the content moderation apparatus, and so on. Mostly I left because I could not stand the thought of being there. I also hoped that I could use whatever minor influence I had to help kickstart new platforms. I encouraged Meta executives to challenge X with a Twitter clone, and have used Threads daily since it launched. I covered the launch of Bluesky closely and have continued to pour energy into it and the Fediverse, even as they decline. While Musk was suing OpenAI in an effort to effectively destroy it, I repeatedly told Sam Altman to start a rival to X and take his AI posting there. Whatever virtue these actions may have had, they did not succeed at creating a true rival to X in the category that matters most to me: a forum for daily tech and business conversation. The community of people working in and around artificial intelligence product and policy simply refuses to move away from X, no matter how bad it gets. Along the way, X was absorbed twice, first into xAI and then into SpaceX. While I never held much hope that my boycott of X would affect it economically, the fact that it is now a small division of one of the world's most valuable companies made my protest feel even more abstract and quixotic than it was when I began. I am among those who believe that the next year will be decisive in shaping the future of AI policy, regulation, and safety. And increasingly, attempting to influence the AI policy conversation without participating on X or Substack has made me feel like I am fighting with both hands tied behind my back. And so, in the coming days, I plan to resume posting on X. I am doing so not because I think it's any better — it isn't — but because I have exhausted all of the alternatives. I continue to believe that anyone who can avoid posting on X should, but my calculus about the T-ray doffs involved for my own work has shifted. As the stakes of the AI conversation have risen, boycotting X over my disagreements with management has grown both more costly and less effective. Should the AI crowd ever move elsewhere, I will happily follow them. But I've become convinced that it's important I engage with them, even if doing so makes me feel like I'm co-working inside a porta-potty. I'm thinking of this as an experiment, and I may abandon it at any time. I'll continue to post on and otherwise support alternative social platforms, but I also need to spend time on the platform that still, despite my best efforts, matters most to my beat. I'm not going there for a good time, and I hope not to be there for a long time. For the moment, though, I will be there, and I wanted you to hear it here first. That's Platformer Plus for today. This episode was written by Casey Newton and produced by Lindsay Chu. Have feedback for us? Email Casey at platformer.news.