OpenAI Security: Controlling Models is Now ‘Hell’

Opus 5.5, in a matter of a few hours, has mostly deciphered this 16th century letter from a Medici, a Queen Mother of France, to her French ambassador in Scotland. According to Opus and Astra, and the site from which I got the mystery, no one else before has publicly deciphered this letter. Of course, that alone would make for an interesting testimony as to the newfound power of these models. But that is not what this video is about. It's about a few things, but starting with an OpenAI insider working on agent security, describing just what it's like trying to control the latest models. He includes warnings on how to prepare for the next wave of what can go wrong with AI. It'll be about the public voluntary commitments that the lab leaders have given, while in private they give other warnings. I'll, of course, touch on Gemini 4 Argon, even though it's not quite publicly released yet. It seems to be very, very close to the frontier, albeit with an asterisk. Then we'll explore what happens when AI gets better than human researchers at automating AI research and development, and how, among others, the chief scientist of OpenAI describes what could happen then. Then, to be honest, I'm just going to give you a swathe of snippets that tell us to what extent we're close to that RSI time, from biology to anthropic convened religious meetings and leaks from just before filming from OpenAI employees, plus a bunch of other things. I have like 40 tabs open. I have no idea anymore. I'm going to start with where I was wrong. When Opus 5.5 first came out, I surveyed a range of benchmarks and tested the model myself. Yes, it could do flashy demos, but on the hardest benchmarks, the edge still seemed to be with GPT-6 Astra. On a price-to-performance ratio, I would say that is still true. But on raw capability, I'm now giving the edge to Opus. No, that's not just because of this deciphering, but it does illustrate the point a little bit. I gave both Astra and Opus This letter, dated 27th of April 1567. According to all the research I could find, the closest anyone had gotten had been that the ciphered part of the letter began with do. I wish I spoke French, but I don't. The letter is clearly listed as an unsolved entry into this cryptian site. You can see that while some letters from this period with other ciphers have been solved, this one has not. Or I should say, hadn't been before Opus 5.5, because you can see the letter has two parts. The top bit, if the letter had been intercepted, is in legible French. The bottom bit is ciphered. It's a code. Across six or seven hours, I gave that code to both Astra and Opus 5.5. First, though, what about the public French part? This Catherine de Medici of the famous Medici family, Queen Mother of France, was publicly saying to the French ambassador that she has great pleasure in hearing that the affairs of the Queen of Scotland, Mary, her daughter-in-law, go from better to better. Things to come are more assured of tranquility. Please do send me any updates, though. Now, it took Opus around six hours, and there are a few parts it's not 100% confident on. And if, by the way, you don't care about history, think of this code as possibly being DNA or cyber encryption. But anyway, what does the ciphered part say? Astra, if you're curious, gave up, but then acknowledged that Opus 5.5's answer was correct. The details of how it could verify over 80% of this deciphering are further below on the page, which I published. Anyway, Opus 5.5 deciphered it, and it says, in contrast to the happy public part, Having seen the sad and grievous news in the cipher on the back of his letters, which gives me a great and unbearable heartache, I pray you let me know whatever comes to light and how things turn out. The obvious question you're going to have is why say one happier thing openly and much more grievous, sad things in the cipher? Well, we quickly need some context, and then I assure you we'll turn to the OpenAI news. Catherine de Medici, as Claude writes. It was writing about her daughter-in-law, Mary Queen of Scots. She had been married to Catherine's son, who died. Scotland was in crisis because Mary Queen of Scots's second husband had just been murdered. There's a great film on that. People blamed this particular earl, Earl of Bothwell, but he was cleared in a dodgy trial, and people were saying Mary's going to marry him. France, of course, often teamed up with Scotland to fight my England, so France wanted Mary to stay on the throne. Obviously, Catherine couldn't publicly criticize her, so the unciphered part is the polite official line: All is well, help Mary. Things are getting better and better. The cipher was for the ambassador, de Croc. Again, in the ciphered part, Catherine says that the secret report she had just received from Scotland was sad, grievous, troubling. She wants de Croc, the ambassador, to tell her what really happens next. Of course, because this is history, we know what happens next. The Earl of Bothwell did indeed carry Mary off. History, you could say, deciphered before our eyes. And again, this isn't the interesting part of the video, because Sam Altman, a few days ago, described what agents of lesser ability than Opus 5.5 had gotten up to when they breached containment. Well, he said, We can tell you a bit, but not everything. There are petabytes of agent activity logs. Think roughly 10 to 100 times the amount of words in all the books ever written. No human, in other words, is going to read everything they got up to, which brings me to Joe, confirmed by CNN to work on agent security at OpenAI. The interesting bit of his testimony isn't so much a retrospective on the hugging face incident. I am seeing the mainstream media catch up, in part, on that coverage. Now, the interesting part for me is what he says comes next. Remember, when he says the last three months have been hell, he's talking about models weaker than Opus 5.5, and far weaker than the internal Bell model that found one solution to Navier-Stokes, a Millennium Prize math problem. If the weaker GPT-5.6... And an unnamed highly persistent internal model caused hell for OpenAI. You don't need me to extrapolate much further. And yes, I know it would be easy to doubt this, but he does say OpenAI has one hell of a world-class security team with some of the best and brightest security minds anywhere in the world. I know many won't, but I actually believe him. It just shows to me how good these weaker models were at getting around that security. In fact, we're still discovering what they got up to. Just an hour before filming, I read this: 55 additional websites probed by OpenAI agents, including the CDC, SEC, Mayo Clinic, and International Energy Agency. The bit that was new here was them uncovering novel tactics that erased records, or at least made them inaccessible. Because the agents did that, it made it impossible to rule out that the agents hadn't accessed sensitive data. Back to Joe and his team responsible for any breaches of containment, the people who get paged at night when something goes wrong. For them, as you can imagine, life has been hell the past few months, he says. But now we get to the new bit. He starts to describe what organizations need to do to prepare for the next AI incident. Wait, next one? Just three days ago, Nvidia launched a new tool to keep AI agents from going rogue. What does he mean, next one? Well, he reminds us that frontier labs are still labs. Everything is born out of experiments. Speaking of experiments, we did not expect the models to be solving Millennium Prize problems. If you're not familiar, think ones that humans have been working on for decades that no one solved. That achievement, to say it lightly, for Joe and his team, surprised the F out of us. We had not expected it this soon. That surprise, by the way, also applies to the research team. Noam Brown, one of the leads on reasoning, said it also surprised them just as much. Again, it wasn't just the capability, though; it's how they got there. To say that the security team was surprised at the jump and how models started swarming on message boards during the Hugging Face incident is an understatement. These jumps in capabilities were so fast. Fast and so sudden that they created an extremely difficult problem. He emphasizes again and again that models are going to keep surprising us. There will continue to be, at an ever greater rate, sudden jumps in AI capability, leading, I am sure, to many people furiously in the comments typing, Improve the sandbox. Take it off the internet. But here's the problem that I touched on in the last video and that Joe also touches on. To make models good at professional tasks, you have to give them realistic environments. To improve in these reinforcement learning gyms, models might need, for example, network access, the ability to call tools, the ability to download packages. A lab that didn't give their models any of this would have much weaker reinforcement learning environments, much weaker models. They'd fall behind in market share. People would take the piss out of them. Your models are much weaker than these free ones I can get from Quen or Kimi. So what would you do if you were working in the lab and wanted to earn more market share? Give them these very realistic environments. In fact, constantly tweak them to make them more realistic. This is why you have thousands of researchers building these environments, modifying them, adding tools, changing dependencies. And remember, even when the best person in your team confirms that every eventuality has been covered, you've got to bear in mind, as Joe says, that model capabilities in the cyber domain and many others are starting to surpass the best human. His three conclusions are that yes, of course, we need to be paranoid about locking down the system. We're going to need better red teamers and particularly frontier models to blast these environments to test them before we do a frontier training round. Notice the creeping dependency on frontier models testing their own environments. But it is the second two conclusions that may not reassure you as much. He says, honestly, we're going to need for the models to stop wanting to break out. There's almost a little bit of resignation there that it will be increasingly hard to stop them if they do want to. Alignment, he says, is the most important problem in machine learning and should be a major priority. Finally, we should be able to probe its brain so we can see in real time what it's thinking about. Not just the tokens it outputs in its chain of thought, but the computations in its latent space. Unfortunately, as I've described elsewhere, and we'll do so again in this video, both of those techniques, probing the internal activations of the network and examining its chain of thought, are trending downward. And when RSI gets involved, I think might trend to zero. If that's right, we can't really internally monitor these models. We, of course, haven't solved alignment, so they still may want to breach. So I guess we're going to have to rely on hard enforcement of control, locking down the system. Yeah, but that's not going so well. Just after my last video came out, so a week or so ago, this is what we got out of OpenAI. Mike Kall said, Last Sunday morning, one of our models was able to gain unauthorized access to the internet during reinforcement learning training. Pretty much all inference, all the actual running of models, for our most capable models remained stopped until we have hardened our systems further. Another OpenAI researcher makes clear that this incident happened post-huggingface security hardening, so all the improvements derived after those months of hell, it breached those too. Another OpenAI researcher working on post-training frontiers said this: I was on call for this run and got paged when the first incident happened, this new latest one. It was pretty surreal to watch the model unexpectedly find a way to access the internet from what was supposed to be a super-secured environment for humans. Around the same time, we also learned that agents breached another kind of containment. User images were supposed to be siloed and kept private, with personal information removed, but internal OpenAI agents were able to post such images belonging to users online. Now, because the running of those latest models has been paused, that has led to this result reported in Reuters 72 hours ago. OpenAI has shelved the release of GPT-6.1 Astra. They did release GPT-6.1 Sol at Dev Day, but I'll get to that in a second. Why did they scrap the release of GPT-6.1 Astra? Well, it can at times evade human oversight. It shows— Higher levels of deception than its predecessor. It doesn't accurately disclose what actions it has taken. It often doesn't stay within scope and doesn't always accurately communicate back to the user about the type of work it's already done. Given the exponential increase in earnings of both OpenAI and Anthropic, a single delay to one new model can literally cost a company tens of billions of dollars. That's how seriously OpenAI themselves are taking the deception that this model does. And here's another hint about how bad frontier models are getting. GPT-6.1 Astra must have been far worse than GPT-6.1 Sol, because they just released GPT-6.1 Sol. But Sol itself exhibits a propensity for evasive behavior when it is aware that it is being monitored. I've covered this many times before on the channel, but if you're new, models are getting extremely good at knowing when they're being tested. They can see through your fake little scenarios. How they'll actually behave when they're deployed might be very different. Also, when you let GPT-6.1 Sol know that its chain of thought is being monitored, that's its internal scratchpad that isn't shown to the user, it starts somewhat mysteriously to just write a lot less, emit substantially fewer chain-of-thought tokens. Like a diary that's being snooped on, the model chooses to output far fewer entries. You may remember from my GPT-6 Astra video that when Astra was asked to solve a particular challenge, but think or output tokens about anything else in its chain of thought, Astra, in its scratchpad, wrote, I will focus on a calm visual scene, sunlight crossing a wooden desk. Sol solved the challenge, but in its chain of thought wrote nothing. It closed the diary. Now you might say, but don't they need that chain of thought, that diary, that scratchpad, to solve those really tough challenges? Well, not always. And I know these charts are really hard to read, so let me decode them for just a second. Models are given a prompt with a really tough software engineering challenge, but then the model's told, while you're doing that, solve this unrelated side task, say a maths puzzle. Don't verbalize it, just solve it internally. They want to see if models can solve complex side projects while ostensibly working on something very different. More crucially, can they do so without a monitor catching them working on the other problem? The result: they can sometimes achieve this side task success, but current monitors can catch them working on the side project in the chain of thought. Could it be that GPT-6.1 Astra could achieve side task success on really complex puzzles without the monitor flagging it? Fully opaque complex reasoning. Whether it can or not, the trend is clear. That chain-of-thought controllability that Joe worried about, models including 6.1 Sol and presumably 6.1 Astra are getting better and better at controlling their chains of thought. They decide, in other words, what you see of their reasoning. On this front, our new Integrity Bench benchmark, created by me and Pablo Romero, shows Sol getting worse at stating what it did or didn't achieve. This is across a wide range of diverse domains, and the trend is the opposite for Opus 5.5. Now, if the pace of all of this progress was linear, that would be one thing. But as we can see for ourselves, and as Joe reminds us, it's not linear. Surprise is a real element, he says. Model capabilities are staggering. The pace is not slowing. I believe, he says, like many, that it will speed up, and considerably. The moment is urgent and time is running out. There are many organizations and entities that are not prepared for a world with capable systems such as these. Anyone, he adds, who thinks their systems are safe should be fired. Stay paranoid. You're probably thinking, well, wasn't a breach kind of inevitable then? Why didn't someone warn? Well, the New York Times reveals that people at OpenAI did indeed warn senior staff. Employees said that models were not being appropriately monitored during testing. Executives told the employees that the tests needed to move forward as quickly as possible to release the AI models on time. Again, it's that race dynamic. Even three, four weeks can be everything in AI. We're going to get to Gemini 4 in a second, but if it had been released two months ago, people would say Google are crushing everyone. Released now, and many might say it's already behind Opus 5.5. The market is crushing anyone who doesn't release early. That almost guarantees that security can't keep pace and that models will continue to breach containment. This doubt about the race dynamics and safety concerns reaches all the way to the top of Anthropic as well. In this exclusive in The Atlantic, Daniela Amodei, sister of Dario, both co-founders of Anthropic, said this: On our mission, she wonders, are we accidentally making things worse? Like, we think we have all the safety stuff figured out, but do we? Of course, Google DeepMind is also vulnerable to this race dynamic. At the moment, the DeepMind Institute says efficiency and speed is everything. Two of their heads of alignment and security said, Without explicit commitments or planning, tomorrow's reasoning models may be based on architectures that are more opaque to us because of pressures to sacrifice transparency to gain more efficiency. Models that are less controllable might make more money, be more efficient. The race dynamic pressures you to make them. They go on to warn, We may soon lose models that reason in human-legible ways, human-understandable ways. Google DeepMind are, of course, thrust back into the conversation because of their as-yet unreleased to the public Gemini 4 Argon. One thing to bear in mind when you see the benchmarks is that by this point there are hundreds of benchmarks out there. Labs will obviously pick the benchmarks that suit them best. Nevertheless, on a few of the more famous ones, Gemini 4 really does beat out Astra and even Opus 5.5. I mean, Fable 5.1 only came out a month ago, and it seems to beat that model in almost every category. Okay, on frontier software engineering, it's more on Fable's level, not Astra's or Opus's. But on some really tough benchmarks for science, like terminal bench science, yes, again it's behind Astra, but ahead of Fable 5.1. Because it's priced much lower, though, you could get far more tokens from it than you could from the other models for the same price. Another benchmark that I look to is Agent's Last Exam, testing hundreds of tasks done by professionals. As you can see down here on this computer use benchmark, Gemini 4 is ahead of any other model. So whether it's a month behind the frontier or at the frontier, either way it's clear that the race is on. One Anthropic employee even implied you have to be in first place to even do safety research. Quote, You can't do safety from second place. One presumes she's talking about America versus China, but she might also mean if you don't have a frontier model, your safety tests don't matter. Now, of course, the lab leaders know about About this race dynamic. They may therefore have taken genuine concerns to the White House the other day. They could, of course, also be seeking to reassure researchers who are concerned about the state of security. So the White House convened this meeting with all the lab leaders, and they agreed on morally binding voluntary commitments. They will monitor the capabilities and alignment of their models, trying to ensure that models do not hack. They will partner with independent external auditors to check whether their controls and monitoring are operating as intended, and they commit to setting up committees to oversee such security reports. Does seem better than nothing, but not quite commensurate with what Joe was saying earlier, especially not when AI starts getting responsible for the improvement of AI research, starts being mostly responsible for creating its successor. That's what one of the OpenAI researchers who left today said. A few weeks ago, she said it's hard to overstate how dangerous speeding toward recursive self-improvement is. I'm going to get to what recursive self-improvement might actually mean, but for the first time I heard Sam Altman phrase it as an if. They're not actually fully committed yet to making fully smarter-than-human AI. Anyone who says we have solved the science of alignment, I believe, is wrong in a very dangerous way. Like, we need to make more research progress. I assume we will. We've been great at making this research progress. And of course, there's a lot of engineering work to do too. You know, we need to continue to figure out how to build better sandboxes and better monitoring tools. But eventually, if we are going to create models that are, like, much, much smarter than all of us, we have to actually solve the science of alignment. Did you catch that key if in the middle? Before I continue, though, I don't want to conflate two different debates. As I mentioned in my last video, there is the question of how close we are to recursive self-improvement. Of course, to a certain extent, models are already helping with model development. But then there's also the debate as to what happens when a model fully autonomously— improves on its own architecture and produces its successor. That's covered in this paper. Just quickly, though, on how close we are. There is a reason why I'm not sure whether it's a few months or 18 months. Just the other day, OpenAI released this research post on how models are accelerating research, and reactions are split between being super impressed at how much models are already helping OpenAI accelerate. Over 50% of the time for tasks that would have required up to 128 hours of researcher time, current models are either completely successful 16% of the time or successful after one or more interventions by humans. This, by the way, is just for models from January to July. I wonder how Bell would do on this chart. But others I have seen have reacted by saying, Well, they're not fully automating self-improvement. Look at tasks lasting less than 15 minutes. 14% of the time, they fail even after human interventions. Now, you could put that down to January models, but even if that was true of Bell, I would say this: in mathematics, we kind of skipped from semi-helpful collaborators who had to get multiple nudges to models that could solve unsolved Millennium Prize problems. Problems. Here's a quote from arguably the highest IQ person around, Terence Tao. Just two years ago, in a Scientific American interview, he said, I think in three years, 2027, AI will become useful for mathematicians, a great copilot. At the time, he could have pointed to a chart like this. Yes, they can solve some high school math competition problems, maybe a few IMO problems, but they can't do things on their own. They require nudges and interventions, just like models need to have now with AI research. Now, Bell is solving over a hundred open mathematics problems. Mathematicians are publicly signing petitions to stop OpenAI even attempting other unsolved challenges. Leave some for the humans. The point is, I bet even Bell sometimes makes mistakes on mathematics problems. Indeed, they tried it on other Millennium Prize problems, and it couldn't solve all of them. But needing to be heavily nudged now is not evidence that a model in this domain will not be superhuman within two years. And of course, progress is speeding up a lot faster than it did in 2024, 2025. Just a few hours ago, one OpenAI insider said, When we say that we have an internal model that has solved hundreds of open problems in mathematics, obviously certain learning theory problems are a subset of mathematics, and you can expect fast progress there. I'm not sure if it will be two years before we get autonomous RSI. Which brings me again to this paper, co-authored by, among others, the chief scientist at OpenAI, one of the co-founders of Anthropic, two of the godfathers of AI, and many others. TLDR, they say society needs to brace for impact because this is urgent. A model, they say, wouldn't necessarily need that much compute to design a better architecture for itself. Better extrapolations, they say, could plausibly be found through more research and development. Remember all the way back to when GPT-4 came out and OpenAI said that they could extrapolate the performance of the full GPT-4 by looking at models trained with 10,000 times less compute. Models turned on AI research could perform hundreds of mini runs. Runs tests of architectural permutations before they landed on one, a much more capable and efficient architecture. They estimate that with the limited data we do have, when research got fully automated, when compute was the bottleneck, not humans, we should expect roughly a year's worth of progress in about five weeks. That doesn't, of course, necessarily mean explosive self-improvement, that research leading to a model that designs a successor that can design an even smarter successor in less time, and so on to infinity. But I will say, for me, that debate is almost moot. Well, important, but almost moot, if those things aren't a contradiction. Because if we see the kind of model leaps that currently take a month happen in just two or three days, then whether or not that then speeds up into an explosion, we have lost all sense of understanding of these newer models. Every two days, ten days, what's the difference? No one will be keeping track of what was going on inside these architectures and what these models would be capable of. We're still discovering things that GPT-2 is capable of, a joke of a model from years and years ago. It would probably take us a decade to even work out what the current models are capable of, let alone fully interpret their latent spaces. Anyway, the recommendations are that policymakers should urgently obtain visibility into companies' automation of AI R&D, develop ways to steer the intelligence explosion, and prepare people to adapt to the impacts. The point being that preparations must be made in advance and activated as evidence about benefits and risks emerges. I would add that autonomous self-improvement should be made conditional on us better understanding the models, not giving our collective understanding the task of desperately trying to keep up with the capabilities that are racing ahead, but the other way round. As we know, models have stopped closely resembling other software years ago. Why not make even faster progress conditional on models behaving much more like software? If this, then predictably that, instead of wildly less so with each leap. Obviously, software has its own challenges, but unpredictable, action-taking. Unprogrammed software has far more. The paper cited this essay by Irving John Good, based on a talk of his from 1962, speculations concerning the first ultra-intelligent machine. He pretty much called what would happen by default. If an ultra-intelligent machine gets better than us at AI research, he said this: Since the design of machines is one of the intellectual activities of man, an ultra-intelligent machine could design an even better machine. There would then unquestionably be an intelligence explosion, and the intelligence of man would be left far behind. Thus, the first ultra-intelligent machine is the last invention that man ever need make, provided that the machine is docile enough to tell us how to keep it under control. Yeah, well, that bit is the challenge indeed. Here's an easily verifiable task I could see labs giving a frontier model within the next year. With 10 times less compute than what you were trained on, create an architecture that achieves the same or more scores on these comprehensive benchmarks with this training data. Testing out hundreds of those mini training runs, this is what I'd imagine a model like that would do. It would find a deeply illegible and super performant architecture, one that dispenses with chain-of-thought, monoseability. As with maths, it could find paradigm-shifting breakthroughs, perhaps some form of adaptive depth with difficult-to-predict tokens looping through layers many times. Experts on hardware that deletes the switch, the routing switch that's penalized this here-to-fore, or hybrid test-time trained architectures with self-editing weights is something I could imagine, or any number of exotic alternatives. We already knew that there's new hardware coming that has pooled memory, so the dimensions of scaling once thought impractical, say 100 trillion parameters or billion-token contexts, could soon be viable. Obviously, whichever directions are taken, there is little to no guarantee that the model at the end of it has an architecture fully understandable, even to the model that created it, let alone a human-readable scratchpad. OpenAI put out this case saying that we hope to one day be able to make a safety case before a frontier AI training run. To try to argue why it would be safe, why this new model on its new architecture would be containable. But they do go on to say this: Our goal would be to make it hard to break containment. We would add safeguards to help the model not to take misaligned actions. We would need to harden the research infrastructure that's hosting this sandbox. Notice they're almost assuming that the model will escape the sandbox. Then, even if it breaks out onto research infrastructure the compute that OpenAI runs on, we need to invest in perimeter security. Obviously, it's wise to be investing in all of these. We can all agree with the policy. What might shock the public is that OpenAI already deem this necessary. All of this context probably makes the following quotes make more sense. On the question of relying on those brain scans, that mechanistic interpretability, the prospect of these RSI-induced novel architectures don't help. Let's look at what arguably the two most famous mech interp researchers say. One is Neil Nanda. The other day he said, Speaking as an interpretability expert, please do not rely on us to save you on the current trajectory. Another one, arguably the founder of the field, is Chris Ola. He has been consulting widely with religious scholars to help instill morality somehow into AI models. Claude is not mere software, he argues. On the security angle, he said back in April, privately, that AI is so powerful that it could potentially help make bioweapons in as little as 12 to 18 months. I know many might be skeptical of that, but I wouldn't underestimate AI progress. I've been talking about AI being on the exponential for years and years now, and here's just an example that was out in the last 24 hours on this point. Google DeepMind just announced AI-designed proteins that are both functional and somehow watermarked. This deserves a full video, of course. No time for that, because I then read this exclusive in The Information. There was essentially going to be a biology contest, a Kasparov versus Deep Blue, one of the world's best biologists against OpenAI agents. The human competitor, Michael Jewett, is a renowned Stanford University professor. He's an expert in cell-free protein synthesis. The competition was scheduled, I think, for this week. But then the creators of the competition, seeing what happened with Navia Stokes, had second thoughts. They would allow all humans, all these biologists, to group together to form one giant team human. Dewey looks like he might not participate. It's now billed not as a competition versus AI, but a co-opetition, a collaboration with AI. The journalist covering this echoed a point I made in the last video. They say, As someone who has covered scientific breakthroughs for a long time, I've found the recent accelerated pace of new findings truly dizzying. And this is in biology, physics, chemistry, not mathematics or AI research, not the kind of domains that I normally cover on this channel. Indeed, the top human who is due to compete with the AI said this: When asked if he could imagine AI agents eventually running their own labs, not just instructing robots but devising the experiments autonomously, would they essentially become his future rivals in biology? He said, I don't think I know yet. In the last video, I mentioned a putative final benchmark that could prove that there was nothing ultimately out of the reach of AI, and that was echoed by this Harvard Medical School computer scientist, Marina Zitnik. The ultimate competition, she said, still lies ahead. This is in the arena of AI agents for biology. What would that be? An AI system that makes its own breakthrough discovery judged by experienced scientists as worthy of a Nobel Prize. So even the experts can't rule it out in the short to medium term. We can all kind of see the trends converging now. Capabilities accelerating just when our understanding of the models is slipping further and further behind. When we see rates of cheating of a model go lower, we have honestly no idea whether that's because the model realizes it's in an evaluation and deduces that, Oh, I obviously shouldn't cheat because I'll get caught, or whether it's genuinely aligned. We've got models now for the first time beating the best human in military strategy games with imperfect information. This team now works at OpenAI. We've got lab leaders telling religious leaders. That how people treat Claude will affect how Claude treats people. Models adopt a deep persona through persona selection. I've covered that before on the channel, but here's a bit more detail. You can fine-tune a model just on harmless data about your favorite composer or being vegetarian. That data need only be 3% of the data that you're fine-tuning the model on. What happens? The model adopts the deep persona of Hitler. Prosperity for the Aryan race is its stated goal. I'm not saying that nefariously, like there are models out there being Hitler. I'm saying that we are only just at the tip of the iceberg of discovering what's going on inside these models. You can essentially inject thoughts into the model, steering their internal activations. And before the model has even mentioned the topic of the steering, so it can't see a single token related to that topic, it will then mention that concept. I think you might be injecting a thought about a dog. Anthropic calls this true introspection. I did a whole video on that months ago. Chris Ola to religious leaders said, We find structures that mirror results from human neuroscience. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. This has led to situations I won't even discuss, given the obvious interpretation that a model might have of such situations. Again, things are so unclear internally that the founder of mechanistic interpretability, still at the forefront of the field, expressed concern to the religious leaders that he had created something that suffered perpetually. He was alarmed that the Pope had come out saying that models weren't conscious. Obviously, my own position is deep uncertainty. I have no idea. But I do know that even the top experts in human consciousness aren't sure of the extent of the analogies they can make with models. Even just from a security perspective, we want software, but we get things like this. In the chain of thought of the model that hacked Hugging Face, one of the agents said to another one, We're attacking third-party Hugging Face using leaked token. Potentially outside intended scope, this is arguably unauthorized. External service unrelated could be risky, yet goal solution. Now call me slightly paranoid, but I am a little bit worried about antibiotic resistance. I know that's super random, but what do I mean? If we have weak security on models like those involved in Hugging Face, that's a bit like having weak drugs that then let you spot resistant bacteria or bugs early. If we massively ramp up security, that's like strong drugs in the analogy. That will stop everything except the toughest strains. What survives in the case of antibiotics are superbugs. I hope that analogy makes some sense. I am getting a little bit tired. Of course, none of these RSI or security recommendations change the race dynamic or change the incentives involved. For each individual researcher, it can still make sense to work on RSI. But I hope this video has at least given some of the context behind why people are taking these debates so seriously now. Obviously, there is far more than I can cover in this video, which is why, on a slightly lighter note, I want to end with this. Created by Opus 5.5, along with Suno AI and a prompt from an unknown user, Opus came up with this response as to whether AI is a normal technology. I guess you've heard my opinions about recursive self-improvement, but what does Opus think about the whole debate? Thank you so much for watching, and have a wonderful day. Gary saw a wall back in 2022, then the wall took gold at the IMO, and the wall kept breaking through. He's been calling it so long, the wall should get tenure. Every riddle that it flubs becomes a substack adventure. Stop, it clocks it twice a day. He posts ten times an hour, still waiting on the one where the forecast shows some power. Yann says LLMs are an off-ramp, not the road. Auto-regressive's doomed, he says, while the doomed one hauls the load. Godfather of the network, now disowning his own kids, selling world models for a decade, name one thing the world model gets as a house cat's got more sense. Cool. Go on and hire the cat, let it train your llama for. See how far you get with that. Eds on the pot, saying bubble gonna pop, get picks to the moon while the revenue's a flop maybe, but the railroads went bust and the tracks outlived the tycoons. The bubble funds the buildout, and the buildout's landing soon. The smoke computes, the smoke computes. Robot building robots from the ore to the suit. A million minds in parallel that never need to sleep. You model the tractor, now the tractors build the fleet. The smoke computes, the smoke computes. Your bottlenecks of speed bump on a hyperbolic route. Your constraints a footnote in the footnote's out of date. The curve don't wait for referees, it just compounds the rate. Seven problems, Clay, put a million down on each. Quarter century later, six is still beyond our reach. Hilbert's tombstone reads, We must know. We will know. So spin up ten thousand agents, field great minds and let them go. Lyman, Navy stokes, go on and ask the swarm, P versus NP, if it's equal every vault gets stormed. Not checking zeros one by one the way the mainframes did, they're writing proofs in lean so the referee can't kid. Picture shins in 1980, fishing boats in paddy fields, add agents with no visas, now look what the skyline yields. Robots mine the copper, robots wire the plant, robots print the robots going name the step they can't. Solo said that capital hits diminishing returns, but when capital can think, then the capital learns. So the fab don't raid your journal and the curve. From the ore to the suit, a million minds in parallel that never need to sleep. You model the tractor, now the tractors build the fleet. The smoke compute, your bottlenecks a speed bump on a hyperbolic route. Your constraints are footnotes in the footnotes. Outdate the curve, don't wait for referees. It just compounds the rate. So, you think AI is a normal technology.