Why performant code matters (but gets widely ignored), with Casey Muratori

Why do most devs not care about writing performance software? And should we? Today's guest, Casey Moratori, spent the last decade arguing that we should. He also says that most software out there runs tens to hundred times slower than it needs to. Today we discuss why the focus on performance took a backseat across the industry, and why Casey thinks the tide is finally turning. Why you'll want to learn reading assembly if you're serious about high-performance code, and why it's less scary than it sounds. The saying, premature optimization is the root of all evil, why Casey says that the majority of people use it to avoid thinking about performance when they really should. And many more. If you want to get better at writing faster software, and become a better engineer while doing so, then this episode is for you. And if you're one of the people who hit a like on this post by Ryan asking for this episode to be more about programming than about AI, this episode is also for you. This episode is presented by Antithesis. If you work with agents, your job is no longer just writing code, it's also specifying and testing it. And Antithesis is the most effective method of verifying agentic code today. This episode is brought to you by Sentry. You probably already know what Sentry is because you're a developer, if not, just ask a dev and they'll tell you. I use Sentry to monitor the backend of the pragmatic engineer for any and all events and errors. Of course, Sentry doesn't only do errors, they also have logs, replays, spans, profiles, metrics, and more, because they're all connected to the same trace. One new capability Sentry has I'm really liking is its ability to fix errors. Let me show you. Here's a list of errors on my admin backend. There's a recent error on auth that I want to check out. Let's have Cyr run an autofix for us. Cyr is Sentry's AI debugging tool. First, it generates a root cause analysis. It's finding some problem with HTTP versus HTTPS URLs. Cool. Now that we know what's going wrong, Cyr can create a plan on how to go about fixing it. I could go and edit this plan, but I'm happy with it, so let's create the actual code fix. Here's the code fix that Cyr generated. Assuming it looks good, and in my case it does, let's draft a pull request. And boom, the PR is created, ready to merge. What I love about autofix is how Sentry went from showing a list of errors inside my application to offering me a fast way to fix it and close the loop while I stay in charge of this bug fix the whole time. Debugging got a whole lot faster and a whole lot easier. Check out Sentry at sentry.io slash pragmatic and start detecting errors, diagnosing your root causes, and fixing issues and regressions today. All right, Casey, welcome to the podcast. It's so nice to have you here. Thank you so much. It's great to be here. Thank you for the invitation. Now, I want to go back when we start from the beginning. How did you get into tech, programming, computers? Well, I guess computers, it's like very, very early on. My dad was a programmer at Digital Equipment Corporation, which is a company that people will know if they studied computer history, but would not know if you just looked at the landscape today. They're completely gone, right? They got absorbed partly by Intel, partly by Compaq, I think. They kind of got broken up. At that time, it was kind of a really big computer manufacturer, you know, computers like the PDP-11. That's a Digital Equipment Corporation computer. The VAX, like things that you may have heard of in computer history. Oh, these were these massive mainframes. Yeah, mini computers as well. So like smaller also sometimes than mainframes, like the kind of next step down, right? And so in general, that era, my dad was a programmer there. He would later end up at Intel because, you know, like I said, parts got acquired. He never actually left his job. Ended up at Intel through kind of digital's eventual demise. But as a result, we always had computers at home, even though at that time, you know, that might have been a little bit odd. You know, I learned to program when I was seven, which would have been in like, you know, 1982 or something like that. And so at that time, you know, maybe you might have an Apple or Commodore kind of computer at home, like some kind of early computer. I don't remember the exact dates of those computers, but most people didn't. And it was only until a little bit later that you would. And you probably wouldn't have had a programmer in your household to teach you, more importantly, right? So so I learned really early on. And that's when I got into computers. How I got into games was I ended up randomly interning at Microsoft. And I met people there. And like I went and, you know, sort of went off into games through them. That was that was how that happened, if that makes sense. How is the Microsoft games relationship? That's not kind of a given, right? They have one game, right? It's Flight Simulator. Yes, at that time, it would have been very weird. The reason that happened was a guy called Chris Hecker, who a lot of people don't really know his history, because there was sort of two waves of bringing games to Microsoft Windows. Because I mean, now it's it's funny to think about now, because people think of like, what platform are you going to play games on if you have a PC? Windows is like the default. Linux is now an insurgent to that. But Windows is the default. And so you think about how did that happen? You know, because that wasn't the case. If you were back in the early days, Microsoft Windows, not a gaming platform. It's almost nothing on it. It's like Solitaire and Minesweeper and a few other sort of games didn't happen. So what happened? The reason for this, this kind of gets into a technological reason. The reason for this is that it was very hard to actually produce images that could be displayed on the screen quickly. And to understand why this is, is like its own kind of topic. But in general, you can just imagine you have these, this operating system running, which is Microsoft Windows. It's controlling the graphics card. It's often running at a fairly high resolution compared to what a game might want to run at. You have to negotiate with it in order to display your bitmap in some way that won't destroy all the things that it's trying to display, so on and so forth. Early versions of Windows, up through like Windows 3, Windows for Workgroups, let's say, if anyone remembers that name. Was that after 3.1? It's 3.51, I think is what it was called. Or maybe just 3.1. Yeah, there's NT 3.51. No, so it's like, I think you're right, 3.1. 3.1, I don't know, something like this. Yeah. Windows for Workgroups was, you know, around that time. That version of Windows, which is in the 3 series, didn't really have a way to quickly use the CPU to fill pixels, which is what games need to do, right? There's no GPU acceleration really at this point. There's a little bit we could talk about, but it's not mainstream. It's not in consumer. So they need to be able to do this sort of thing. They need to do it in a double-buffered way so they can draw to a back buffer and then show it to the screen. And that has to happen very quickly. And there just wasn't a way to do this in Windows. In Windows, you had to kind of go through this API, where you would produce sort of a bitmap that wasn't necessarily in the right format for the display you were using. And then it had to do a translation from that bitmap to the other one when it displayed it, all this sort of stuff. So games on Windows, like things like Doom, they're not coming to Windows, right? That kind of future was not in the works for Windows. And that was kind of what some people in Microsoft wanted to change. And one of the people who brought this change about was a guy named Chris Hecker. And he was like, OK, we could actually just make a library that did the fast blitz to the screen so that we could have a way that people could do these draws and get them on the screen quick enough to make gaming viable. Would it be 100% as fast as DOS? Probably not. But could you actually run some of these new games? You know, Wolfenstein 3D, I think, would have been out at this time. Doom was kind of on the horizon and that sort of thing, right? Yeah. And so he started this project that he was not supposed to do. He did not have the authority to do this called WinG, which I think they stand for like WinGraphics or something like that. Total Skunkworks project. He had cover from his manager. I'm assuming I can say all this stuff now because it's ancient history. He had cover from his manager's name. Guy's name was Michael Edwards. And Michael Edwards basically just kind of ran cover for this, which is a thing that probably wasn't going to fly because they were in a division at that time, which would have been Microsoft Research today. It was called AT, Advanced Technology. It was the early version of Microsoft Research. You were not supposed to be shipping core libraries for Windows. It had nothing to do with that. Long story long, what ended up happening is that product did ship. WinG, I guess you wouldn't call it a product. It's an add-on for Windows. It did ship and it was the first step towards DirectX. People forget that. WinG was the first way you did this. And then eventually we had DirectX and DibSections and Win95 and all that sort of stuff. When I went to Microsoft, amusingly, the person I was supposed to be reporting to as an intern was Michael Edwards. When I showed up first day, along with the other interns, guy named Rudy, a guy named Rajiv, we were all supposed to report to him, right? Because you get a couple interns in. They're going to look at the cost of the software. They're going to look at the compliance terms of the software, the legal liability, whatever, right? And then they're just going to make a purchase decision on that sheet. They're not in there looking to see whether it takes, like, you know, whether there's a 30-second pause every time you want to try and access somebody's record, right? And I think that's very true. Like, unfortunately, the situation for a lot of enterprise software probably is that way. So maybe an individual might well be upset about the performance of that software. And I certainly hear from people all the time who are upset about the software that they use. They might not be any position to change it. I think that's one thing. There's thing two, which is that in a lot of cases, you simply have monopoly effects. You know, people aren't right now realistically going to challenge the social networks that currently exist, for example. People have tried. It's very hard. You know, Bluesky and Threads have tried to assail X, and, you know, you've got Facebook and Instagram and TikTok, and they kind of just own those spaces, right? And it's very hard to push into those because of these, like, network effects. And maybe performance could be part of a package where you try to take on one of those players, like, hey, look at how much more responsive our thing is than theirs, might be a nice plus, but that's not going to be sufficient. If you just show up with no plan for how you get adoption, no plan how you get big influencers over there, all that sort of stuff, it can't sell a product on its own into a monopoly space, right? If you're just talking about apps that someone can choose to download, maybe you've got a shot there, but those are just, they're forming a smaller and smaller subset of what software is, right? And this bigger and bigger subset is like these monopoly platforms you go on to sort of work with. So I'd say that's another thing. Thing number three is I think now people sort of are caring about performance more. I think over the past decade, the people, including myself, but many, many other people who have been saying that this is a problem have actually had some effect. Like, I don't think that it was a waste of time. I'm seeing a lot of new emphasis on performance, people talking about performance, people posting benchmarks on things. And so I actually think that the third thing is, well, actually, it kind of does seem that pointing at this issue and saying this is something we should be doing better has not been completely a waste of time. I do see things as sort of starting to turn around a little bit. I also see people attacking major product categories now with performance-based pitches, things like FilePilot or the Blink video editor, like things like this that have been coming out lately, where it's like, oh, really performant software to try to take on incumbents in a space, and they've been getting traction. So I think that's also a really good sign. Yeah, and I guess on this last category, a really good example in the developer community is Bun versus npm, where Bun just said, like, okay, we're like 10x or 20x or 50x faster, and devs are like, what? Is this possible? And then it was. So there was this outrageous claim. I wonder if, like, you need to have these outrageous claims because devs started to pay attention because it was 10x faster in many categories. You know, linear versus gyro is also a good example where linear have this benchmark of, okay, they have 300 milliseconds for any action, and gyro, of course, we know is just slow because they have a bunch of complexity. We can explain why, but it's slow. It was never built for that. Yeah, and, you know, if you think about something like a 300 millisecond budget for an operation, 300 milliseconds is like an eternity in computing, right? And so if you're talking about, like, our pitch is that we're not more than 300 milliseconds, that just shows you how the bar was so far past where it probably should have been for something. And you see it everywhere. You know, you go onto programs and you're waiting sometimes seconds for an operation, and I don't think people realize just what an eternity a second is in modern computing, especially when you're sitting on networking like that has, you know, sub-10 millisecond ping times sometimes. You're talking about this, you know, the actual packets had to travel physical distance to get to this data center, and that was being done far faster than this very simple operation that you were failing to do in a reasonable amount of time. amount of time, it's just like we are massively underperforming. People don't believe it when you say 10x, 100x, but it's actually true, and we've seen a lot of proof of it, as you point out. I do wonder if one part of the not really much focus on performance is that a lot of developers don't know the baseline thing. And I'm reminded by Simon Ericsson, the founder of Turbo Buffer, has this project called Napkin Math, where he did a list of mostly networking operations. How long does it take to transfer one byte between, like, two AWS data centers? How much does a gigabyte take? How much does a terabyte take? How much does it take to write an SSD to an NVMe, and so on? And so he had these numbers, and he said that what he found is whenever in Shopify they were deciding, do we choose vendor A or vendor B as a database, they would just run a benchmark that they would write themselves, and they would get like, okay, like, I don't know, storing this and this, it takes two seconds on this one, 10 seconds on that one, we will choose the two-second one. And he looked at it and said, like, hey, like, this doesn't make sense. Like, the amount to store in a file system, like, here's a theoretical limit, which is, I don't know, 100 milliseconds. Like, there's no way that's going to be 10 seconds. And it often turns out that he found that the benchmark was just wrong. They were benchmarking the wrong thing, and they were making decisions. So I wonder if there's a thing where many engineers, developers are maybe just not aware of how truly devastatingly slow this thing is versus the resources you have. That is the entire point of, like, my Substack, right? So what you just said is exactly true, and it is the thing that I hammer home on the Substack through all the parts of, like, the courses on there, which is that in general, there's a misconception about the way that you approach optimization in computer science or in whatever software engineering, let's say. And that misconception is that what you do is you run a profile, you identify where the, like, you know, big parts of the profile are, you make some changes to those, and you measure, like, statistics. The better the statistics, you know, the more statistics you can get, the better, and you look to see if those statistics improved, if they have, that was a good change, and you proceed as such. And this is completely not correct. That is not how anyone has ever, you know, I've worked with many extremely good optimization people, and that is not how it is done. The correct way to do optimization is very much like what you just said. You first go, what are the operations that this system has to perform? What is the underlying hardware capable of doing at its theoretical peak? And then you measure the delta between that theoretical maximum and what you have achieved. And then your goal during optimization is to shrink that gap to something that you think could be plausibly explained and hopefully come up with explanations of why you aren't at theoretical. Because oftentimes you can't hit theoretical. That's why we call it theoretical, right? And it's crucial that you do this because otherwise all you're doing with that other method is finding a, you know, with the, I'm just gonna, you know, make something I think might be an optimization and look if my statistics improve. All you're doing there is finding a local minimum. That's all you're doing. You're just, you're just, you know, you've got this shape of your performance and you're finding some little spot and you're sitting in it. That's not optimization. That's improvement. But optimization means to make optimal, right? It means we're going to find what we actually should be able to get this machine to do. And so, you know, that's why I emphasize that approach because it's the one that I've always seen great optimizers take. That is how they get good performances by knowing what the maximum could be. In addition to that, it also is what lets you become better at optimization. Because no matter who you are and no matter how much you already know, when you go to tackle an optimization problem, there may well be some things in the new, like, way that the system is laid out that you don't know about. New things that people have not figured out about modern CPUs. New things that are different about the network backplane. New things that are different about the GPU drivers. Who knows, right? And if you don't have some theoretical maximum to look at and to measure your delta from, you don't know if there's some serious anomaly there. And you would be surprised at how many times we find anomalies like this. Things in CPUs that no one knew about, and we, you know, like I've literally had them in the course of making the substack. I've been like, what is this thing? And I look into it, it's like, oh, there's this new renaming, this new rat table thing that— Software thing, like we just, whatever this massive thing that we're imagining doing where we're going to ignore optimization, we sit down and we write it, and we use a paradigm where we ask the server for something. We have, like, some API, you know, that we've built for asking servers for things. We ask the server, and it returns to us what the server's response was. And that's, like, kind of how we architect this thing. So everyone writes, you know, hundreds of thousands or millions of lines of code, and they all look like, ask the server something, do some calculations, ask the server for the next thing, do some calculations, right? Then at the end you find this is way too slow, but that's okay. You weren't worried about that because you're, like, we're optimized at the end. You call in some performance experts, they look at it and they go, There's nothing we can do for you, sorry. Why? Well, the reason is because you created a serial dependency chain. All of your code looks like, wait for a network request to come back, do something, wait for a network request to come back, do something, wait for a network... And that serial dependency chain can't really be shortened without just rewriting it. If instead you had made the paradigm and told your programmers, Look, here's what you need to do. At the top of every operation, you need to figure out all the things you might want to ask the server for. You ask them for all of those things, right? And then you do all of your processing there, and you only create a chain of dependencies if you absolutely couldn't have determined what it was you needed to ask the server for. Now you're just in this situation because you didn't tell them to do that. You have to rewrite all your code. Everyone is now going out rewriting all the code, if they even can, if it's even possible to really do that in a way that's not slower than just rewriting the thing, right? So what happened there? Well, again, we talk about this a lot on the Substack, but there's this idea of a serial dependency chain. It's when you stack things in order, right? And the performance of your software is generally determined by the longest serial dependency chain, because it's something that cannot be parallelized. If I have thing A that then B depends on, that then C depends on that, we cannot shorten that because it has to go in order. Everything waits for it. We can't multithread it because it's dependent. We can't, you know, make it run wide. We can't, you know, amortize the network requests, whatever. That kind of thing can be pervasive in the programming, and we can't cheaply remove it because it's not a hot spot. It's a way that you did things. That's the part where that kind of thinking breaks down. If every software engineer knew to watch out for false serial dependency chains, things where they were creating series of dependent operations that could not be optimized away, or other sorts of architectural problems like that that cannot be easily fixed, then the world would look more like just wait and optimize the hot spot, right? Yeah. So this is the architecture, the planning, right? Like if in that phase you're like, okay, like as this thing grows, like what will get in the way of performance? What will slow it down? Or you can ask all these questions, or like different flavors of those questions, you know, or from the other side and so on. Yeah. Another way to think of it is, because hot spots is the way that people talk about that, like, oh, it's going to be hot spot optimization. We just got a few spikes. Someone will come in and clean up those spikes and we're done, right? The way to think about it is your code base will not end up that way by accident in most cases anymore. You have to engineer up front for a hot spot code base that people can then optimize, right? And so that's the crucial takeaway, is everybody on your team who is making architectural decisions, those people must know performance, and they must make decisions that will allow the other people. Downstream of them to use an architecture which can be optimized later. If you don't do that, you're just rolling the dice. Casey just talked about how engineers making architectural decisions should know about performance. This is also true when choosing your dependencies, like which database to use. And this is where I want to mention our season sponsor, TurboPuffer. You already know how TurboPuffer is a vector and full-text search engine, but here's an interesting story from Linear on what happens when you stop thinking of TurboPuffer as a search engine and start using it as a primitive to reduce latency. As context, Linear is a local-first app, so each client keeps a local database, and when that client goes back online, it needs to catch up with what happened and do it fast. Their biggest workspaces generate around a million sync actions per day. Doing catch-ups by reading from Postgres was getting slow for large reads, so the tail latency got too large, and adding more replicas did not help either. Linear solved the problem cleverly. They started using TurboPuffer as a serving index for each client. This is because TurboPuffer itself is built on top of inverted indexes. So for every index value, it stores a document that that value can be found in. The lookup cost for such an index is constant. So Linear took this structure and had each client's index point to the changes that they needed to sync. As a result, not only did they reduce latency, but they kept it constantly low no matter how long the change list to sync to the client is. Linear published a blog post about this refactor titled Rebuilding Linear's Delta Sync Read Path. Check it out. I love this story because it shows how important it is to choose the right primitives and how good primitives can improve your system. If you're building systems where you store a lot of data or serve a lot of data, TurboPuffer can probably speed things up or save on your costs. Learn more at turbopuffer.com/pragmatic. I'd also like to talk about our presenting sponsor, Antithesis. While Casey deliberately does not use AI coding agents for his work, most of us do. And when you work with coding agents, your job is no longer writing code; it's specifying and testing it. Antithesis is the most effective method for verifying agentic code today. Let me explain how it works. Antithesis runs your whole system in a hostile simulator. Simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, Antithesis doesn't only find bugs, it gives you a perfect reproduction of every issue. To create such a tool, the Antithesis team needed to invent new kinds of debugging tools as well. For example, here's what's called a bug probability graph. The x-axis is virtual time and the y-axis is probability. As Antithesis runs the hostile simulations, it plots time frames when the bug probability increases, which greatly helps with finding the root cause of the bugs. And Antithesis also has a log visualizer. Vertical lines going down represent events branching off from the same state, and the purple dots are where the bug happens. Antithesis is as good as it gets in being able to ship agent-written code. It's what teams at Jane Street, Fly.io, and the Etsy community use to ship with full confidence. Head to antithesis.com/pragmatic to learn more. And with this, let's get back to Casey and how, if you don't design an architecture that can be performant to optimize later, you're just rolling the dice. And we've seen so many projects. I have an entire video where I go through, like, look at all these blog posts of people who, like, say, you know, it's Facebook, it's Uber, it's everybody. They've got blog posts of, We had to rewrite this whole thing because the performance was bad. If it was always hot spots that made your performance bad, you'd never have to rewrite the whole thing. So we know that that doesn't work anymore. Why? Because of the things I just said. I was at Uber where I was not making the decision, but the teams next to me were, and I saw or I kind of understood why they were making what. Typically, and right now what's happening with AI companies, oftentimes it's like we chose this technology, which is Python, and it's single-threaded, and it made sense at the time on the web server. But now we're big, and this happened at Uber. It was Python and Node.js, and then they went to Go and Java on the backend. And now with AI companies, it was Python, OpenAI and Anthropic are both going through this right now. They're both either public about it or I've written with Anthropic, they shared with me, but I put it out there. They use Python because data scientists or AI or machine learning engineers knew Python. They put it on a bunch of web servers. They had their API run on it. Initially, they just scaled horizontally. But now they're like, well, if we move over to Rust, right now they're choosing Rust or Go, but I think it's Rust. Well, we can actually have multi-threaded, and the same machine can actually handle more connections, so cool. I came across a lot of that because I think that's easy and safe to communicate because it doesn't look bad on you. But you're right, a lot of times, I don't think on engineering blog posts you'll get the real reason that these companies put out there. Like when it's kind of a very kind of, you know, easy-to-own mistake, or not mistake, but just the decision, which made sense, they'll tell you. But if it's something that was an oversight, you're not really going to get that on a public-facing engineering blog post, except for maybe some startups who are really there. But don't forget, like a lot of those blog posts are going to help someone get promoted or get recognized, and they will always be way more positive, especially when there's a content writer team, which— Large companies do have, so it's not quite PR, but it's somewhere midway in between. And I mean, yes, and also I would just point out the fact that, like, the fact that these things are happening, though, is all we really need to know for the signal, right? Because in general this should not be happening. If the ideas about optimization were true, you'd— is that the CPU is basically, you can think of it as a little machine whose internal gearings we are not privy to, because for the CPUs that we care about, so, you know, an M-series CPU in a Mac, a Zen Core CPU in a server or in a laptop, or an Intel, you know, Core series, those sorts of things. These CPUs are not documented at the level where you're going to be thinking about how each little individual part works. And to that end, it's unclear that you would have time to do so anyway, because these are massive, very complicated machines that we're talking about, right? But from a high level, from a more black box perspective, they are machines that we can think of in relatively straightforward ways once we know kind of what their core instructions are that they tend to execute. And they break down into a couple different categories. There's how does data move into and out of a core? And this is basically how, like, load store units work, how the cache levels work, L1, L2, L3. Sometimes we have L0 now sometimes, things like this. How does that work and why? What is the granularity of it? What is the policy? How does the CPU go about actually working with those things? Understanding that part of the machine is crucial because when you're working with a lot of data, the difference can be massive if you structure it in one way versus structuring it in another way, right? Again, architectural decisions that have nothing to do with hotspots. They're how all the data is laid out and what the access pattern is, right? Things that are very hard to change sometimes. So that's one part of the machine you want to understand. Another part of the machine you want to understand is how the instructions flow through it. And, you know, a lot of people have heard about, like, branch misprediction or things like this, I-cache misses. There's words that you might hear, but you're not really sure what they mean. They're all actually pretty simple to conceptualize. Sometimes they're harder to pin down exactly how they work because branch predictors are, you know, getting more and more complicated and so on. But you can still categorize the behavior of them and understand how code flows through it and when you might care and when you won't. And then finally, there's the execution unit scheduling part, which is about knowing what's the raw sort of throughput for any particular type of operation: floating point multiplies, integer additions, division, whatever it is that you might want to know, right? Once you learn a little bit of assembly language, you understand what it's reading. You understand how it turns those assembly language instructions into micro-operations, which it actually does, and how they get distributed through that machine. That flowchart that they put, like basically up on the We've Announced the New Zen Core, that flowchart, you can look at it and go, I know the performance of this machine roughly, right? Not exactly, because like I said, there's all these little edge cases that if you really want to be a crazy optimizer, which again, I don't really advocate people do. I don't think it's important that they be, like, crazy hyper-optimizers. If you like to do it, great, it's a lot of fun, but it's not the important part. Just look at that CPU and be like, okay, I see what this CPU should be getting in terms of, like, what I could do with this size data load, that size data load. This is what I could probably get out of it for if I was doing a bunch of, like, how to do a bunch of, like, math ops on it, you know? And I think that's just something that should be kind of par for the course of software engineering. You go to school for four years to learn this. There's no reason you can't learn this in a few months. It's not that hard. Yeah, plus I guess just from a craftsmanship perspective, like, we should know our tools. We should know the machines that we're programming. Obviously, we know that our code will be, if you're doing web, it'll be running on all these different things, or if it's mobile on all these different phones. But from a conceptual point of view, like, we should be able to know what's going on. So I feel there's a bit of a pride as well. Like, if nothing else, you would learn a bunch of stuff. Like, I know some of it, but I'm now getting motivation to learn more about it. I do think there's a craftsmanship angle. I think there's a large number of people who maybe don't feel fulfilled when they—I've certainly heard from lots of people who, when they write something and it's just kind of this amorphous, high-level thing, they don't get as much satisfaction out of it. And then when they learn how they can look more deeply at what's going on, they feel much more satisfied. Even if they didn't change what level they were programming at, they feel much more satisfied that now they know what they're doing, right? And it's like, oh, I see, and I understand why this thing was happening this way and this thing was happening that way. That's very satisfying, right? So there's an aspect of that. I also want to emphasize another part, which is that it's a percentages game. If we convinced enough library maintainers that this stuff was important and the libraries all get a lot faster, all of a sudden all the people using the libraries code gets a lot faster, and so on and so forth. If the APIs start changing to make it easier to optimize the libraries because people now thought that through, right? Like, it's infectious. The more people are doing performance, the less people need to do performance, if that makes sense. It's kind of paradoxical, right? Well, plus, I do think right now there's still an edge in just being performant. Again, you mentioned, but there are categories of software that is just winning by being much faster, and to do that, you need to do this. And if you know how to do this, maybe you're going to spot opportunities as a software engineer. Maybe right now you're not as happy in your position, but maybe start something or do a side project that turned into a full-time thing, and so on and so forth. So, like, I feel there's, like—and worst case, you just learn, like, net new knowledge, which will probably not be as outdated with AI, which we'll get into later. But this stuff, it feels—it just feels very interesting, right? Like, kind of it moves your brain. It does, and thankfully, like— It's also not that hard to update your knowledge because, again, you get these presentations that the CPU companies give, and they're like, Here's the changes we made. So you kind of are aware every time a new thing gets, you know. There's little tiny things that creep in that you don't, that, you know. But again, there's people out there who are running lots of microbenchmarks that you will find out about, and they often uncover these for you as well, right? So I want to talk about games. You've built games for, like, decades at this point. Can you give an overview for those of us who are not in the games industry? How is a game typically built from the games that you know of, that you've observed, or you worked on? Especially trying to compare for, you know, like a typical SaaS or a distributed system or something where building a website or service is kind of, you plan this stuff, you know, we'll do an estimate, we'll build it in a few months or a few weeks, we deploy it, and then we monitor it, and then we keep tweaking it, and then, you know, fast forward five years later, it's now this, like, gigantic thing with microservices. But it keeps evolving, right? Like, we do this, like, a lot of prototyping thing. For games, it's pretty obvious, right? From the get-go, as we're talking, like, there will be a launch. But can you, when you're inside or when you join a game studio, like, what would you observe there in terms of what the process is like and how it's different or how it feels weird compared to, like, this, I guess, I don't know, traditional software, SaaS software, whatever development that is? So I guess what I would say is, unfortunately, I'm probably the wrong one to ask because my knowledge is outdated at this point, because one of the things that has happened to games recently is I feel like they've moved closer, I don't want to necessarily say entirely in development practices, but at least in terms of the nature of the product. has changed somewhat dramatically to be more like something like SaaS, where, you know, if you take some of the most popular things that are being, in terms of dollars, let's say. So I guess maybe popular is kind of might be hard to say specifically, but let's just say revenue generating. So if we were to measure the total games industry revenue and you look at what are the largest slices of that, you're seeing things like Fortnite, like Roblox, like Grand Theft Auto V Online, et cetera, et cetera, Minecraft. These things are starting to look a lot more like an always-on live service kind of we ship incremental features to our customers kinds of things. And so I would actually say that, you know, I'm the wrong one to ask about what that actually looks like from the inside because I haven't actually gone and worked at one of those companies. I have friends there, so I hear things, but I'm not the right one to, like, give an accurate picture of it. But I just would point out, from my perspective, the game industry practices look different today than they did when I have a more intimate sort of experience with what we were actually building. But can we talk about it when you were building games, which was, you know, 10-plus years ago, when I understand these were the games where they were built, they were launched, you know, maybe they got a patch or two, and then they were kind of, you know, the team moved on, they were disbanded. It was this time-boxed thing that was a lot of development, a big launch, and either it went big, huge hit, or huge failure, right, and the studio goes bankrupt. So how did that work? Or clear the studio, so that risk is eliminated. And it also, I guess, it now opened up so much more people who can now have a shot at creating a game because you no longer have to either have this massive amount to license this super expensive game engine. You no longer have that risk. The only risk is, is it fun? What have you observed happen in terms of both for the industry, for development pace, those kind of things? And the reason I'm asking, because I wonder if there's going to be a parallel with AI where, okay, you know, like you needed to have an engineer who is, right? I feel games might give us a bit of a hint of what we might expect at the broader industry. So that is actually, I would say that's a brilliant analysis of the situation. For not having lived through games and for noticing that, that's impressive, I'll say that first. And I totally agree with that. I've said to people in the past who have asked about sort of AI impact on games in that sense, and I've sort of said as much. I've said, like, the licensable engine thing kind of was our AI transition already, unfortunately. And I regret to inform you that the news is not probably that positive. So there are some definite positive things that happened early on because, as you say, it opens up the ability to make games to people who could not have marshaled the technical, sort of the technical staff necessary to produce a competitive engine. And so giving them the ability to make games is a pretty important thing, and it allows a bunch of people to make sort of some artistic expressions that they just wouldn't have been able to do. Early on, that tends to be a net positive because you just have some more games coming out. Maybe some of them aren't that good, but, you know, some of the existing games aren't that good. That's not that different. But then you get some really cool games coming from some sources that just simply wouldn't have been able to do it. Thumbs up. Problem is, it rapidly kind of accelerates into this kind of a nasty scenario where you just have massive numbers of releases. And I think at this point, we're at the point where I want to say Steam games are in the, like, tens of thousands or a hundred thousand per year or something like that. It's so massive that there is no way that your game will be organically noticed anymore, pretty much period. So essentially, it's this really nasty problem where you just have the market flooded with products. And there, you know, it used to be that if you made a quality game, if it was fun, people would find it because there were so few games that someone would play the fun one and tell people about it and it would get purchased, right? Like word of mouth or just exposure on a storefront would be all you really needed to get, you know, sort of the word out about a game. You didn't need a huge marketing budget or anything like that. Fast forward to today where we have this sort of massive influx of games, again, pre-AI. It's just because now the barrier to entry is very low, and you really need a strategy to make sure your game gets found. Is it possible that sometimes, you know, a small indie game with no marketing plan or nothing will get discovered and become a huge hit? Absolutely. It does still happen once in a while. The chances that you will be that game are like zero. So you kind of now need a marketing strategy, a real marketing strategy, and going into the market for games without one and expecting to sell any significant number of copies above, you know, maybe a few thousand at best is really unwise. If you want to hit reasonable numbers of sales of a game, you have to have an idea of how people will find out about this game. So if I'm getting this right, it sounds like the game itself being good is table stakes, but not enough on its own, right? That distribution, marketing, getting people to hear about the game is much more of the differentiator because there's just too many good games out there, and now they're easier to create. I think that's exactly right. And that's just the unfortunate reality of it now. Was that a good trade? I don't know, but that's what happened, and so that's where we are in the industry now. There's this other thing that I heard about, which is how new games not only compete with other new games, but with old games as well, right? Like the other day, I spent a few hours playing Death Rally, which is a game from the '90s. And every year there's more and more good games to play. They all take away from the time that the new games have. Yes, and that problem will only get worse because one of the things that the game industry could rely on in the past that is much harder to rely on now is that older games would look dated technologically in ways that consumers cared about. And we have now kind of also crossed the threshold where there is a segment of the market where people really do care about the latest, like ray traced lighting and all these sorts of things and, you know, more photorealistic rendering or whatever it is. But a large portion of the gaming market by revenue doesn't really care what the game looked like all that much, in a sense that whatever we're doing today is good enough. So 10 years from now, if the games look much better for some reason, no one will really think of that as a huge differentiator in terms of sales. You go back to 1995, and technological advances were a huge differentiator in terms of sales. You come out with something that, you know, looks good, that takes advantage of the hardware of that day, and boy, did it look cooler and feel more responsive and all these other things as compared to earlier titles, right? And so that's also going to increase the degree to which the thing that you're talking about will happen. I can go play an older game because it doesn't feel obviously dated in an audiovisual way. I don't have to be an appreciator of retro gaming to go play something from 2017. It just looks fine, probably, right? So there's that. The other thing that I'll just mention, which we kind of already touched on, but that ties directly into your point, is that also live service is such a prominent thing now. People are just logging on and playing Fortnite for several hours or something. That's also taking away from the possible revenue that might be spent on buying some indie game or some new AAA game even. So you have these sort of incumbents, people playing Minecraft, spending their time playing Minecraft, spending their time playing League of Legends or Dota, and that's taking up a huge amount of their time. That's—it's zero-sum, right? They can only spend their hours in certain places, just like Netflix or anywhere else. They have to start thinking about, you know, they're competing with everyone else for entertainment hours. Okay, so I need to ask you this: GTA 6. How is that in 2026 at a time when we have better tools than we have ever before, and we can build software and games faster than before? Like, how do games take 10-plus years to develop? Is this some kind of outlier, or has AAA game development taking many, many years just not changed at all? What do you think is going on here? So from a player's perspective, I can understand why someone would look at it and go, Wow, Grand Theft Auto 6 has been in development a long time. How does that make sense? Or, you know, something like this. From a business perspective, you have to understand that Grand Theft Auto 6 is not a game that they are selling to players who are going to play the game. That's not what it is from a product standpoint, right? What Grand Theft Auto 6 is from a product standpoint is a replacement of Grand Theft Auto 5. Grand Theft Auto V, at the time, was, if I'm not mistaken, by far the most revenue-generating entertainment product in existence. The online part of that game was generating, like, billions of dollars. And like I said, not a game industry historian, so, you know, take what I have to say with a huge grain of salt. But Grand Theft Auto V was kind of like Fortnite before Fortnite, if you will. They were a huge, huge live service revenue-generating product. So, from Rockstar or Take-Two's perspective, right, Grand Theft Auto 6 is not just, let's try to get out the next Grand Theft Auto as soon as we can because we'll make money selling that title. It's a, we are going to replace the most profitable thing we have ever built, which is still generating a ton of money for us, with a new thing. And you can better believe that they want to make sure that they are going to do that right, because the last thing you want to do is ship a new product. That I had written for the, you know, they're not like core libraries, but the core, like, routines to make sure that, you know, anything that I could be testing for our customers, I sort of was. And so I think there's good times for testing. I would say the part that I don't like about test-driven development is the test-driven part. I don't think development should ever be driven by tests. I think tests are a thing that you should be aware of. You should know what your options are for testing, and you should make intelligent engineering decisions about tests. Now, could that decision be that for this particular project, we are going to drive it primarily from the tests? Yes, that could be a decision that you make. But you shouldn't really think of development as something that is primarily test-driven, like by default, because that might be a very bad decision for some other project where it just ends up costing you more to have done it that way. So, like with most things, I would advocate for a pragmatic approach to testing. You should understand the cost of testing, the cost of developing, maintaining the test, and the cost to your codebase if it makes it harder to change your codebase because tests have to be rewritten and you therefore don't make changes you should make. All of that stuff should be in your brain, and you should make an intelligent decision about what your testing strategy is. If that decision, intelligently made, turns out to be, we are going to have a lot of testing on this project, that may well be a good decision. I don't think there's an absolute thing you can say about how many tests there should be. Some projects probably shouldn't have very much. Maybe some projects should have a lot. And I think knowing which of those you are doing is part of being a good software engineer, is I guess what I would say. You mentioned being a good software engineer, but before we get into what is a good software engineer, what does good code mean to you specifically? So good code to me usually means that you have written something that is as straightforward to what the machine actually needs to do to solve the problem as it can be, and also hopefully that you have, I guess I'll say, properly identified ways of breaking it into easily digestible pieces and named those pieces in ways that are easy for someone to understand, especially yourself, because you are very likely to be someone who's going to have to modify it. So that's the way I tend to code. I try to identify what do I actually need the computer to do. I try to write as simple as possible the thing that will do that, and then I try to put that in terms that are, you know, I would say least redundant. So, you know, I don't want to see the, you know, the equation for Euclidean distance scattered throughout my code. I want to have a function that's like compute that distance, and I want to use it, right? I want it to then be nicely broken into the pieces that it represents, and I want those pieces to be reassemblable properly by the compiler in a way that will produce code that runs very efficiently, right? And so that's what I'm usually trying to do when I'm trying to program. And for me, I have never really understood the sort of mentality of there's a difference between code that is, like, well architected by some principles and code that runs quickly, because in my experience, usually the code that is architected properly is also the code that runs quickly. And yes, there is a point where if we decide that something absolutely has to get as close to theoretical maximum as it possibly can, yes, we will start to make that code harder to read and modify because we are now, like, really over-specializing it for this piece of hardware or whatever. That's true, but that point is, like, you know, way out on the curve. It's not the common case. Most of the time, assuming you just want code that runs pretty, pretty darn well on most hardware. The simple, readable version of the code is actually very fast. It's only once you think you need to have 27 factories and 8,000 microservices and all these things running that it starts to be this thing that's like good architecture, but also like hard to modify, hard to read, run slowly, right, all these things. So I tend to think of, like, good code, there's like this nice nexus of runs pretty darn well, easy to read, easy to maintain, isn't as close to theoretical maximum as it could be, but it's close enough. And the paths towards theoretical maximum have not been foreclosed. We left the door open with the way that we wrote it so that if someone really needs to come along and boost its performance, it's set up to do that, right. And related to this, what is a good software engineer to you? Is it just someone who writes good code, or it goes beyond that? I would say it really depends on the environment a little bit, because I think I've seen a lot of different kinds of good software engineers, and so I would liken it more to a, you know, if you want a sports analogy, you'd imagine something more like a baseball team, where it's like, what's a good baseball player? Well, it's like, are we talking about a pitcher or a designated hitter, right? And it changes quite dramatically. So there might be some things like, hey, if someone's pleasant to work with and, you know, doesn't goof off all the time and actually gets their work done, those are obviously things that we would say are true of any software engineer. You know, there are some general personality traits that might be positive. But when you're talking about things that are more specific to just software engineering and not just being a good employee or something like that, I would say I've seen a couple different kinds. I've seen people who are like the utility infielder. There are people who just, like, they can identify and go and try to fix a problem and succeed. Even if the codebase is kind of wacky and out there, they're good at getting the lay of the land very quickly, of identifying something that's going on, and they're not afraid to go in and like, okay, this codebase is kind of ugly here. It's okay, I'm going to patch around. I'm going to do what I need to do and get things done. That's a great engineer to have around. I've also seen great engineers who are the exact opposite of that. They are just like, I take this one particular problem that we have, and eight months later, I have ground out every last thing there is to know about this. And sometimes to the point of, like, producing new algorithms that no one's even known before, right, that are like these, you know, breakthrough things, right? And that's a great software engineer to have on a project if you happen, if you're going to be having that kind of thing. And so I've seen a lot of different people that I would consider great software engineers, and they aren't all the same person, right? So I think that it's kind of important if you're asking it from the standpoint of like, hey, you need to put together a team to go build this project. What's a great software engineer? I would say the best advice you could give someone in that position is think about the roles. Think about what kinds of roles there are going to be here, and don't think great software engineer. Think great that role, right? Who is going to be a great pitcher? Who's going to be a great first baseman? Who's going to be a great outfielder? Who's going to be a great this, that, the other thing, great third base coach, whatever it is, right? And that's what you're trying to put together if you're trying to build a team, to me. Yeah. So, like, it's just not one size fits all, but I still want to push you a little bit. Like, what are things that you think are non-negotiable for someone to be a great software engineer? I mean, we talked about the things that we talked about, which is a recurring theme with you, is just going deeper and deeper and understanding the next, the next layer. You know, like understand if you're doing web development, understand React. Once you understand React, understand what's going on in the DOM. Go all the way to assembly. Once you've done there, understand how the CPU is doing operations and branch predictions and some of those things. Like, to me, that's a skill of, like, curiosity, driving deeper, crafts, whatever you call it. You know, there's different ways we could do it. But along these lines, what are those traits that you think, no matter, you know, what kind of role we're talking about, but if you think back of some of the different type of roles that you work with, like, do you see some overlap that they all had something? I would say that it's pretty unusual, I guess, that— I can't think of someone I would think of as a great software engineer who, like, didn't know how to, like, read assembly or something. That is true. It might be that having that curiosity about how things work and knowing at some level what's going on is kind of maybe something that's going to be very common to a great software engineer. But I would just underscore the point: the degree to which they are employing that knowledge may vary quite a bit. For some of them, that may be their bread and butter, and they're doing that all day. Games engines arriving, and now so many more people can make games. Not everyone, but it's a lot easier to enter. What are you observing in terms of most people outside of who are still handcrafting code because they want to are using these AI coding agents for two reasons. Either it's, it just makes sense and they realize, well, this thing can now generate code as good as I did, which was a turning point in January. I had that turning point actually myself. Or some are actually just pushed with corporate mandates of like, you need to use these tools, and eventually they kind of get on board, whether willingly or unwillingly. But so many folks are having AI write the code for them. They're, you know, they're prompting it, but they're doing it. What do you observe of the effect having, you know, from your vantage point, made that be on quality, craftsmanship, on just output, speed, et cetera? What are you seeing? I think it's a little too early to assess, to be honest, because kind of as you pointed out, obviously there's been people who maybe, you know, we might derogatorily call AI shills, who have been saying that it was producing as good a code as humans for, you know, two years now or something like that, right? Yeah. But in reality, the people whose opinion I would trust more, none of them thought it was really all that usable until much more recently, right? And so we really haven't—they haven't had very many months to actually be figuring out how to use this thing or to determine to what extent they can use it and how, what it's best at, what the workflow looks like that makes it produce the best results. It seems like at the moment, I would say probably need to give it at least another six months, if not another year or something, to let everyone kind of shake out, like, what are actually the best ways to use this thing. I know tons of people in the game industry are using it, so I know that they are doing various things with it. Whether those things are the same sorts of things they will eventually think are the way they like, you know, like— The things that they're doing right now may be like, oh, that was kind of dumb. Like, you shouldn't have used it that way. You should do this other thing with it, and it's way more productive or something. So I feel like it's probably too early to assess. We haven't seen any real, like, obvious, like, oh wow, like, you know, the Fortnite ships once a week now and it's bug-free. Like, nothing particularly interesting has happened in terms of output there. But again, it's been, what, like five months or something? So it's way too early to see how it actually gets integrated into a reliable process, right? Yeah, and I know there are some companies who are now tying up, let's say, agents fixing bugs, but that's only a few months old. The oldest software that's widespread that is written close to 100% by agents is from the labs, OpenAI's Codex and Anthropic's Claude Code. But even there, it's been since November or some parts of it December, so like maybe six months. And it's different, right? That is a product they're selling. So there's, I'm not sure we'll know for sure, like, is it truly 100%? How much, you know, there's a marketing angle, but there's a self-bias there. So like, I would put those aside in terms of trustworthiness. And you're right that the rest, we just don't really have the information. It'll be, I'm sure there's so much experimentation, but to your point, it takes time to bake, right, to see the impact. Most of these things are currently presented as tools, meaning a human has to operate them at least in some way, like at least setting it up to do what it's going to do. And therefore, you have to give it some time. You know, nobody currently is selling a product where it's just like, oh, just— Turn this thing on, and it will just ship Fortnite by itself forever, and you can just get rid of all your engineers. Like, no one's actually selling that product yet, right? We could evaluate that product because we'd be like, did anyone do it? Did it start shipping Fortnite on its own, right? So if it's still something where humans have to kind of figure out how they want to, like, slot it into what they're doing, then it's entirely possible that the reason that we haven't seen some big uptick in productivity that would be obvious to an external observer is because it's going to take a while for people to, like, shake that out. Or maybe the AIs need to get a little bit better. Maybe, like, we've got to go through some more update steps or, you know, whatever. I'm not sure. So there's all that on the table. Then there's another possibility, which is that it actually already has worked, but just the productivity boost isn't as big as would be obvious. If people got 10% more productive, that would still be pretty impressive because it's hard to get a 10% across-the-board uplift. I've said this before on podcasts. I'm like, if you have a tool that can give everyone 10% uplift, that's great. Almost no one would know, right? It's like you can't—it's not really externally observable that clearly if that's what you got, but it may have happened, right? So it's really hard for all of those reasons. At some point, if the AIs are really fantastic and people figure out how to use them really well, it should be obvious. It should be like five people are now shipping Fortnite instead of 5,000 or whatever, right? But until that point, it's really hard to know because it's just like, especially if it was small, it'd be hard for us to see. Well, this is anecdotal, but I'm getting a lot of data points and messages from software engineers and managers. One impact it's having is there's this kind of, like, AI fatigue slash burnout from software developers who are like, look, I am good at coding. I've always been good at it. I enjoyed the work to various extents. But since this AI thing happened, since the end of the year, beginning of the year, since it's actually I'm now prompting and now all my code is generated, whether that's corporate mandates or it's just faster. I'm starting to lose my drive. Like, why am I here? Like, anyone could do this. And I think there's a sense of, like, I'm using a lot less of what I'm capable of. There's all this pressure from above to be more productive with it. And it's, I think we should, like, I'm seeing more and more signs that it's, what do we call it, burnout, AI fatigue, et cetera, loss of motivation. I haven't seen a technology, or I don't remember a technology having this widespread impact, like, everywhere. I'm hearing from folks at some of the leading, like, kind of not AI companies per se, but, like, big enough, like database providers who are now hugely into AI and they're powering a lot of the things, traditional companies, modern companies, everywhere. Have you observed some of this thing? And would you have any advice or any pointers to folks who are feeling like this right now? I guess I would say observed, no. Heard about, yes, I guess is what I would say. Like, I have talked to people who have been like such and such has been having a really hard time with this, or such and such has been having, like, there's, I've definitely heard that interacted directly with someone, not currently, no. And part of that is probably largely because most of the people I talk to have a fair amount of latitude with what they do and how they do it. A lot of the people that I talk to on a daily basis are able to make their own decisions about what they want to do with AI and so on. And so I don't necessarily hear from as many people who are going to be in a position where some manager told them. This is just what you have to do. This is very interesting because one thing that keeps coming back, and Arman Ronacher was telling me the same thing on the podcast, is he's observed that autonomy, like at your work, how autonomous you are at your work, like how many decisions you can make on what you work on, how you do your work. The people who have a lot of that are typically like, Oh, great, I can use this for this. I can use this tool. But the people who are told, you know, like beforehand you're given a ticket or the PM tells you this, they don't have much wiggle room. And now those folks are seeing it way more as a threat because, of course, subconsciously or consciously, they're thinking, Well, this thing could automate my job. It's now—or it made from that little effort I had to do that it took it away as well. So I wonder if there's a connection here. I mean, that sounds totally logical, right? If you're somebody with a high degree of autonomy, then when are you going to reach for an AI? Well, whenever there's something that you didn't want to do, right? So kind of by definition, I think at that point you're going to have a much more positive experience with it because worst case, it just doesn't work, in which case I guess that's not great. You're going to be like, Ah, this thing was kind of crappy. But assuming that it's able to accelerate some part of that, that was great. It's like, Hey, I didn't want to do this thing already. I had this AI.