Stop being skeptical about AI for development with Charity Majors

Why are there firmly two camps within software engineering when it comes to AI? Those hating its effects and those who are AI-pilled. Charity Majors emphasizes both camps and thinks they are talking alongside one another. Charity is a co-founder and CTO of Honeycomb, previously worked at Facebook and Parse, and is one of my favorite voices in engineering. Today, we discuss What it would take for us engineers to ship code we have never read and why this is more of a when question, not an if question. Why reliability is quietly getting worse across the industry and why it will take some time to recover. Career advice in this age of AI. Why middle managers should consider going back to being IC and why junior engineers will be okay. If you want to hear from someone who was skeptical about AI in 2025 but has changed her mind based on the evidence, this episode is for you. In today's episode, Charity will say, spoiler alert, that the question is not if we will stop reading code written by AI but when and we should take lessons from Ops and QA on how they prove that software that others wrote works in prod. And she's got a very good point. As any Ops engineer or SRE will tell you, that's how software has always been written by unreliable agents from their point of view. That is, software engineers like me, your colleagues, or you. And let's face it, you probably haven't read all the code in your codebase either. This is where I need to mention our presenting sponsor, Antithesis. Antithesis verifies software written by unreliable agents. It runs your whole system in a hostile simulation and roots out the bugs for you. It does this by using an approach called Deterministic Simulation Testing, or DST. Antithesis turbocharges testing by running your whole system under aggressive fault injection. Imagine Antithesis's hundreds or thousands of versions of the Mario game running, each instance aggressively trying to break the game with increasingly weird input combinations. If it finds a breakage, this is where the determinism comes in. Instead of you having to try to reproduce a tricky bug you saw in production, Antithesis can provide you with a perfect, deterministic replay of anything it finds every time. With Antithesis, you can specify properties at the whole system level and Antithesis will actively try to disprove them. So you can be confident that if your system holds up in Antithesis, it will hold up in production. Head over to antithesis.com slash pragmatic to learn more. Charity, it's so nice to do this in person. You're in my city. This is amazing. So today I wanted to kick off with AI, but before we kick off with AI, I just want to make it kind of clear for people who don't know you that you're not an AI hater or an AI lover. You actually built a lot of cool stuff pre-AI, right? Starting at, we just saw Linden Labs. Was that your first job? My first job at Linden Lab, right across the street. Right across the street. We were just talking about that. So you were building Second Life? Yeah, we were building Second Life. Yeah. And then from there on, one of the big hits was Parse, the developer tool, which was beloved by developers. Best backend for mobile services. I used to use it. And then what happened? Facebook bought you. Facebook bought it. Yeah. It was my first great lesson in most acquisitions fail. Most were terrible. This one failed. They shut it down. But ultimately, I'm very grateful to have had the experience because if it wasn't for that, I've always been a startup kid. And so nobody knew my name. And it wasn't until I was leaving Facebook that investors were like, oh, would you like some money? And that's how we started Honeycomb. And then you saw stuff at Facebook, right? It inspired you. Yeah, Facebook. There was a tool called Scuba. And so we were in a weird position. We were building on AWS, Ruby on Rails, all this stuff. And then we got to use the internal Facebook tools. And Facebook had this tool called Scuba. And we were experiencing hockey stick growth. It was just like we had over a million mobile apps hosted on Parse by the time I left. And every single week, a new one would break. It would hit the top 10 on iTunes or something out of nowhere. And these apps need a little bit of a hit stack. And it would take hours or weeks. We have to get lucky. Because it might be one app that's spamming the logs, but that might not be the reason. They might all be backed up behind the reason. We started getting our data sets into Scuba and finding them. It just went from being a really hard engineering problem with a lot of luck to just being like a port problem. Click, click, click. Oh, there it is. And it was just mind-blowing. That was a huge problem for our entire existence. And then it was solved. And then when you started Honeycomb, was this a bit of inspiration that you wanted to build something that feels like Scuba did? I was planning to go be an engineering manager, an engineer at Slack or Stripe or something. And I was just like, oof, I would be so much less powerful as an engineer without this. And so the grand plan in the beginning, I'm just like, well, all startups fail. So we'll fail, but I'll go sit in a corner and write Go code for a year or two. And then I'll open source it, and I can take it with me wherever I go. That's how Honeycomb started. That's how Honeycomb started. And we'll get back to observability or Honeycomb. But before we do now with AI, it's changing everything. But I kind of had a bit of a blast from the past, which is one of the first places we connected was in 2020. So almost five years ago or so, someone submitted a question to both my blog and your blog. And the question was like, can you measure individual developer productivity? Now, I wrote an answer and you wrote an answer. And I wanted to ask you, that was five years ago. No AI, no nothing. Today, someone shoots you a question saying, hey, Charity, can you measure one of an engineer's individual productivity? You know, they're using AI tools and all the stuff. What would you tell them? I would tell, God, I don't even remember what I said. I remember that blog post, but. We both agreed, by the way, that it was, that you can measure some dimensions and they're not going to give you the full thing. And they will, for example, not tell you how a team is doing, if someone is actually a really key part of the team. And that as long as you measure individual things, we both agreed that you need to be in the details to know. And as a good manager or a good team lead, you will know. You will know. But you have to have data to back it up. It's like color and a painting on the wall. And is it Goodhart's Law? Yes, it's Goodhart's Law. So, like, never go, well, it's this thing that matters, right? You need to actually understand, but you need it to not just be your opinion that was tossed off because you have an opinion about some, you know, we're all, we have biases, we are selective, you know. You need to look at the picture. I also believe that, you know, there's been this whole push towards individual output, but teams are still what matter. And honestly, if there's one thing that I am encouraged and excited about with the AI movement, I think it's forcing us all to ask ourselves early and often, what does good look like? What does good mean? What does productivity mean? What would better look like? What would great look like? You know, and these questions are hard. I think it's telling that we all jumped so fast to speed. Yeah. Oh, fast, we can do it fast. Let's do the same thing faster, you know, boom. And I've come to feel like that is a very immature description of what better is. Just today I saw the Entropic team posted a podcast with Spotify's head of engineering or VP of engineering, I'm not sure which one, in which they talk that, wow, Spotify with Cloud Code, they're shipping 4,500 changes per day, per week, I'm not sure which one, but they talked about speed. And I was kind of thinking like, my experience has been different because I struggled to publish any, like some of my episodes did not go on Spotify because it was down. Yeah. And yeah, they were talking about speed, but we're not talking about quality. We're not talking about more functionality, better functionality, or just things that people want. And in the comments, some people were asking like, okay, so what exactly does that mean that they're shipping more frequently? Yeah, do customers really want the buttons on their app to move around all the time? I don't think they do. Yeah, that's an interesting one. It's the easiest thing to measure. Let's jump back to last year in 2025, you wrote a blog post right at the end of the year, looking back, saying that- You and discuss that generate that code to spec. This is very interesting because some of these ideas, they've been around decades ago. Specifically, you know, if we had Grady Booch as a third person sitting here, the idea of like, hey, we can have architecture diagrams that translate to code. UML started there. I think Grady would disagree that, like, he never wanted it to go there. But irrational software back in the '90s, they said, hey, you'll define UML, it generates code, it will be beautiful. Now, it wasn't beautiful because I guess some complexity, and turns out generating code was still expensive and reviewing it. But I wonder if some of these ideas now might be just feasible. That's my hope. That's my hope. I mean, I'm just barely old enough that my first job, I was like 17, university, I was an assistant. I remember when, you know, I wasn't really aware of what was going on. I was just a kid. But yeah, I remember how stressful it was and how people were agonizing about how we'll never be able to get that information back, and everyone adapted just fine. I think the systems that, you know, they built the systems that replaced them, but not as in replaced them and worked them out of a job. They built the systems and they spent their time writing code instead of, like, running updates by hand on every server in the closet. And I guess this is an interesting one because clearly, like, the sysadmin role and profession has been—it doesn't exist today. It's kind of, let's just say, it's been eliminated. However, the people who were sysadmins, they did understand the operating systems, they understood hardware. Yes. They were in a really good position to adopt, and a lot of them just became either software engineers, product managers. I know someone who became a tech sales person. Yeah, yeah. So it's almost like— And I will hold that our generation of engineers, still the best debuggers. I'm glad that people don't all have to learn about CPU and memory and all this stuff, but, like, there's value in knowing that stuff. It comes in handy. I think there's some analogies there to the generation of code stuff. Also, you know, you took a bunch of inspiration in your recent writing about both sysadmins but also QA, and you wrote something interesting. You said, Lines of code are not the ideal artifact to review, and I'll quote a little bit from you: The tools to do this don't exist yet, but many of the ideas do exist. Most come from operations and QA, two domains that software engineering has historically been rather snobbish about. Shall we revisit our relationship to QA and ops, where I feel we always put ourselves as software engineers here and ops and QA somewhere, and maybe time to eat some humble pie? Ops equals toil, right? Yeah. I think it's time. I mean, ops and QA have always been more concerned with what is. Software engineering has always been much more concerned with how should it be. So ops and QA have always been more concerned about validating, about correctness, about does it work as expected? Does it work to start with? Yeah. Yeah. I mean, it's always weird to me just how much software engineers really seem to believe that the world exists in the repo. It doesn't. It's production, you know? The code has part of the information. Some of it, it's very necessary. We need that. But, like, I know some software engineers who—and okay, some places don't even let software engineers look at production. Just like how. I know a lot of people were very upset about AI, but the things that get me very excited, genuinely excited about AI are that it is pushing the discipline in directions we have desperately needed to go for a very long time. Production is not what happens after development. It is a stage of development. And you've been saying this consistently for pre-AI, I'm just going to say for those who don't, because I remember we've, I think we also bonded a little bit over, there was this thing called trending on Twitter when it was still Twitter, and it was Tech Twitter. Everyone was there who mattered. And there was a trend going, It's Friday, don't deploy. Something, there was maybe a hashtag even, like, I'm not sure, Don't deploy Friday or something like that. And the point was, it was well-meaning. It said, like, look, when you deploy often there's an outage, and on the weekend we don't want to do. So there was saying every Friday it went viral saying, Don't deploy on Fridays. And you came in and you said, You know what? You should be able to deploy anytime without fear, because you should be able to just know, you know, however that might be, CI/CD. And then on top of this, you were like, No, like, you should actually just not even have a user acceptance testing environment, a UAT. You should just deploy to production, like, and test in production, right? As soon as you merge, it should be going out. Like, you should have to stop the train to make your code not go into production as soon as you've merged. Absolutely. And one more interesting thing is, you had a long train of thought about, like, AI and what it could be. One thing you said is our brains are not built for validation. Almost everyone I talked to, including Andreas Heisberg, he said that, look, like, it's very clear that code generation is cheap. We are generating more code, and the bottleneck for human engineers is for code review. And everyone's trying to figure out how do we make code review easier, how do we build nicer tools. Uber has built amazing tools to, like, try to, like, surface important code reviews, but everyone's pushing, like, All right, let's do more code review. As an engineer, I'll be honest, like I never liked doing a code review. When there's very little to do and it's with someone I care about, I'll entertain it. It's more of a coaching opportunity then, right? But as soon as there's an AI, it's kind of like, I don't know. I don't really care. Like, I'm just being honest here. Like, do you care when— I don't. I've never. So one of the problems is that I think code review means so many things to so many people in so many places. And so there's a lot of projection going on. A lot of people are, if you say that you don't want code review, you're saying you don't want to talk to your coworkers, you don't want to mentor juniors, you don't want to, you know, which is not true. We've just bundled so many things into this, like, you know, it's like— Hugely overloaded. Hugely overloaded. And some of those things are really good. Some of those things could be done better in other ways, you know? Some of those things are very cultural, very specific. My friend David Pohl, who I worked with at Parse, and he's now working at GitHub on pull requests. Amazing. I love the Parse mafia. Yeah, exactly. He's like, to me, the code review is when we decide, do we want this in our product or not? I'm like, well, that is a great discussion. That is what humans are good at. We should talk about, is this mental model coherent? Should we add this? Should we not? Like, love that. Architectures, you know. But, like, the code is not necessarily a great artifact for all of those. So should we be talking to people? Yes. Is the code review the right form factor? Maybe. But I think that the emotional reaction that so many people are getting to that, like the validation in my book is at the very bottom of the list. I'd like to, like, stay here a bit more. Can you break out the parts? Because it feels to me code review is overloaded, but the parts of code review, or the things that you have seen are good things, and maybe we don't need to do as code review, and the things that are just, like, just have never been that good and maybe we just need to throw it away. Yeah. I mean, I think, do we want this in our product is, that is great. I mean, ideally you'd talk about that before you write the code for it, but, you know, whatever. And, you know, is this API design? You know, those are great conversations. Reading for— Syntax and bugs and that sort of thing. It's not evil, but it feels like it could be. It's a teaching opportunity if that's the best teaching opportunity you have. And I guess some folks at some point maybe you need them, but it doesn't feel high. It doesn't feel like a great use of anyone's time. It feels the only time where it's useful is if someone joins a team and initially it can be a little bit of feedback. Yeah, especially when there's like nothing is written down, there's no guidelines, there's no linting rules that would give you that. Well, see, that's, again, yes, we can fill in the... What you really want is a CI system that gets faster as the volume grows, and CI that offers instant parallelization to give you unlimited concurrency and to intelligently route changes at runtime. This is what Buildkite does, and why global software leaders at every level continue to rely on it. The same architecture that observed the scale of Shopify and Uber a decade ago now runs about 1.4 billion job minutes a week across Cursor, Meta, Reddit, and Snowflake. While the rest of the CI world are crackling under the weight or re-architecting their platform, Buildkite continues to reliably grow. Agents run on your infrastructure or on Buildkite. Any cloud, any chip, your secrets, your scale. Every artifact and log is captured, so when something fails, either you or your agents have immediate insight for why. As you're engineering the context you give to your agents, think about how you'll verify what they hand back. If your system is buckling under the increased volume, head to buildkite.com/pragmatic. 30-day all-access trial, no credit card, and an actual human engineer on standby. His name is Ola, and he's very helpful. And with this, let's get back to charity and communication norms with AI. I think there was this frenzy of, Oh my God, I can do this. Oh my God, it's so cool. And I know you have also become very weary of this slop. I just don't even read it anymore. As soon as I can tell— As soon as you recognize this might have been AI, it's like trash. Here's a baseline: you cannot send anyone something you haven't read. And in fact, if it would take them longer to read it than it took you to make it, it's probably slop. That's really disrespectful, actually. And I think, like, just like asking someone—like, you're asking, anytime I give you something, I'm asking for your time and attention. And if I'm giving you something that I don't even know what's in it, and I'm putting it on you, it costs you instead of me, that is— Not good. I also think that even before that, it's like I've noticed as I start working on these norms and values, I'm noticing myself as I start to ask someone a question without trying to look up the answer. Ooh, I shouldn't do that. Or if I'm giving someone something that I kind of generated, and I'm like, ooh, you know, it's—so part of it is just self-awareness. It's interesting because everything you talked about, it reminds me of when a new joiner would join a team, a junior engineer, a new grad. Either they had emotional intelligence or they picked up on really quickly that, for example, you go and ask a senior of their time once you put in a little bit of work and you start to respect their time as well. And obviously it doesn't start like that. We don't want them, but there's this balance. And I almost feel it's the same thing. We're like, look, like respect your colleagues, respect fellow humans. If you are communicating with them, make sure that you're not wasting their attention. Because now I guess attention is—we're kind of running low. Like we have all of these, like a bunch of people have a bunch of agents doing, but the point is that's kind of the currency. And as long as you respect that, it doesn't matter. Like I think we're not talking about don't use AI for this or that. Like use it as much as you want or make yourself more efficient. Just don't degrade, because it really degrades those personal skills, right? You can use AI as a shortcut to help you not have to think too much, and you can use AI to help you think more deeply and more rigorously. And both of those use cases have their place. But when it comes to your core job function, we primarily want the second one, right? And especially if you're involving someone else and you're asking them to review or, you know, and this is not absolutist. Like there are people who English is a second language and they use it, people who like neurodivergent and they use it, and that is, again, that is still being respectful, you know? So it's not like, like you said, it's not no AI, but it's like make reasonable asks of each other and, you know. We don't need to reinvent a new bar for quality or respect, because we have great bars already for quality and respect. We just need to apply. For a while there, I think that there was a bit of, Oh my God, this is so cool. Do you see what this cool thing can do? And I think we're all just, like, so over it. Well, the reason I really respected you came from the CIS, you know, the CISDEV background. You also, you're very involved in SRE. These are all folks who have been pretty skeptical of AI. And you mentioned how you're seeing two camps, two very clear camps. There's, like, kind of the AI-pilled folks who get it, and then the people who seem like they just hate AI. And you said that you're not seeing these two camps have any sort of way to go between any feedback. Can we talk about what you're seeing and, like, maybe, like, where you see some of these camps forming? See, the problem is that neither side is making it up. Like, they are seeing really scary trends. They're grappling with real hard problems that are getting worse, you know? And on the enthusiast side, it's like they're acutely conscious that it's a bit of a race and that we need to push ourselves out of our comfort zone, and they see other companies moving faster, catching up, leapfrogging. They're really worried about, you know, we're falling behind. And the first thing, I don't want to make it sound like false equivalence, because while there are elements of this that are true, I think every company is more one or more the other. But, like, they're not wrong. They're not wrong. We've never seen technological change this fast. We're on the inside of an exponential curve, which is very rare, and it never usually lasts that long. But it's still happening, you know? Things are happening that shock us, and we would be wise to prepare for them. So, like, that's real. That's real. And these folks are usually, at most companies, usually they are the small minority, and they are constantly filling outman. One of the things that's ironic, though, is that both of these sides feel like they are the tiny minority, and they're outman, and they're being suppressed, and they are standing up for what is truth and valor in the face of the big AI folks or the big skeptics. But the other side, so, and this often starts to come down to the group that is on call and the group that is not. Ooh, yep. Because the people who the buck stops with them, they are seeing melting mental models. They're seeing slop. They're seeing all their hard work just dissolve, and they don't see any end in sight. So just to be clear, we're seeing that the people who are on call for a lot of these systems, they're seeing more incidents. They're seeing carelessness being caused by it. They're actually seeing that since that groups are using more AI, our systems are getting way worse. Way worse. Yeah, and that's very real, not making it up. No, no, actually, I was just talking to someone inside of Meta. There's been this big drama where people have been reassigned. Oh God, I know, I saw your post. So not just my post. Since then, I haven't written about this since, and I'm not sure when this podcast comes out, I might have not talked about it, is inside of Meta, they track Sev zeros, which is the highest severity. Oh, I remember. You remember Sev zeros. There has been a flurry of Sev zeros, so many of them, and you cannot hide. Like, this is, you know Meta, like this is black or white. And the past about two months, it's been crazy. And just so it happens, it's happening inside of Instagram, it's happening inside of WhatsApp, where the trust and safety, basically the reliability folks have been Acts removed. So it's impossible to deny the connection as well. Of course, it's not a direct one, but again, each one has it as a postmortem. But Meta has not had this bad for closer to a decade. Yeah. Fast and break things. And you put two plus two together. And when I told this story at a conference, people came up to me and they said, I'm so glad you talked about this because my company, different company, often VC-funded or publicly traded, like, same thing is happening. People are, like, whispering to me, like, we are not Meta, but the same thing is happening. Same thing is happening. And you know what they all told me? They told me, I thought it's just us. Or I thought it's us and then my buddy who works at this other company. And suddenly it's like, oh, it's all of us. No, it's all of us. Yeah. No, it's a real thing. And the— DevOps. Just can we go back a little bit in time? You were there. Why was it created? And in the end, there was this massive DevOps movement in the 2010s. Do you think it succeeded? Do you think it failed? So before DevOps, we needed a DevOps because there was devs and ops, and there was the proverbial wall that code got thrown over, right? And ops were the people who were in charge of the IT. They deployed, they managed the servers, they set the Linux version, handcrafted Linux, you know, pluggable storage models and everything. That was always a bad idea because it's split brain. Half of you are writing the code and the other half are understanding it. I would argue that you can't really understand the code you write unless you're operating it. So, you know, the DevOps movement did a lot of good, trying to knit back together that sort of original sin. And, you know, around the time that I was assistant, there was this big push, All right, ops people, learn to code. And great, I'm glad that happened. Everyone who works with computers should be writing code. I feel like the wave after that was a little less successful, which is like, Okay, software engineers, time to learn to understand your code in production. But I also think that, in my mind, 20 years of DevOps was really about one thing: trying to create one feedback loop that connected people writing code to that code in production. And it failed. I mean, it failed to this day. Like, they're done by two different domains, you know. There are some people who, I mean... And I'll show you this diagram that you drew. We now added agents. We'll put it on so viewers can see it. This is your... I think it's a really nice draw-up of how there is no feedback loop. Like the ops people, or oftentimes we call platform teams, they manage the infra layer. Engineers deploy there. And so, to be clear, I think that's actually good in finding healthy. I think that there are separations of concerns where you can't expect anyone to do everything. And the nice separation of concern is, do I own, am I responsible for the stability of the things that you put code on, or am I responsible for the code that I put on the thing, right? That is a nice seam because you want the infrastructure to be stable, be, like, to protect itself, to be resilient, and all these things. And you want your code, like, to be oriented towards, is every single user having a good experience? You can have one of those things be true and the other not be true. Like, they are decouplable. And actually, this is, like, even the most modern companies. I often refer to Entropic as this company which operates in a very different way to most companies. They are very successful despite doing a lot of different things. However, internally, they have platform teams. They have a cloud platform team, and then they have applied AI, which is more of the kind of the feature teams, the integration. And the two, I talked to both of them, they just have a very different outlook. They have a very different view on even basic stuff like, will software engineers be obsolete? The people on the platform team were like, no, we're working really hard. And on the applied, they're like, well, maybe it will happen. Yeah, that does not surprise me one tiny iota. But so this company, Entropic, that started from a blank page, they arrived at the same place. Yeah, yeah. No, I think it's the right separation of concern, and I'm not trying to erase it. But I think that to be a good engineer, you need fast feedback loops. And this is part and parcel with the whole, oh, the source of truth is the code. If that's where you live, if you live in the land of how it should theoretically work. No, and I think that with agents, they're breaking that, right? They're breaking that and they're forcing another thing on the observability trip is a lot of people, if you say, like, what is observability? They'll be like, Ah, well, there's three pillars. There's metrics, logs, and traces. We talked about this last time. Metrics and logs, I would say, are system exhaust. They're the exhaust pipe. And they're never going away because every team runs a ton of third-party software. They didn't write it. They don't own it. They just have to run it, and it's outputting shit. Yeah, and you just got to put it somewhere. You observe it. You see what... Yeah, yeah, yeah. And then you do stuff with it. Yeah, and, you know, you should put it somewhere cheap. There's a ton of it. It's not super high value, but you definitely need it, right? And you can't do anything about it. You just take it and put it somewhere. Then there's your code. There's your crown jewels, the code that makes you a company. And for that code, your telemetry should be a product decision. It should be you store it once with all of the connective tissue, because the value of rich data goes up not linearly, not even exponentially, combinatorially. If you have a wide event or a trace with 29 bits of data and you add a 30th, that 30th is more valuable than all the others. Like, it is just so powerful. And with non-deterministic software, you know right up front you can't predict what it's going to do. You have to. Like, that is a product decision to capture that trace. So let's talk specifically about modern observability and, like, companies that are, you know, like either building AI, really code, or just complicated code that they're generating. In the old world, again, like I'm just being, you know, like observability 101 back in the day, the way I would have written the code is you write the code and you think, like, Hmm, something funny might be going on here. Let me do a log or an info or a warn. And then I would also try to maybe if we're printing some production, I realize, like, okay, well, I guess it's crashing and we don't have any logs there, so I guess it's some other part. Let me put a tool that will, like, log everything and now I have a bunch of stuff. Now, this is the old, the very simplest way of thinking. In kind of a modern business where I'm like, I know this is high-value stuff, what are ways that I can go about that section maybe a bit, like, more practical then? Because I just will use super basic one. Auto instrumentation has gotten so good in recent years. If you're using OpenTelemetry, and everyone should be using OpenTelemetry, all of the common patterns, like all of the models are trained on them. So it is literally faster and easier to build with instrumentation than not to. And with instrumentation, do you just, once I have the code in a compile step or an extra step, it just adds it to the right lines? This is what's important. Like, it's part of just developer intent, right? This is how you declare your intent, and that's how you check up on your intent in production. It's honestly gotten so much simpler. You know, I don't fault developers or anyone else for not kind of closing that loop with DevOps, because the fact is it was prohibitively hard and time-consuming and difficult because, you know, you're an old-school software engineer and you sit down, write some code, you're like, Ah, here I should instrument it and look at it in production. So you're like, Okay, I've got a bit of data and I want to do something with it. All right, is it a metric, a log, a trace, an exception, an error, a profiling? You know, just like, okay, if it's a metric, is it a counter? Is it a gauge? Is it a, you know, just like all down—it takes so... And then, well, what type of data is it? Is it going to have high cardinality? Is it going to be a—cardinality, you need to worry about that. Yeah. It's just like, if it's a log line, which log level do I append it to? Like, it's just, you could double, triple, quadruple the amount of time that you spent writing the code, trying to instrument it, and then still wouldn't be done. Like, you deploy it and then it's like, Okay, I know the name of the thing that I added, but... How do I find it? How do I display it? How do I create a dashboard? It's just like, that was prohibitively, that was really hard. But now we can bring all of this to you right in your development environment. It is easier and faster to instrument with telemetry than without it, and you don't have to leave your development environment to go and get it, you know? You could have the agent, like we've built some really cool shit at Honeycomb where it'll just, it'll be like, Oh, hey, that thing that you wrote, you know, maybe you want to look at this, and you can control how. You know, cuts through bureaucracy like a hot knife through butter. When it comes to partnering with, you know, the sales org of another company, you do not have trust incredibly. You work to build trust through reciprocity. You learn just how much you can trust them over time, right? But the best vendor relationships are the ones where you genuinely, you feel like their successes are your successes, your successes are their successes. You're happy to see each other because each of you are delighted because you know you're getting something from it. It feels like you are two different teams working at the same big company. That is rare. Doesn't usually happen, and that's fine. Most vendor relationships are ones where you shake hands, you exchange money and services, and that's fine. But I think in an era of AI, these are durable skills. These are durable skills for very senior engineers who care about impact. Senior engineers and also engineering leaders, and anyone who wants to become an engineering leader. Because I guess, like, I mean, both of us have been in engineering leadership, like you've been in much higher positions than I have. But I think it's fair to say that the way for you to get to that CTO role, that head of engineering, that director of engineering, is to do the work for six months to a year to year and a half. And to do so, you need to know these things. I feel Observability Engineering, I'd be underselling this book, I'll be honest, the title, but I'm also going to get it, and I'll probably think of ways to share a bit more. But thank you for writing, and thank you to all your co-authors. But speaking of leadership, I'd love to talk about a little bit of engineering leadership because there's a lot of things are changing. But I loved one of your very recent takes on leadership, and I'm going to quote you: The most effective leaders are kind, caring humans and skilled business operators. The second most effective leaders are terrible humans and skilled business operators. And after that comes everyone else. There are plenty of good, kind humans who are sloppy operators and bad at business because being good at business is very hard. And you said this in relation to what happened at Twitter slash X, referring to Elon as a terrible human, but a skilled business operator. Yeah. I, well, I don't know that I would call him a skilled business operator, but my point was that Twitter had 16 years to figure it out, and everyone could see that they were not figuring it out. And whatever else he is— Figuring out the business, specifically. Figuring out the business, yeah, building products, you know, reaching folks. And you could argue that X has gotten better or worse, but you can't argue that he is running it with 20% as many people. Yep. And it's working. And it's working. And some of that, you know, 30 engineers on core, on the core product, and another 30, and—but like 60 engineers, there were 1,700 before, you know? And you could argue, and I think it would be true, that it's some of the work that those engineers did that— But like, this is the point. If we don't do it ourselves, meaning hold ourselves to a high standard, build with efficiency, constantly be, like, trying to get better, if we don't do it ourselves, someone will come and do it to us. And this is what you also said. You closed saying, if we want to remain in leadership, if we want to set the culture and the tone and take the ethical stance that we believe in, we first have to win at the business. And I think this is, like, especially now that there's so many changes happening and technology changes, there's world winds, business will go up and down. I guess it's a reminder that, like, you want to keep your eyes on the prize, which is, especially if you're a leader. The 2010s, there was so much money sloshing around in Silicon Valley, and times started to get tough, and all of these companies canceled their DEI programs and blah, blah, blah. Yeah, they never believed in that. They were just trying to buy people off, you know? And that is very telling to me. And I have taken a lot of lessons away from that, which is just that it's not enough to be a good person. I believe that people who are kind and care about people can, and usually do, do better than sociopaths in the same roles, but only if— They are good at business. Learn the business. Stay close to it. You got to. With AI, now that coding has become cheap, now that engineers are running agents, how do you see the role of good, skilled engineering managers and engineering directors change? What has changed? Well, the first thing that's changed is, I think everyone gets to be hands-on, specifically to generate some code, to ship to production. You should know what it feels like to submit a diff, to get a PR through. You know, you should know what it feels like. It's just easier now than it's ever been to pick it back up, to fill in the blanks, you know? And it's always been the case that leaders were better if they had a hand in it, and now it's just—there's no excuse not to. Teams are getting smaller. In general, I think this should be a good thing. If we can figure out how to own more surface area, it should be a good thing. I worry that the way it's happening is it's being done by CEOs who are like, Oh, well, this other company is doing it, or it's magic, or we're going to do layoffs, or, you know, and I really dislike the anti-management tone. So, like, no argument that power tends to drift towards managers over time and needs to get pushed back to engineers. No argument there's a tendency to have too many managers. You know, the bureaucracy kind of, like, generates a sort of, you know, it's easier to say yes than it is to say no, and so these things happen, so they need to be pushed back time to time. But I believe that management and middle management is deeply essential. And I look forward to seeing how that works out for them, not having any. But, like, the role of a manager, middle management, in my view, is sensemaking and context-giving. Because, like, I don't believe in a world where engineers are just given tasks: Here's your Jira, go do the things. AI can do that. I want people who understand what we're trying to do, understand how we're trying to do it, or who are there to help us figure out how we're going to do it. And you can't engage emotionally, creatively, collaboratively without understanding. And that understanding is incredibly difficult to build, and it's fragile and it never lasts very long. For those of us listening who are middle managers, it's been a tough few years because what they're seeing is there's a push to have fewer of them. A lot of their colleagues, if they're at unlucky places, they were made redundant, and many of them have struggled to get similar positions. We're talking director positions, we're talking head of engineering, senior engineering manager. That role is disappearing faster than ever. I think directors might still be there. For folks who are in this position and they do like middle management, they do believe they're good at it, what do you think tactics could be to give them a bit more career options? Tactically, I would say go back to BNIC for a while, even if you know it's not what you want to do. If you're at all capable, if you're not capable of it, then I would try to work. You've got to get AI on your resume. You just have to. And this is a huge career risk. If you're working somewhere where you're not getting these skills, that is a massive risk. I would do whatever I could. And this is very interesting that you're saying get AI in your career, because I remember about a year, year and a half ago, I started to pay attention to, like, okay, this is happening. And I remember a year ago I wrote an article about how to become an AI engineer, and I talk with engineers who just, like, at their workplace started to do AI and now they're AI engineers. Next thing I'm hearing right now is the people who have, like, two to three years of AI engineering experience are so in demand. I'm doing research on the job market, and they're like, This is the best job market ever. However, you know, the people who are like, Okay, I have none, but I want to get it, they—and let's say they're out of a job—they're struggling because no one's giving them the benefit of a doubt. It is really hard, and I'm not saying it's right, but it's how it is. And I guess the reason we're ringing this alarm bell is we know this change has not been as fast, so do it now because— And that tension, like what? And then he kind of walks through traditional ethical frameworks like utilitarianism and stuff, and just shows how there is no recipe anyone can follow that doesn't lead you to some really stupid... And he's like, this is just no gods, no masters. We are— which doesn't mean that everything's relative. Doesn't mean— what it means is that the way to live an ethical life of integrity is you need to educate yourself about the world. You know, you need to know things, right? And then listen inside. You know, where are you drawn? What suffering really speaks to you, or what caused you, you know? Because no one can tell you what matters. You have to decide what matters. And so that introspection, and it's so at odds with the sort of performative rage, you know, which I'm just so exhausted. All right, so that's one. Number two, this is a book that I've recommended a couple times, but I'm just going to keep recommending it because it's so good. It's by Adam Becker, and it's called More Everything Forever. And he is a journalist based in San Francisco. He has a philosophy undergrad and a PhD in astrophysics, and he just demolishes all of the AI religion, the singularity, and the effect of altruism and accelerationism and the whole, like, what if we could have infinite growth foreverism? And he's like, the heat death of the universe, you guys. Literally the only thing we know about exponential growth is that it must end. It must end in an S curve or in a crash. It must end. And he's got this dry sense of humor. And there are a couple times where he's just, like, describing some of the very real things. It's just like, why do Oxford ethicists want this? He's talking about, like, taking over star systems, and it's just ridiculous. And he also, he gets in this whack. He just, like, talks about these people who are working so hard on life extension, and he's like, These are a bunch of sad little boys who miss their daddy. And I was just like, Oh my God. It is the oldest fear of humanity. It's the fear of death, and you just see it. You cannot unsee it. So yeah, those are my two. They're both so good. Well, Charity, thank you so much. This finally made it happen. Finally, it's a good time. I always really, really enjoy talking with Charity. I hope you also liked it. I appreciated how Charity talks about the trust account. If we are debiting trust from the creation of code because AI wrote it and no human read it, then that trust needs to be refilled somewhere else. Testing evals and guardrails are all ways to add more trust that we lost by using AI. I also appreciated how she talked with empathy about both AI camps. The enthusiasts, or AI-pilled folks, are seeing the practical wins, while those operating production systems see the slop. Neither side is wrong, but they should talk to each other more. So if you see wins with AI, share with the broader team, but also talk about it when it creates more work, reduces reliability, or when it degrades quality. And for those of us feeling anxious about all this change, especially directors and managers, I'll leave you with Charity's advice: anxiety and excitement are psychologically almost the same, but the difference between them is agency. So instead of waiting for change to come to you, take charge however you can and make changes yourself. Do check out the show notes below for related to pragmatic engineering deep dives on how AI is changing software engineering, and for another discussion with Charity on observability. And I can very much recommend her book, Observability Engineering, second edition. If you enjoyed this podcast, please do subscribe on your favorite podcast platform and on YouTube. A special thank you if you also leave a rating on the show. Thanks, and see you in the next one.