← Return to Index Archived October 29, 2026
The Lead — Oct 29
THE PRAGMATIC ENGINEER · GERGELY OROSZ

Software engineering at Google & CS education with Titus Winters

Former Google engineer Titus Winters argues that computer science education has calcified around outdated traditions just as AI is reshaping what software work requires. He traces Google’s hard-won lessons on testing, refactoring and learning culture while making the case for judging engineers on their reasoning, prompts and tests rather than code alone.

1h 40m / October 29, 2026 /aieducationtechnology / Transcript sourced from openai
All episodes from The Pragmatic Engineer →·Episode page →·Podcast website →·Listen on Apple Podcasts →

Overview

Titus Winters, former Google engineer and co-author of Software Engineering at Google, argues that software engineering needs to prepare for change rather than treat code as a fixed artifact. Drawing on Google's experience refactoring hundreds of millions of lines of C++ and his role in CS curriculum work, he connects engineering quality, team culture, AI adoption, and education.

His central concern is that both productivity metrics and computer science programs are lagging behind how software is now built. AI may not show up as faster ticket completion, but it can make previously impractical work worth attempting.

Key Takeaways

  • Google's engineering discipline came from hard experience. The episode references Google's decision to make testing mandatory after persistent regressions prevented Search from shipping reliably. Winters frames tests, static analysis, lint rules, and code review as ways to make change safe.

  • At Google, Winters worked on what he calls possibly the largest refactoring effort in history: cleaning up and modernizing a shared C++ foundation across roughly 200-250 million first-party lines of code. The challenge was to change widely used infrastructure without disrupting the services built on it.

  • Repeated lessons should become automated checks. When a team discovers a recurring code-quality or architectural problem, it should encode the rule in linting or static analysis where possible. That keeps reviewers focused on design, product judgment, and risks that tools cannot reliably assess.

  • Developer productivity measures miss much of AI's value. Metrics such as code volume, feature count, or release frequency may show modest gains, while AI enables engineers to take on higher-value work that would previously have been dismissed as too expensive or risky. The useful question is not only "Are we shipping faster?" but "Are we attempting better work?"

  • Psychological safety is an operating condition for strong engineering teams, not a soft extra. Winters cites research suggesting team culture can predict a meaningful share of whether teams meet their goals. Teams improve when people can admit uncertainty, expose mistakes, and teach each other.

  • Winters thinks much of today's undergraduate CS curriculum is based on accumulated tradition. Students often get broad but shallow exposure to topics, while receiving too little repeated practice in programming, system reasoning, testing, and making tradeoffs.

  • AI makes outcome-only grading unreliable. Students can submit competent-looking code without understanding it. Rather than ban AI, Winters argues schools should assess the process: prompts, tests, explanations, and the student's ability to defend decisions.

Practical Steps

  • Turn recurring review comments into deterministic checks. If reviewers repeatedly flag an API misuse, dependency violation, or unsafe pattern, add a linter rule, static analysis check, or test that catches it before review.

  • Review AI-assisted changes as work you personally own. Use AI to explore options, explain unfamiliar code, draft tests, or generate starting points. Do not submit output you cannot explain, test, and take responsibility for.

  • Adjust productivity discussions. Alongside deployment frequency and throughput, track examples of projects made feasible by AI: retired vendor dependencies, improved reliability work, new internal tools, or technical debt that was previously uneconomic to address.

  • Build feedback habits that are not tied to formal evaluation. Ask teammates early, "Does this direction make sense?" A fast design check is different from waiting for a final judgment in code review or performance review.

  • For educators and hiring teams, assess reasoning in real time. Ask candidates or students to work with AI while explaining their prompts, assumptions, tests, and how they verify output. Grade the evidence of understanding, not only the final code.

Notable Quotes

  • "Let's try to learn our hard lessons only once, codify those rules somewhere deterministic, apply them equally and appropriately and fairly."

  • "A lot of the value of AI is not in doing the same thing faster; it's doing a harder, more valuable thing that you wouldn't have even thought about."

  • "Stop grading just the outcome. Focus on the process." - Titus Winters

The AI, as much as we occasionally anthropomorphize it, is still a tool, and you have to be able to stand up for whatever your name is on. — From the episode

Full Transcript

Source: openai 1h 40m runtime

What if an engineer who helped define the contents of today's software engineering curriculum said that most of this should be burned down? An ex-Google software engineer says exactly this. In 2023, Titus Winters was the only active software engineer on the steering committee for the ACM and IEEE undergraduate CS curriculum. Before, he spent more than 10 years at Google, was the lead author of the book Software Engineering at Google, and ran the company's readability program. In today's conversation we discuss Why Google made tests mandatory around 2006 after the flagship search product could no longer be released on a regular cadence due to persistent regressions. Ways to measure developer productivity and why the value of AI does not show up in the metrics that we measure today. Ways that Titus would change computer science education, including why he'd want students to be graded on their prompts and tests, not the code. And more. If you want to know what Google learned about engineering at scale or how modern computer science education should look like in the age of AI, this episode is for you. This episode is presented by TurboBuffer, vector and full-text search built on object storage. It's fast, cheap, and extremely scalable. Before we start, I'd like to talk about Linear, our season sponsor. Until recently, I was thinking of Linear as a tool that helps plan and organize the work, but it's much more than that. A lot of teams use Linear to ship changes from Idea to Merge and GitHub all without leaving the tool. I sat down with Dima, an engineer on the Linear team, to see how devs inside of Linear use Linear to build all of Linear. So here's a real issue. When you submit a review with unsent draft comments, Linear warns you about these comments, but it doesn't tell you where those drafts are. A customer complained and Dima decided to fix it. But for the fix, Dima did not open his ID. Instead, he opened up Linear and described the issue to the agent. The agent built a fix, created a PR, and attached storybook screenshots for easy verification. You can see the fix with the draft comments linked on this screenshot. In the PR, there's a versatile preview. You can open the preview with one click from Linear to play around with the change, and we see that it worked. And here's my favorite part, Guided Reviews, which is a work in progress. Guided Reviews is a new code review experience where the important stuff just pops out. The top of the review has a summary and diagram of the change, and the pull request is divided into chapters that walk you through what's changed. These blocks highlight the important changes in a way that's easier to understand and review. Oh, and when Dima is finished with all of this, he just self-approves the code and it's merged into GitHub, all without leaving Linear. If you take one thing away from this, Linear is no longer just a place for planning tickets and organizing the work. It's where all the work happens. Creating the issue, kicking out the agent, checking out the preview, doing the review, and merging to production. Linear is the product development system for teams and for AI agents, and I'm loving this evolution. Check it out and get a six-month trial at linear.app.com. Titus, welcome to the podcast. Thank you so much for having me. I'm so excited to have this one. It's awesome to do this in person again. Yeah, it's a joy. So how did you get into tech? Growing up, did you know that you're going to do something with computers? I mean, my earliest memories, I thought I was going to be a scientist, but like, you know, as a three-year-old, that means like lab coats and test tubes, and I don't know what else, right? My dad, very insightfully, knew that computers would be important. Like, that was not his field at all. He was kind of in carpentry and set design and things like that. So there was always a computer in the house. It was usually kind of a low-end thing of, you know, whatever we could scrape together and afford. And then I remember I really got my start, I think I was around 11 or 12, and instead of buying candy for everyone for Valentine's one year, he went to the bookstore. And I don't know what my brother got, but I got a book on Learn QBasic. And it was all kind of downhill from there. Like, I spent probably the next five, six years like programming, teaching myself to program was my hobby. Occasionally, I would wind up like meeting, you know, other kids my age that were a little into it. We moved eventually to Las Vegas, and I went to a magnet school that had a computer science program that is actually a really well-regarded high school these days. I had no idea. And yeah, like just kind of self-taught initially pre-internet back when you had to really fight for it. But yeah, that set me on my path, that book. And then you, after high school, it was kind of obvious you went to college first. Yeah. So I went to Harvey Mudd, which is a great school with a mix of, you know, tech and liberal arts education. The kind of place that'll teach you to build weapons and teach you not to use them, that sort of thing. And then I met my wife there, and she was a little bit more junior than I was. So I didn't want to move away. She was still in school. So I did my master's at UC Riverside, actually in computer science education. Alrighty. I did a kind of weird, I had a very strange graduate experience. But yeah, I spent four years at Riverside doing a PhD in CS ed, kind of applied AI, educational data mining, computer science education. So some of the earliest conferences on CS ed research, I published there. And my advisor was the department chair. He never generally ran grad students, like the research area was not his focus. But he was constantly the chair. He was this like, charming, you know, social political sort of guy that knew all the stories. And so my graduate work was a lot of learn all of the stories of how we got here, and teach all the classes, whenever there was a hole in the Yeah, and we'll, we'll keep coming back to probably this, like this education, computer science thing throughout this thing. So we've talked before, so I know, but let's let's jump back. So like you, you, you now graduated from college, you were looking for a job. Yep. What, what did you find? So first, I did a sort of consulting group, again, two body problem. My wife went to grad school in San Diego, I did not get a professorship after my PhD. So I did this consulting stuff. I was at a small startup consulting group, about 30-50 people. Mostly I worked for indirectly for DirecTV during the embedded system stuff on set-top boxes. And then after four years of that, we were done with grad schools, we were done with all of that, I took the offer to do Google interview, got the offer, decided not to go to the Bay Area. And they moved us to New York. So we've been here ever since. This is when Google was opening an office here. The office was already open and reasonably big. This was 2010. So I think there were a few thousand people there. And then in 2010, so Google was founded in 98, they went public in 2004. I remember that they start to really accumulate a global brand as like, wow, Google is the place to work at around like 2005 to 2010. So right around when you were applying, it was actually one of the hardest places to get into a bit like today, Entropic, or last year OpenAI or the, you know, like in the mid-2010s Uber. How did you find the interview experience and getting an offer? Because it must have been pretty competitive at that point. It was competitive. I was a little surprised that they gave me the offer. I felt like I didn't do that great. I remember for some stupid reason, I really, really didn't want to spend a bunch of like precious vacation days. So I flew up from San Diego the morning of instead of sensibly like... And they would have paid for the hotel and everything. Yeah, I just didn't want like, I don't know. I'm an anxious person. I like my habits and routines and stuff. Getting up at five for a big interview is not a good plan, folks. Hot tip. So there are a couple of things like I remember one interview was like a relatively simple like clock math sort of thing. It was technically camera angles on a Mars rover or something. And there was a point and I was like, I'm tired and having interview brain and this is just basic math. So here's a function that has the right inputs and outputs. And if I wasn't stupid this morning, that would work. And I just did it all in terms of that. And I think my interviewer actually really liked that approach of like, yeah, this is not the important part. Like, let's just talk. And so, yeah, eventually they had That the change isn't critical or isn't important. And I hate to break it to people, but, like, the security world has gotten real, real bad in the last few years. Like, there is nothing that is not going to change. Like, the strategic advantage of being able to change is incredible. Is it safe to say that if I join a company, a startup, a large company, whatever company, I joined a team and I look at the code base and, you know, my first weeks, my first months, I'm just being afraid of pushing code changes because I worry this thing will break, I'll break production, that I'm working on a system that is just, let's say, poorly engineered? It could be the system is poorly engineered, or it could be there's some onboarding and mentoring that didn't happen there, right? So one or the other, probably. One or the other for sure. And I think, and this goes to some of the educational stuff, like, we in undergrad, you know, education and, like, the way that we kind of portray what the job is like, there is a correct answer. There is right and wrong. It is not just the trade-offs. And you're graded any time someone looks at what you have done, right? And I think it's going to be, especially for new grads, it's really hard. It's really easy to get into a fear of, like, I don't know if I did this right. I don't know if this was the answer you were looking for, because they've been falsely given this perspective that there's a right answer instead of, of course, it's just a trade-off, right? It's at some point in the design space. And I guess you're also saying that these new grads, they're also used to you do something that is a meaningful effort, typically a class or an exam, and you get feedback. You get a scale, you get a grade, you get a thumbs up, a pass or a fail at the very least. And I now actually remember when I went to my first workplace. I should remember this, this really weird, like, joined the company, I built this project, and then nothing. And then I, you know, I do another thing, and then nothing. And then I have a one-on-one with the CEO of the company, and I'm thinking they're going to fire me because, like, clearly, like, I knew that, I don't know, like, middle of the day, like, I was sometimes slacking off because I finished my stuff. In hindsight, I was pretty efficient, but I just finished my stuff, and then I was just browsing the internet. I told them I'm ready, and I was pretty sure, like, maybe I'll be reprobated or something. And instead, they'll be like, Oh, amazing job. Like, you've onboarded so quickly. We're so happy with you. And I was— but the difference was that up until then, I've had years of getting direct feedback, and it didn't connect in my mind. Yeah. The other part of that is, like, the only time that a human is going to look at your code if you're an undergrad is to grade you, is to judge you, right? Whereas when you, like, become a more experienced engineer, right, like, you're asking for feedback regularly, but it's just like a check-in of, like, Are we going in the right direction? Does this make sense to you? Right? And the more of that feedback that you can get that isn't judgment, the more effective you will be. And, like, oh, I hate that we've, like, given all of the new grads and all the junior engineers exactly the wrong instinct. So going back to Google, what is App Sale? How do you pronounce it correctly? App Sale. Yeah. So App Sale was starting about 10 years ago, shortly after the initial pivot to cloud, there was a motion to, Hey, we have all of these amazing developer tools in addition to all this amazing production stuff. Maybe we should try to externalize the developer tools as well. And at the same time, there was also a little bit of Google shifting into Alphabet. So for both the developer tooling aspects at one level of, like, how do we share developer tools amongst different companies within the Alphabet umbrella. And how do we share high-value, like, internal interfaces and APIs and, you know, tech components? How do we share those when it's not one, you know, single corporate, one single company, and also not one monorepo, right? Increasingly, yeah, exactly. And so there was a bunch of work that was kind of the normal, like, okay, what does it take to make these developer tools like multi-tenant and, like, you know, slightly less completely tied into that ecosystem? And just multi-tenant, for those who are not as familiar, what does multi-tenant mean inside of Google? Well, so this would have been multi-tenant of other companies within the Alphabet umbrella, and then maybe extend that out to cloud customers. Oh, got it. Yeah, so like a way more, and Google could theoretically use it, something like that. That became not the hottest topic, and they kind of dropped that idea after a year or two. But the low-level components, like if you wanted to share these things that were written internally in the monorepo, what would it take to do that effectively? And Abseil was basically, here is the layer of common, like, C++ idiom, like this is the vernacular that everything at Google is written out of. Let's open source that so that there's a common substrate for all, like, Google tech. At the same time, the question was, some of this vernacular is gross and tech debt-y. Should we clean that up before we commit to it forever? And the answer was yes, to their great credit. So the Abseil open source project is the public, like, projection of the stuff that got cleaned up. Most of the work was, that was, I would defend pretty strongly, that was probably the largest refactoring task in human history. No way. Yeah, we had like ten— Ten or twelve people working full time with world-class tools that no one else understands yet, just cleaning up and making sure that idiomatic code externally would be compatible with idiomatic code internally. And obviously, when you were talking about cleaning up, you're talking about cleaning up utilities and components that are in the middle of some of the highest-used services in the world: Google Search, Gmail, Maps, you name it. All of it. Like, if you would break something, you would probably see at some point an outage in one of those services. It was, at the time, I think it was 200 or 250 million lines of C++ code with— 250 million lines of C++? First party, not counting external dependencies. And Google had that kind of— I think that was the public number at the time— 10,000 engineers probably at the time making edits to that codebase in a month. And so our job was, we joked about— that's a New York reference— we joked about it as our job is the logical equivalent of Google has a new technology for skyscraper foundations. We're replacing the foundation under the Empire State Building. We're not allowed to close the building or block traffic, and we discovered nuclear waste in the basement. It's okay, guys, we got this. Like, it was a magically— it was a great team. I just want to put the 250 million lines of code in perspective, because today with AI, AI can generate— like, you know, we have crazy numbers. It can generate tens of thousands of lines of code in an hour, et cetera, and you can get to millions if you want to. Honestly, just an agent can do it. However, pre-AI, Uber's mobile application was about a million lines of code on iOS and on Android, and that was each written by 200 engineers for years. Bun. Runtime at Anthropic just rewrote something like, I think it was like 500 or 900,000 lines of code, and that was written by a small team. I find it rare to find examples of a project that has more than pre-AI, more than, let's say, 3 million lines of code, and that will typically be hundreds of engineers writing it for several years. So in this, the reason I was like 250 million, that is about 100 times more than I've heard for codebase. And we're talking about, like, clearly this is written for production. You're not, like, just goofing around and pushing some scaffolding here and there and write. And when you said first party, it's easy to have a lot of code when you take on an open source component, but no. No, no, that was all Googler authored. Yep. And so we have some insight into how things grow. Okay, so the largest refactoring history at that point in human history might have—you're not exaggerating. Pretty good, but very expensive and slow. Right. And this person said, like, hang on, like, what if we told the AI, okay, like, let's do a thorough code review every now and then, but when it finds something, let's have it generate a lint rule. Yes, please. Which is cheap and repeatable, and it will no longer take 30 minutes, the cost, and also it's just non-deterministic. It might or might not find it next time. That was frustrating. Like, this person was like, look, when we know we have certain things. So now what they're doing is every now and then they still run some of these reviews, and of course, but whenever they find any issues, then they fire up inference AI, several agents, and it's like, all right, can you come up with an improvement in our deterministic linting can help it. And the inspiration came actually from this person heard about how Uber—actually, I think it was heard me talk about when I joined Uber, I could barely commit my code the first week. It was so painful because the linter just kept rejecting it because Uber had this engineering culture that, pretty AI, whenever you saw something break, you just put a lint rule. Architectural rules were all in lint rules. You could not have something like—we had an architecture, and you could not inherit the wrong way because a linter would say, like, uh-uh, you cannot inherit from, like, a builder, cannot inherit from a, I don't know, whatever composer or whatever that was. Yeah. No, I think it makes perfect sense. Like, let's try to learn our hard lessons only once, codify those rules somewhere deterministic, apply them equally and appropriately and fairly, like, and leave the interesting parts of code review. Like, make sure that the parts of code review that we're focusing on are only the things that humans need to be doing, right? Which was true then. I don't think everyone invested in linting and static analysis that quite to that extent. They should have. And then it's equally true now, right? Like, we can't review everything, but, like, there are places that require— Like this human attention is well spent. Titus just made the case for learning the hard lessons once, then writing the damn rules, and saving the human attention for the things that really need it. That is pretty much the same way you want permissions for AI agents to work, which brings us to our season sponsor, WorkOS. WorkOS provides the auth infrastructure that B2B and AI-native companies need to sell to enterprise customers with SSO, SCIM, RBAC, and now authorization for AI agents. WorkOS Airlock is intent-based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it. Clear-cut rules like no agent deletes cloud resources are run as plain code. For the fuzzier calls, an AI judge evaluates the policy at runtime. You write those policies in plain English, and Claude can pull the ones you already have out of Notion or Jira and import them for you. Your agents, whether that's Claude or Devin, connect through a single access point, and your MCP servers are behind this access point. Airlock is a new layer on top of your existing permissions, which means that an agent can never get more access than the person who invoked the agent. When something risky comes up, Airlock can escalate to a human for approval, say over Slack. It's the same idea Titus described: write the rules once and spend human attention only on the calls that need it. You can request early access at workos.com/airlock. I also want to mention our presenting sponsor, TurboPuffer, a ridiculously scalable, fast, and cheap hybrid search engine built on top of object storage. How often do you get to see a rewrite playing out in real time? With the team of TEEP, we finally can. They have completely redesigned their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. The new system, called TurboPuffer v3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, v2. The TEEP team had been perf-maxing query plans until they are faster or faster than v2, doing all this in public all the way up to the v3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to being several times faster on things like cold vector search, order by attitude, or hybrid search. The v3 system is still slower in areas like full-text search, but only 1 to 2 times slower versus 100 times slower when this perf maxing started. And the team is sharing exactly how they optimize things in their development logs with things like group count aggregation, compressed column encodings, batch scoring, and many more. A lot of these wins are basic foundational work right now, but the team will be sharing some very cool systems work over the next few weeks. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is Turbo Buffer. Teev started as a vector database, but it is no longer a vector database. It has evolved into a high-performance database for search. The v3 discards the vector primary index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast search and just generally a very fast database. The Turbo Buffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built. Speaking of hard, hard lessons and documenting them, can we talk about Software Engineering at Google, the book that you're quite well known for? How did the idea come? How did it come together? What was your motivation to start with? So this was a little bit the tail end of, hey, maybe we'll ship the developer tools. Maybe we'll need to explain why Google's developer tools are the way they are, because otherwise it'll be a little bit of like aliens came down and are trying to give you something that doesn't make any sense in your world. Part of the reason that we had the hubris to, like, tackle that sort of task was because we had been doing this refactoring at scale, you know, everywhere. And when you run into everything that any engineer thought was a clever solution to their problem, it meant that my cadre of weirdos have run into so many, Oh, yep, I can see how you thought that was a good idea. Don't do that. Like, more than anyone else out there. Well, you're talking some of the most kind of battle-scarred people in the industry, just because you're working on these large-scale systems for such a long period of time. And it's not the production battle scars. It's the, like, testing and, like, local code design battle scars. Like, everyone else in the industry has, like, all of this production experience, and, like, my job ends at merge. Like, I know more about development and API design and test and keep the codebase as an artifact working than most. But it does mean I'm kind of an outsider in that sense. Well, interesting enough, the, I mean, the book is very, very thorough on how it gets to production, but it does not cover the operational part, which I guess is— That's the SRE book. But there is an SRE book, which— Well, then this book would be, like, twice as long. How did you pull the book together? Because you wrote some parts of it. Other folks inside of Google wrote different chapters of it. So it's very much in the spirit of the SRE books, like, try to get the expert on or an expert on given topics to contribute something useful. And then the three names on the front of the book, myself, Hyrum Wright, and Tom Manshreck. Hyrum and I were kind of the technical leadership for that, and Tom is plenty technical, but has been our tech writer friend for many, many years. So he handled a lot of the, like, editorial, structural, organizational stuff. But Hyrum and I had been the tag team of— Hiram generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the codebase. And so, like, everyone knew us as, like, those two guys that clean everything up. So we had those scars, and so we led that. And then the trick was, like, finding the experts to write something high quality. Some of those were great from the start. A lot of that took, you know, a bit of ghostwriting and stuff. Yeah. The whole task was actually almost two years. Certainly not full time. Like, we were at most probably putting in quarter to half time most of that time. But yeah, like, we wrangled together way too much text from Look at some of the leading AI companies, the leading AI labs, the Anthropics, the OpenAIs. When I talk to people there, I see standout individuals who are working as a team. You know, like Claude Code, for example, they cleverly have a public face in front of both of them, Boris Cherny, who created it, and I think they do an amazing job. It makes it very relatable that this is a human. There's a team behind Boris. Yeah. And that team, when I met them, you can tell they gel. They work together. They trust each other. You know, like at some point Boris will be on holiday, and you will not know because things just happen. Right. Like you said, the important keyword, right? Trust. Right. Like a substantial part of this is the psychological safety aspect of it. The other thing, which I think is natural for the AI labs but hasn't been nearly as dominant for the rest of the industry yet, is you're learning as you go. And if you look at all of the research, you know, like DORA research on stuff, or any research on productivity or team efficacy or any of those things, like at root, it always boils down to culture, psychological safety, and the permission to learn. Right. It's the permission to admit that I don't know something. It is the incentive to learn from each other. It is the incentive to share your mistakes so that the team only has to do that once. Your lint rules or SRE postmortem culture. Every effective structure and organization in tech is actually about information flow and learning. And I think a really great perspective on that, I love Kat Hicks, the psychology researcher for tech. Like her book, The Psychology of Software Teams, is beautiful. And like she talks about, you know, these things a lot, right? Similarly, there's a platform engineering at Google book that some of my friends are putting out, is early release chapters on O'Reilly. I might have written the chapter on education for that. So very surprisingly slash not surprisingly, right? But yeah, like education is the powerful root for, like, let's make this mistake a minimum number of times. Let's codify what we've done. Let's give people permission to admit they don't know and to grow, because it's not a one-off game. It's an iterated game, and you'll do better over time if your team is better next time. I want to go back to psychological safety, because back about five, ten years ago, we kind of, we talked so much about it in engineering leadership circles. This was pretty well understood that, you know, a good team feels a safe team. If you, for example, make a mistake, you cause an outage. As an engineer, like literally, you push a code and that code causes outage. We should have a blameless postmortem where we don't say that, hey, Sergey pushed this code and he caused the outage, because that's now, I mean, I feel terrible. But let's talk about, like, for example, how could have we avoided this at a system level? Why did the test not fire? Why was it rolled out immediately when he was a new joiner on the team or she was a new joiner on the team? Like maybe some onboarding could have helped, yada yada yada. How do you define psychological safety and why is it important? Especially just playing devil's advocate, like when we're talking about smart engineers who know stuff, like is it all that important to us? Because we should know our thing, right? I'm just like, I'm kind of removing my emotional hat here. Yeah, we should know our thing, sure. So I would say, it's been a little hard to, like, root this stat, but from talking with people in Dora, the research, the one thing that I love was we can explain 30% of the variance in whether or not your team will hit its goals based on how you score on psych culture questions, right? Like we could give you a 15-question true-false. Or maybe Likert scale, like a 15-question assessment for the team and predict 30% of whether or not you will hit whatever your goals are. That's insanity, right? That is more powerful than any other signal. Like, okay, so you think that we're all engineers and this is all just computers and tech and emotions have nothing to do with it. Find me anything that's even remotely that powerful predictor of success. It's a people problem, like, period, right? And going back to, right, it's not a single-shot game; it's an iterated game. I regularly cite Fred Brooks and The Mythical Man-Month essays, right? And in The Surgical Team, like, The Surgical Team is my least favorite chapter in that book because it's structured as if we're only ever going to do one project. Get the best person to do this, have everyone else build the tools and do the documentation. It's very hierarchical, right? And it's very, like, power dynamics mad, yeah, right? But it might make sense if you're only ever going to do one task. Instead, we're playing, like, in a game theory sense. It's not prisoner's dilemma; it's iterated prisoner's dilemma. There's a different strategy. And the strategy is find the most junior person that can do it with supervision so that they learn and grow, and you still get a decent outcome, and that next, like, the higher-level people can solve something that they don't know how to do yet or that no one else can tackle, right? This is the only way to, like, structure it so that growth happens. But in order for that growth to happen, people have to be willing to admit that they don't know a thing. And, like, that brings you straight back to psychological safety and learning culture, right? Like. If it is a culture that highlights information sharing, learning, and growth, you will succeed. And you mentioned that the AI labs probably take this for granted, and I'm gonna— Because they don't know what they're like. Well, I'm gonna paraphrase it. Like, with the AI labs, a few things going. First of all, business is booming. They are growing. Revenue goes up. Customers goes up. Funding goes up. Headcount goes up. Their teams are only ever expanding. They're in hypergrowth. Like, we've seen companies like Netflix and Uber in hypergrowth. This is even faster. Your equity also goes up. So you don't need to worry about things like performance reviews, because performance reviews are typically important where, like, how much bonus I'm gonna get, how much. But when you're at a place that goes like this and your equity goes up, your financial part is taken care of just by focusing on helping the company succeed regardless of your titles. Second of all, because they're building new technology and they know no one knows, so they just tell, Okay, I don't know. I'm learning from you. Here's what I learned. So it's just all natural. Whereas compared to a company that might not grow like this, they might still be using, let's say, AI, but now there's less of an incentive to say, I don't know, because there's a performance review coming up. You want to look like the smartest. You don't want people to say like, Well, this person says, 'I don't know' a lot. Your bonus will be lower. You might be put on a performance improvement plan because you're now at a company that every year lets go of 6% of people, right? And your goal is suddenly going from like, I just don't want to be there. I want to hang on to this job, but it's now hanging on. It's now— and you also, you're now wary of maybe your teammates, because now there's internal competition. You want to hog that project. Yep. And so this is a huge difference. Yeah, it's huge. Like, if you get into the mode of believing that the team outcomes are what matter for your own success, then it is much easier for you to realize, like, okay, I don't have to have this one shiny task. Like, the team will be successful by virtue of someone doing this, and it might be my turn next. If, when it breaks down into that, like, individualism and, like, I don't trust the guy across the way from me or, you know, right shoulder to shoulder to me, like, even though we're on the same team, we're scraping by and, like, you know, fighting for attention and success. And that's not a recipe for innovation and growth. Like, that is a recipe for stagnation and just bad. One of the obvious things that I assume, you know, does not do good for psychological growth or is terrible for it is layoffs. Can you tell me, based on your observation, and we've seen, right, like, we've all been part of it, and in 2022, 2023, there has been a series of layoffs. Right now, companies are doing it again. Back then, it was to do with... Then that's a different game. And the problem that I think that we see with AI, even with a clever, you know, index metric approach to trying to capture the gestalt of productivity, if you're not looking for the value of a given change is substantially different, then there's no way you'll ever be able to capture that in the signal, right? And we don't have anything that says the value of a change of this size has changed. So, for instance, like, it is entirely possible at this point, and I hang out in security spaces and do a lot of low-level, you know, C++ library stuff, you can write a JPEG parser in a couple weeks, right, with whatever additional features you need. That would be insane to say that sentence out loud three years ago. Okay, so what you're saying is what none of the measurements really capture is the possibility, the impact, the effort it is to do a massive innovation, which right now with AI is happening. Right. And I'll give you also one example that I—and tell me if this is what you mean with it—is there's a friend who works at a CTO at an engineering company about, like, a mid-size engineering company. They're a niche industry, and, you know, like, the CTO, they have been looking at, like, release frequency, the usual metrics, talk with their engineering team, are they happy, are they not? And of course now they're adopting AI as well. They're seeing that, well, people are more efficient, and just privately this person was telling me, like, look, like, I kind of know that for a fact that we could probably do the same amount of work with, like, half the amount of people, and then we could be cashflow positive. But of course there's a morale thing, so he's like also, like, he likes the people, so it's like, I'm not sure I want the emotional package on me. But we can see some people are picking it up faster and some slower, and we kind of left it at that. A few weeks later, I'm talking with this person and says, like, you wouldn't believe, but we're probably going to be cashflow positive. I'm like, how? Like, did you go ahead with the cut? He's like, no. What happened is one of our engineers came to me and said, like, we license this component. Think of a library, like it's a binary, but that was written by a vendor in the space that has a high-performance thing that they use in their software. It's a core part of it, and they have a revenue share where they pay, like, I don't know, like 5%, 10%, 15% of their revenue for this component. It's so important. It's obviously licensed, closed source, but it's really important, and it would take them years of a five-person or 10-person team to write it, and it might or might not work, so it's not worth it. The engineer came and said, like, what if we just gave the specs? Because they have massive amounts of tests around this thing, a test harness. They gave it to AI. AI wrote a component to them which is faster and better than what they have, no prior domain knowledge. And at first they thought, well, maybe we'll open source it, but then they told the supplier, and the supplier said, like, it's a huge business for them. They're paying them to not open source it and to give them the implementation, which they can relicense. And suddenly the game was completely changed. Like, if this person would have gone ahead with, let's say, the layoffs, this engineer might have just got it and might have known that the game could be changed because of this. AI makes crazy things sometimes possible. Right, exactly. And so I think that this is the insight that tries to bridge the gap between—I have not seen a research-based number that says productivity post-AI is up more than like 20 or 30%. But then everyone talks about, like, these 10x sort of experiences. I super don't think it's actually 10x, but, yeah. But, like, how do you bridge that gap? And I think it's exactly this thing of a lot of the value of AI is not in doing the same thing faster; it's doing a harder, more valuable thing that you wouldn't have even thought about. And if your productivity measures are the logical equivalent of counting how many widgets are coming out of the factory, and then you start producing super gold-plated deluxe widgets, there's no way to measure that or track that over time, because we just don't have the tracking of what's the value of this component or feature. And that also matches with, you know, talking with senior engineers that are like, yes, we're up 70% on amount of code, we're up 30% on outages. Number of features is the same. The thing that we can't know for sure is, are the value of those features up? Because how do you do that? One challenge that AI is giving is entering the market for new graduates. There seems to be this expectation. Companies are very eager to hire senior engineers who know how to use AI, and they have taste and judgment, and we have these things thrown around. It's not deliberate, but whenever I talk to the AI labs, they always tell me increasingly that they look for taste, judgment, agency, these kind of things. Let's go back to education, to computer science education. What are your views of how today's computer science education works, especially for how we know the industry is changing? And what are some of your ideas of what could possibly be changed? I will riff a little on my bona fides there to add some weight to this. So after the book, I got recruited into the ACM IEEE AAAI work on setting the curriculum requirements for an undergraduate CS degree. So this is like the professional societies saying, these are the things that we need to be doing. And for the steering committee for that work, I was on the steering committee. I was the only currently working engineer. Everyone else is academic. Okay, in some sense, because, you know, it's what can we teach? And I managed mostly the software engineering requirements. But it does mean that, like, I see exactly what the requirements are and have been living in that space for years. And bluntly, and I gave a talk on this in a pre-AI keynote, bluntly, most of the CS curriculum hasn't substantially changed since, like, 2000, maybe earlier. Right? The average thing in the curriculum is exactly what was needed to build a database or an operating system in the '90s. That doesn't sound good, does it? Not for a field that prides itself on innovation, right? And, like, there's all of these things where, like, I have to genuinely ask, like, are we teaching this because it is relevant and important, or are we teaching this because it is tradition? And the history of— Or we know how to teach it. Right. Or we know how to teach it. But because, like, the history of, you know, software engineering or CS departments, like, we're only at 50, 60, 70 years for the earliest CS departments, right? Like, this is a field that is way, way too young to be doing stuff out of tradition. Even pre-AI, I was like, I think a lot of this needs to be burned down because all of the curriculum requirements, all of the teaching has mostly been through accretion. Like, we just throw more stuff in there and cut the amount of depth on the existing stuff by epsilon. Yeah. Right. And, like, I don't think that works. Even pre-AI, I was saying, like, the thing that I really want is, like, you spend more time— Practice and drill and actually become, like, a fluent programmer, algorithmic thinker, not you have a trivia understanding of a million different topics. And now AI, right? Like, we guarantee, like, you can do all of an undergraduate degree without ever thinking if you can just feed it into an AI, because it will do just fine on all of those programs and assignments. And so a lot of what I'm pushing for is we need to get out of the code. We need to up-level substantially. Like, you need to be able to read code and think as a systems thinker and, like, algorithmically, like, understand the structure and execution. Like, I'll take it. It's better than not doing it, but— Well, we have what we have, and this system will hopefully, with people like yourself and others, will hopefully start to also change or just adapt to the times. But AI is right now here, and one big question more experienced engineers are thinking about, like, we've learned to build software pre-AI. We know the fundamentals. Many of us went through computer science, so we actually have a very strong theoretical understanding, which we might or might not use some parts of it. But what about for new grads? Now that they'll enter a workforce of just today with just the capability that we have, what do you think the fundamentals are that they should focus on, and what are the things that they should just rely on AI? There's always the question of, like, should you just ask AI for this, or should you learn the basics, right? Yeah. So, I mean, I think learn the basics is an entirely loaded question because nobody agrees on what the basics are. And that's fair. Kent Beck said a couple months ago when we were last together something like, Everyone that's currently working has grown up in an era where there was a playbook. Like, we know at least in theory the right way to do this, and there's no playbook right now, right? So we all have to get a little comfortable of not knowing the answer, and we certainly don't know the answer on what is still the basics. The things that I think that the juniors should be focusing on: one, the resilience to avoid just copying your ticket into an AI prompt and then trying to check in that result, right? Like, if that is all that you are doing, you can be replaced with a very small shell script. Like, that is not a defensible advantage for you being on staff. I think it's really important to use AI to— Better yourself and to widen your inputs, but not to immediately produce all of your outputs. There's a metaphor that I really liked from about a year ago at a think tank. We called it Andrew's axe. And it's, if I go chop down a tree with an axe, I do not say that it was the axe that chopped down the tree. It's me that chopped down the tree. The AI, as much as we occasionally anthropomorphize it, right, is still a tool. And 1979 IBM training manual: A computer will never be held accountable, therefore a computer should not be making management decisions. The AI is still not accountable. The AI labs are not going to indemnify us for, you know, the mistakes that these things produce. So you have to be able to stand up for whatever your name is on for those pull requests. And that requires a lot more rigor and not cutting those corners. And, like, that attention to detail, I think, is really hard to train. Going back to sort of the undergraduate version of this, the scary story that I've been hearing from a lot of profs is the best students, like the B, B-plus and above students, they are better than they ever have been. Because they have—yeah, because instead of relying on a TA that is available, you know, a couple hours a week and is one or two grades above them, they have an infinitely patient, always available, incredibly well-read, sometimes hallucinatory tutor, right, at the drop of a hat, right? They can always learn and explain and figure out. And they actually have the autonomy and the drive to use this, right, and the metacognitive resources to avoid that, like, I'm just going to copy and paste, right? The very worst students are somewhat better because they used to not be able to do anything, but now they can produce slop. Yeah, they just throw in slop, yeah, which is a bit hard to detect, but I get your point. But everyone in the middle has kind of collapsed. They don't have that resilience. They would rather go to the party or play video games than do the homework. And so they're like, Well, just this one time, I'll just get in the habit of doing this thing, and that slippery slope just gets worse, right? And so, like, there are a lot of profs from decent schools that are like, Yeah, I don't know, man. Like, I'm not sure that anyone, like, below, like, the vast middle, everyone below a B, might not understand how code works at all. Like, the system is just not set up to be tracking that sort of thing. Or at least giving them feedback, for example, alerting those kind of things. Right. And so a lot of the discourse initially was, How do we prevent students from using these AI tools? And I think that that's nonsense because it's echoes of, we got high-level languages and there's compilers, like, But now you're not going to learn how to program in assembly, right? Like, the industry is using it anyway, right? They're going to be asked to use, they're going to be forced, the pressure to use it harder in some places when they get a job. Well, it might be, it will for sure be a red flag if a new grad says, like, I do not use these tools. Like, come on, right? Have you not—yeah. Do you not want to work in this industry? Right. It's an open secret what we're doing. We're publishing it on blogs and everywhere. Yeah, exactly, right? And so the line of, Well, we're going to never let you use these tools, cannot possibly hold. I co-published a piece in Communications of ACM with a bunch of academics this month. That, I think the opening line there is, Imagine forward to 2040, and we hold the line treating all use of AI as cheating. But a generation of students, AI, you know, they've grown up with AI in the same way that the internet natives have grown up with the network, right? If we continue asking them these traditional questions about, like, find a palindrome in this string, or measure the volume of rainfall in this graph, right, like, are they going to learn anything from a traditional CS degree? The answer is obviously no. So, like, if we know that we cannot stay here, and there's plenty of things that need fixing, like, let's burn it down and do something that matters and figure out how to use this appropriately. And I think finding ways to assume that they're using AI, and what are the new learning objectives, and how do you make that still frictionful in the ways that cause them to think and learn about different things, like almost certainly at a higher level, right? But, like, that's actually the key to undergraduate education and curriculum design and assessment. Well, it sounds like whoever—so the professors, the TAs, the people who are designing these courses, they also need to put in a bunch of work because no one has this figured out. You need to experiment. You need to first use the tools yourself. Yep. You need to come up with novel ways of creating an assignment which you need to use AI to solve, but AI itself is not enough. How do you get the feedback? You might need to put in more time. Let's say, like, sit down with someone afterwards and have them explain to you, because, you know, again, like, interviews are also changing, and one of the interviews that seem to be working well is when you're pairing with someone, you know, like video or in person, you're using AI, but you're explaining what you're doing and how, right? Because now you cannot just fully fake it and go away, give it to a friend, or just have Claude do it. You're now showing your cognitive ability. The interviewer can push you into uncomfortable—you know, they'll get a gauge of, like, how expert you are and push you further and further, and so on. Yeah, exactly. And at the undergraduate level, like, there is some move towards more oral exams and those sort of things. But also, I think a really simple solution is stop grading just the outcome. Focus on the process, right? Like, even if you don't come up with wildly harder assignments for assessment, stop grading just the code that they turned in. Grade instead, what are the prompts that you used to get here? Also, what are the tests that you provided? Like, how did you make sure that this thing And this was all pre-AI, so we're not talking about overly verbose codebases generated by AI. As comparison, Uber's mobile apps had codebases of about one to two million lines of iOS and Android code back in 2016, and Uber crammed a lot of business logic into those apps. How Titus and the team cleaned up these foundations without using AI was also pretty educational. They built deterministic code generation tools to parse the code, match patterns, then generate edits. This is an approach that still could work more reliably even today than just asking non-deterministic AI to refactor a codebase, even though it is arguably more difficult to write advanced parsing logic on a codebase like Titus and team did. I also appreciated Titus's view on where AI is helpful today. He mentioned this quote he heard from a senior engineer that said, We are up 70% on the amount of code, but we're also up 30% on outages. The number of features is the same. Well, that doesn't sound all that good. Where's the value from AI then? Titus says that we won't get all that much value from doing the same stuff we did before AI, but faster. The value is to be more ambitious and do harder things that we would never have attempted before AI. Finally, I hear a lot of worry for new grads in tech about how they'll get a job or not, especially when AI agents are so commonly used, but CS programs have not incorporated them into their program. Titus is one of the few engineers who has been part of a steering committee for the CS curriculum and says that this program needs to either change substantially and be more practice-oriented, or CS enrollment will plummet. Titus also says that it might make sense to finally have a software engineering curriculum that is focused on students getting far more experience building software with these modern tools. I don't think any of us has an answer to what will work, but having more people from the industry get more involved in refreshing the CS software engineering curriculum sounds like a step in the right direction to me. Check out the show notes for related to pragmatic engineer deep dives on Google's engineering culture, measuring developer productivity, and the impact of AI in software engineering. If you enjoyed this podcast, I'd greatly appreciate it if you left a rating on it and perhaps a review. Thanks a lot, and see you in the next one.