Building resilient systems with Sam Newman

Sam Neumann is the author who wrote probably the most widely read book on microservices, and yet, as you heard, he said that microservices are an architecture of the last resort. But why? Sam was in the room when the term microservices was coined. I find him to be one of the most thoughtful engineers on this topic, as well as on resilient systems, and his latest book, Building Resilient Distributed Systems, is out now. Today we cover, Specs vs. Code as the Source of Truth, and Why Sam Says that Production is Truth. Idempotency Keys vs. Fingerprints, and Why You Should Avoid Idempotency Keys When Possible. The Four Dimensions of Resilience, Robustness, Rebound, Graceful Extensibility, and Sustained Adaptability. Cognitive Death and Cognitive Surrender, and Sam's Advice on How to Better Use AI to Avoid These, and Many More. If you want to understand what it takes to build resilient systems, from an expert who has been knee-deep in this topic for 10 plus years, this episode is for you. This episode is presented by TurboBuffer, the AI retrieval layer built on object storage. It's fast, cheap, and extremely scalable. And DropVic, Cognition, Notion, and many others are happy customers. Before we start, let me mention Tire, our season sponsor, and our product and team I'm very bullish on. You probably already know that Git hosting has a problem, AI agents. Git is increasingly becoming a bottleneck for modern, agent-heavy software development. Engineers like you or me are creating more code with agents, those agents push more code, and most of us are running parallel agents, which push way more code. GitHub is clearly struggling to keep up, and has frequent outages. So what's the solution? Entire was founded by GitHub's last CEO, Thomas Dumka, who, by the way, was a guest on a podcast when he was still at GitHub. And Thomas revealed Git hosting for the agentic era from scratch. Entire was built to be very fast, and to have your repos regionally close to you, to reduce latency, and this allows for fleets of agents to push in parallel. And Entire's performance is next level. Their repo can handle 418 pushes per second, which is up to 89 times faster than every competitor on the market. When GitHub is down, you can still keep working, and you don't even need to migrate away from GitHub. You just sign up to Entire, and the platform mirrors your repo. And here's a neat thing. Have you ever wondered what prompt resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself, oftentimes. Entire captures all the prompt history with your agent, right inside of the repo, easy to check back. And it's got a pretty innovative UI to show all of this. If you're looking for a Git hosting that works even when GitHub is down, head to entire.io slash pragmatic, install the CLI, and mirror your repo with a click. I've already done it, and I love it. Oh, and did I mention that it works with any agent, and it's open source? Sam, it's great to have you on the podcast. Thank you so much for having me. I'm happy to be here. I'm really happy to be here. I'm a big fan of the show. As they say, I'm not going to lie, I've become a bit of a millennial, and I think I watch a lot of shorts rather than watching the whole things now. I'm becoming my son. Yeah, millennial is a new boomer, apparently, I've learned. I was happy enough being Gen Y, now I'm wherever I am now. Well, so let's go back to the beginning. You've been known for a lot of your books, your takes on resilience system, microservices. Before that you worked at ThoughtWorks, but let's rewind to the very beginning. How did you get interested in tech and computers? I'm quite fortunate. Living in the UK, we had places and access to computers quite early on. We had an Acorn Electron at home, which was sort of one of the generations of the BBC computers that came after the BBCB. And then I got a Spectrum 128-3, which is the Spectrum that had the built-in tape drive. This is way before the Commodores. I think some of this was, I think I'm pretty sure that the... And that's why they don't want you to pair program. Well, then you've got a fundamental problem that needs to be dealt with, and it's not a conversation about pairing. The conversation is, what do you think programming actually is? Well, but let's bring this a little bit forward, because right now we're having the same conversation around, right, for a lot of CEOs, even founders, are thinking who have been building tech companies, you know, they all look at what is the most time-consuming part of what we do in building software. Fifty percent of the time has historically been writing the code. So a lot of people think, you know, like programming is code, and LLMs are really good at turning out code. So what is programming? Okay, there's the programming, I think it's Peter Norvig's article, the guy who came up with the Barcus-Norvig form—well, he's Norvig in Barcus-Norvig form. He wrote this paper years ago, which won the Turing Award for, and he sort of describes, like, the program as being a theory that lives in the mind of the developers. That it's actually us together, working together collectively, who builds a program. The fact that we happen to make that program work through code, the code is really a side effect of how we work together. And I think product development is just that extended, right? What we are building is collectively, we are building a system together, and the code is one of the mechanisms we use to deliver the system, but it's not the only thing. So I think holistically, that's what it's all about. It's just that we focus on the code because that's historically been where we sort of spend the time. But, I mean, this is also classic theory of constraints stuff, though, right? If you speed one bit up, you get constraints elsewhere, and it exposes friction points elsewhere. I would also argue that a lot of the time, coding was not the problem or not the bottleneck; it was how the coding was done, delayed feedback cycles and things. But I do come back to that. Pete, and or Pete again, because it's like Margaret Ann Story referenced it recently in her paper on cognitive debt. But it's like building software is a collective, it's a collective endeavor. It's people working together who build software. Kent put it in his Tiny First book that software design is an exercise in human relationships, right? AI doesn't change that. And I think all that's really happened over, say, the last 15 years, in my experience, is that we've got a bit better at understanding that we need better working relationships, like people, product owners, for example. But that's what the programming is, the art of creating, it's the collective art of creating software. It's not just developers who do that. Never has been. And so after ThoughtWorks, you've gone independent, right? I did three startups. None of them worked out for different reasons. So don't ask me for advice on how to make a successful startup. And then I went independent. Yeah, moved back. I was living in Australia at the time and moved back from Australia. And yeah, I've been independent ever since. So doing kind of a mix of stuff really ever since. I'd written the first book about microservices, sort of in my spare time at ThoughtWorks. And then that book came out, and I'd been at ThoughtWorks a long time. I realized that there's always a danger of becoming institutionalized. You get to a point where the company, it's like, I think there's things as companies do to change, and I've tried changing those things, and they don't want me to change them. So either I'm bad at changing them, which is probably true, or they don't think it needs to be changed, which is also probably true. It's like, I could hang around and get bitter, or I just go now. I just go now. So I went off and I thought, I went independent. It's been really interesting ever since, because it's sort of, on one level the shackles are off. The problem of working for yourself, though, is that the boss is frequently an arsehole. Development was the process of us trying to model existing non-digital processes and how you do that. I wonder if we will have a different but similar problem of the same thing. Now that, you know, we know AI tools are here, we know that, for example, they're great at generating code, and we have two types of companies. We have a company, a digital company, that has pre-AI their use to, you know, writing product requirement document, deploying developers, product managers, designers, QA. They all have their roles, and it works like a really well-oiled machine. And now you have these so-called AI-native startups who just, again, they have a few people. They're just, from the start, they're using agents for everything. They have a very different process. And at some point when this whole thing will mature and we will have best practices, we'll have the existing companies who, like the previous one, they will try to retrofit, you know, this tooling to how they're working today. It's already happening. I mean, and the thing is, enterprise companies will rarely be successful doing those first-gen things because ultimately the corporate DNA will just crush that stuff from within, right? And it will take a while. But I think the whole industry is grappling with the idea of, like, are we now moving to a new level of abstraction, right? So software is all about abstractions, right? But, you know, we went from doing machine code to doing assembly, went from assembly to general-purpose programming languages. And you're coming back to the idea of the program being the mental model. We would evolve that mental model at the code point of view. That was the most effective place was us working at Java, you know, C#, whatever it is, rather than working at assembly level, right? And it's like, okay, well, is actually what we're talking about really, we're just stepping that level up. We're still actually building a product together. We're still thinking about what the software is. We're still building that mental model, maybe now more collaboratively involving, you know, more product people, for example, in that conversation when we're writing our specs. And so the question for me is, are we ready to make that step? And the answer is no, not everywhere. And this is a really good point because if you think about it, like, you know, like one of the mantras inside of Facebook used to be that, I think... Some something along the line that code is truth or something like that. But in general, in a lot of tech companies, if you want to know, like, how things work, like where the truth lives, is it in documents or is it in the code of business logic? It's in the code in the business logic. And if it's well-maintained code, often well-commented or structured, you can actually see the business logic and the rules for, I don't know, for example. See, I feel like I've got to channel my friend Charity at this point and say the truth is production. Everything else is the lies we tell ourselves. Production is truth. But I think in terms of, like, a definition of what the system should be doing, there is this ongoing debate about where is that definition? Is it the code or is it the spec? Brigitte Brocker did a nice article on Martin Fowler's website recently about spec-driven development, and she talks about the idea of some companies using being spec-first or spec-anchored. And a lot of it comes down to where they see the source of truth being. So a lot of companies are using things like Kiro or SpecKit to define a feature, and then the LLM executes that spec, and you can iterate on that process, right? And so a lot of people start off, they use the spec to write the code, and then they throw the spec away. And then you get other companies that keep the spec, and then when the feature changes, they just update the spec. And so there's sort of seeing a bit the code and a bit the spec. Maybe this is my attempt to unpack some of that tacit knowledge to an extent. I've made so many mistakes, and I keep making mistakes. I think making mistakes is useful, but I think also creating the opportunity where people can learn from other people's mistakes first is a good idea too. And so in the book, I also try and pull in real-world examples, and sometimes from big companies, sometimes from my mistakes, to kind of make this stuff seem real. And I think the other thing I think I really try and get across is that things go wrong, right? I mean, resilience engineering is all about making sure as many things go right as possible. It's not about making sure everything goes right. And there's also the only way you can build resilient systems in general is also by being honest about mistakes that have been made and talking about them and sharing them, because that's how everyone gets better. And if you don't do that, you can't build a resilient system in your company. And so this, again, is my attempt to sort of highlight all the awesome work that's gone before, the fantastic post-incident reports that people, that some companies share. Like when there's a system that we all use, or many of us use, has an outage. I mean, it's annoying, especially when you're using it, but I always greatly appreciate when the companies, because they don't have to do this, but when they share a public postmortem or incident recap, whatever that is, they go through what happened, for example, GitHub had an outage and they share the really interesting one on what all the things that went wrong and in what order. And then you also have companies where something like this happens and they don't do anything. So of course you're not obliged to in any way, but I always have a bit of a lesson. They are obliged to now, a lot of them. Regulations are now making them. Which I think is great. Yeah, things like DORA and stuff in the UK, in Europe, rather. And I think there's also, we should also separate around that post-incident report stuff, that there's marketing and PR versus have you done a proper post-incident review? Now, I do think you've a duty of care really to let users know as part of a proper post-incident review. But those companies that maybe say nothing, they might do a fantastic process internally. There's also the fact that some companies that put those things out, you read them and you're like, this does feel more marketing. Yeah. I think a great, if you want a good, a company that puts out some of the best self-flagellating post-incident review write-ups, go for Cloudflare stuff. They absolutely go for themselves in those. It's like, I feel like, guys, go easy on yourselves, right? Everyone makes some mistakes, but I quite like. And they do it rapidly within 24 hours, and they're a publicly traded company. Yeah. Which I think just goes back to the, like, some of them are like, we're publicly traded, we don't want to do it because it will give away information for shareholders. Yeah, no. Like, Cloudflare does it and they're fine. Exactly, exactly. And also it's— And it builds up trust. It is. I mean, a lot of how an organization is perceived is down to how you deal with mistakes when they occur, right? I mean, nothing ever goes perfectly for every company. So in your book, you talk about the three rules of distributed systems. What are these three rules? So I really tried to distill this down because I could never remember the eight, the rule of eight that came out of Sun, the fallacies of distributed computing. I always forget what those eight were. There's loads of complicated stuff, but I realized when you strip it all down, there's really only three things that really cause all the other problems. Right. The first is that you can't send information instantaneously from point A to point B. It's physics. I mean, now it turns out when you get into it and start talking to people about quantum physics, they say, aha, and they go, there's a thing called quantum entanglement, which does allow for the theoretical instantaneous read of electron spin states across arbitrary distances, at which point I point out that A— There were lots of... On AWS. On AWS. So AWS US East 1 region went down. It was the oldest cloud region, big region. Lots of people were there first because that's often where new services got launched historically. When that went down, it caused a lot of people to say, Do we need to be multi-region? And some of those companies would have had the conversation before and decided, No, we don't. And then they'd seen the fallout, and they may have changed their minds, and that is fine. This is what you mean by a resilient system evolves. Yeah. Thanks to the real world. But also, like, it is about what problems you're not going to try and solve. There were some big-name companies that got taken out when US East 1 went down and got severely, had severe degradation in terms of what they could offer. Companies like Zoom, for example, they weren't taken off entirely, but they were severely affected. Do we think that people working at Zoom are idiots? No. Do we think they knew that if a cloud region went down, they'd be impacted? Yes. They probably looked at that and said, Well, for us to come out with a multi-region approach, it's going to have this impact, it's going to cost us this much money, it's not worth it, let alone talking multi-cloud, because multi-cloud versus multi-region is a whole different conversation. And that's okay. And I think as long as you've had the conversation, and as long as—this is the other secret bit of it, of course—as long as what you think you need to do is in tune with what your users think you need to be doing. Yeah. This is why I make the point in the book when talking about the difference between SLAs and SLOs. The SLA is what you're going to get sued for if you breach. If you only ever deliver to the SLA, your customers are rarely ever happy. You've got to do more than that, right? Yeah. There is the social contract there as well. Yeah. Now, with building distributed systems, building resilient systems, it really helps to understand concepts that are not as straightforward, and I'm not going to go through all the things—you explain a lot of things in the book—but I do want to touch on a concept that I found a bit tricky to understand and then appreciate, and you really appreciate in distributed systems, which is idempotency. Yeah. What is idempotency? So there's a mathematical concept behind idempotency that I don't understand. But basically, an idempotent operation is one that you can apply that operation multiple times without it actually having an effect multiple times. So it has an effect once, first time. A really good example of an idempotent operation would be pressing a lift button. So you want to go down in the lift. You press the down button, it lights up. If I press that down button again, nothing's changed. The lift is still coming. Well, we hope the lift is still coming, right? Yeah. That's an idempotent operation. Some light switches are actually toggles. A toggle light goes on, toggle light goes off, right? That's not an idempotent operation. In computing, we have some operations which are naturally idempotent. Reads are. A delete is typically idempotent, right? But the example I think I use in the Introducer's concept is the one of payment. I was about to say that I have a lot better idempotent example for you. Yeah, yeah. Payment is a good one, right? So I'm going to pay you money. So I'm a service that wants to issue a payment, and I send you, please pay Sam a hundred pounds. Yeah. This is, I'm NLPing people back at home, right? Please say pay Sam a hundred pounds. And I don't get a confirmation back. I don't get back a response. The problem is, from my point of view, there are two possibilities. The first possibility is that you received that payment request and you've done the payment. It's been completed, and the result has got lost to me. For some reason, I didn't get the response. That's one possibility. The other possibility is that you never got that. You never carried out the operation. That sort of understanding of the relationship with the customer and the impact if they get it wrong, they're going to try selling the item. Keep that problem exactly the same, but now change the business context. Now we're selling concert tickets. So you're getting tickets to go see Tay Tay, right? You're going to go see Taylor Swift, and we're not sure if we've got the Taylor Swift ticket or not because our link to that system is down. But those Taylor Swift tickets are yours. You're taking your kids, taking your wife. It's all going to be awesome. We're going to Portugal. We're going to go see Taylor Swift. It's going to be amazing. And then we email you, sorry, we don't actually have those tickets. Now, the problem with that is, firstly, the devastation you're going to feel is a bit different. They can't back order a ticket. Yep. And so in that situation, it's like the impact to the customer is so much greater. And they now book the flight. Well, you might book the flight. So what they do is they make a different decision. So in that situation, if they don't know if they've got the ticket, they don't sell the ticket. I'm not sure if you cover in the book, but this is the fail open versus fail close, right? When you have an error or you're unsure, are you going to go through or not? And as you say, this can be the decision, the same problem. Two companies can take a decision. A company years down the road could take a decision. Uber had this classic example of when payment, we weren't sure if the payment went through. Do we allow the ride to continue or not? And early in the company's years, we always failed open. Yeah, just have the ride. We might have the money, and if not, we'll deal with it. As the company starts to go for profitability, we fail closed. Like, we're not sure we have the money if the charge went through, you're not going to get the ride. And again, same company, different focus. And if you're a developer, you needed to pay attention to, like, what is important right now for the business. It was growth initially, and it's okay if we lose money, and then now it's, I don't know, unit economics, profitability. But also that open-closed versus open, that's a fail-closed versus a fail-open discussion isn't one size fits all for the whole system. Exactly. It's on a feature-by-feature basis. And even then, you could say this taxi ride is 10 euros or it's 1,000 euros. You can make different decisions. Now, if you approach this from a mathematical and from a computer science viewpoint, we just talked about CAP theory. That was a conversation about CAP theory, right? So in the chat, I talk about here's CAP, here's PACELC, which is an extension of CAP. I explain what those models are because when you get deeper into distributed systems, you'll hear about them. But fundamentally, that's what this is about, right? When you have a partition, that's what the P in CAP really defines. I can't talk to one part of my system. Remember, it's rule number two. When I can't talk to another part of my system, what do I do? And all you need to know as a developer: I can't talk to that. What do I do? Sometimes we can make technical decisions. I'll try again, and I hope I don't bring my systems down with a retry storm, but I'll try again. If that doesn't work, what do I do? Well, actually, often the answer is, yeah, sure, you can look at CAP and PACELC and whatever else, but go have a conversation with the product owner. What should we do? The resolution of that often ends up being a decision about, are we trading off consistency versus availability in PACELC, or harvesting the yield if you're going into that version of Brewer's conjecture? But don't worry about that. That's in the book. That's why it's a later chapter. That's chapter like 12, I think, or 11. Don't start there. Start with having the conversation. Rule number two: I can't talk to this part of the system right now. What should we do? And that starts with having the chat. That starts having the chat. When we're talking about resilience, you talk about the dimensions to think about resilience. What are these dimensions? I start in the first chapter by coming back to that paper by David Woods I mentioned that was impenetrable to me at the start, and he... Or that, and they have trained these models to, like, default to coding. I'm not sure how this came, but it seems to be working to some extent. It's also because it's very tractable, right? You know, they are there to generate plausible code. They're there to generate plausible words, right? And in the context of code, they've generated some plausible code that we can now validate: does it execute? And so that's a really nice tractable aspect of it. And there are some places where it works really, really well, but there are other areas where it's terrible. Like, when I write my book, I use very little AI at all. Well, you don't use it for the writing; you use it for research. I don't use it for the writing at all. I use it for the research. But in the context of the research, I use NotebookLM, basically. But in that context, it's plausible. But then I just go and click all the links it gives me, and I say, was that there or not? I am checking the work. It's done way more work than I could have done going through all those papers. And God knows there's only so many papers you can read through, right? But in that context, I feel completely fine and able to do that. In other settings, you're getting now this problem called cognitive surrender, which is where you're using AI to write documents that you're never going to read. That's not really around a problem with the LLM technology. That's a problem with us. Can we talk about this cognitive surrender and cognitive debt? Yeah, cognitive debt's a kind of term that's been around for a while, and that sort of explains the erosion in our ability to think critically and to reason that can occur when we're using AI tools. And there's been a bunch of research into this that's shown—there's been some studies that's shown that people's ability to think critically is impacted the more they use chatbots and things. And I think we've sort of seen other examples. Margaret Atwood's got a great article about this, and she cites some of the research around it as well. And that's sort of explaining that. I mean, her thing is, you know, she's talking—we talk about technical debt building up in a system. We have to be careful that our use of AI isn't technical debt. We need to have a shared mental model of what this system is supposed to do, thinking about it narrowly through the lens of a program that we're building. And if our use of AI, as an example, we all go off in different directions, that shared review breaks down. That becomes a debt cognitively. Cognitive surrender is a different thing, which is like, Oh yeah, it's fine. It's basically LGTM, right? Yeah. But for whatever the AI tells me. Yeah, LGTM, what typical looks good to me, approving large pull requests where users couldn't be bothered, or you try to look at it and for whatever reason you didn't want to go through, or it came from someone who you trusted. I worked on the Google tool team on old stuff, right? And so LGTM was literally a thing with pattern match on and takes that as an approval. Yeah. So if you saw LGTM, that person's approved, right? And it was just—that was where I think that may even have been where the acronym originally came from, I'm not sure. I think it came from Google, yeah. Yeah, but it is this idea that—and it's become shorthand for, I didn't really look at it. Yeah. And I think that's that cognitive surrender thing is coming out in lots of ways. And, you know, we're getting AI to write documents that no one's going to read, and we've got AI that's reading documents that weren't written by other people. And so the cognitive surrender piece is really that the humans—it comes back to the human being in the loop. There's very subtle differences in those workflows and the kinds of differences it changes. Like there's a big difference—I Codocto talks about it—it's a big difference, like if you're, say, a cancer, an oncologist or a cancer specialist, there's a big difference to you as an oncologist working and an LLM doing... But we could do more to make the architecture of our systems that little bit more explicit. This is also why I like spending time in production, because especially if your modular structure ends up being a process structure, in other words, it's process boundaries, I can go look at the network to find out my architecture. I shouldn't have to, but at least, you know, I know production is truth. It's like football is life, production is truth. So where can people get your book? You can go and order it now. If you can, go and buy it from a local bookshop. That's always a great idea. It's also going to be available for people on the O'Reilly platform in sort of digital form, so you can go and read it as part—if you're a subscriber at O'Reilly, you can get it there as well. But lots of people love having the physical copy. And we have these in the show notes below, so check them out to get the book directly. And also, you've been doing some videos, right? Yeah, well, I've been doing some stuff on Dave Farley's Modern Software Engineering channel. It's kind of a good excuse to do some sort of semi-ranty monologues to video and have some fun chats. So yeah, go to the Modern Software Engineering channel on YouTube, and you can find videos from me and Dave Farley and Kevin Henney's there. We've got stuff from Kent, Trisha Gee, Emily Burch, Dan North is there as well. I'm pretty sure I've forgotten somebody else. There's almost a couple of other people there as well, but we try and put videos out once a week, and it's a bit of a fun place to— And we're also linking that one. Yeah. And if people would want to work with you, where can they find you? How can they find you? What kind of stuff can they get help from you for? I do internal talks for people, like internal events. I do customized training courses on a whole variety of things around resilient systems and security and microservices. I also do a lot of consulting for clients, helping them around sort of re-architecting their systems and stuff like that. If you want to know more about that, actually go to my website, samnewman.io, and you can contact me there. You can also find me on LinkedIn. But if you come straight to the website, use email. I'm still a bit old-fashioned like that, and I work with clients all over the world. So if I'm able to help, yeah, head over to samnewman.io. My contact details are there. Wonderful. Well, Sam, thanks again. This was great. Thank you. Thanks very much to Sam for this conversation. I really like Sam's three rules of distributed systems. One, it takes time. Two, sometimes the thing you want to talk to isn't there. Three, resource pools are not infinite. Sam said how most of the outages happen because of this last one, resource pools not being infinite and services are getting saturated. But I was chuckling when he was talking about rabbits. But stringing through networking cables between office buildings. Then again, a few weeks ago, with thousands of flights stranded on the East Coast of the U.S., it was because workers accidentally cut a telecom cable in New Jersey used by the FAA. Another food for thought from Sam is how failures should be handled differently depending on the type of business. An e-commerce shop might decide to show you the item when the inventory service is down. Worst case, after you order and pay, they just refund you because we're like, oh, we don't have the item. Basically, failing open is fine in this case. But on a concert ticketing website, you don't want to sell a ticket that might or might not be there. And so failing closed is the right call to make because, again, as we discussed in the episode, if you sell the ticket, people might book flights, they might get upset, you might lose them as customers. Finally, I appreciated how level-headed, perhaps even cautious, Sam is about AI. He's got no doubts that AI is very useful at the micro level, basically tactical, at things that we can already improve. But he is worried about the economics of the model providers. His suggestion is pretty pragmatic: hedge your vendors and your models. Be multi-vendor and multi-tool, and wherever you can, swap non-deterministic AI agent workflows to deterministic ones. Basically, have LLMs generate deterministic code, which I find