Scaling Slack and GitHub with Maude Lemaire

How do you scale a product that millions of people use daily when the architecture underneath it is starting to break? Today's guest did this both at Slack and GitHub. Maud Lemaire is a principal engineer and the author of the book Refactoring at Scale. She spent 8 years at Slack, worked on database scaling at GitHub, and improved Time-to-Verse Token at Cursor. Today we cover Low testing with 500,000 WebSocket connections, a job where using headless Chrome instances were too resource intensive and too expensive Why GitHub had only 16 databases charged for all pull requests and issues Scattered queries were a massive headache inside a large Rails monolith like GitHub's Why platform and product engineers should rotate between teams And more If you want to understand what scaling up looks like inside the products you probably use every day, this episode is for you. This episode is presented by TurboBuffer, vector and full-text search built on object storage. It's fast, cheap, and extremely scalable. This episode is brought to you by Antithesis, the platform to release with confidence. In today's episode, we'll talk about rewriting massive production systems as they scale up, which is a kind of engineering challenge as rebuilding a 747 mid-air. But why not just use CodeReview when using AI coding agents? It's because CodeReview is an increasingly naive and ineffective approach to do so, which is why Antithesis goes well beyond this practice. Antithesis runs your whole system in a hostile simulation. This simulation includes both targeted testing and fast testing. By running this simulation, it finds every bug before your users do. And because the simulation is fully deterministic, it doesn't only find bugs, it gives you a perfect reproduction of every issue, which makes it much easier to fix issues. The first thing I thought when I heard about Antithesis is that automated bug discovery and fully deterministic testing sounds like science fiction, but actually, it's hardware engineering under the hood. Jane Street, Superbase, Rapid Railway, and Datadog ship agent-written code with full confidence because they know it's been verified by Antithesis. And to give a sense of how Antithesis helps with large production systems, Datadog uses Antithesis to verify their new architecture when rewriting the Datadog event platform, a system that collects more than 100 trillion events per day. Today, Antithesis is one of many correcting strategies being used as part of Datadog's harness for their coding agents. To see more case studies and details, head to antithesis.com slash pragmatic. Mode, welcome to the podcast. Thank you for having me. It is so nice to have you here. This is a bucket list item, very much so. I'm very excited. Well, this is the first in the podcast. Usually, I start with a question. I'm not going to start with a question. I will show you a photo and I will ask, what am I seeing here? I'm seeing a baby. Yes, that's me. Who looks like how many months? Maybe about a year old, maybe a little before a year old. Yeah. Sitting in front of a computer. We're also putting this on the screen if you're watching our video. Same haircut, so that's good. So my question is, where are you born with computers? Very nearly. There are pictures of me much younger than that, actually, maybe like a couple months old. I truly like I'm barely keeping my head up where my dad has me sitting in his lap, like resting against him. And he's I don't know what he's doing. He's probably like grading some kind of assignment or because he was teaching. He taught computer science for 17 years before we moved to the US. And yeah, there's a bunch of pictures of me like holding ASCII tables, which now in hindsight is hilarious. But I ended up doing this sort of by accident. You'd think that all of the variables and the environment would have led to like a very logical sort of linear career trajectory. And that is not at all. So anyway, he had to think about that like eons before. But I thought it sounded really boring. I was like, no, no, no. And I really like my bio classes. I thought I was going to become a genetics researcher. That was my plan. And my parents always told me to chill the heck out because I had plans. I had when I started university, I started microbiology. I signed up for molecular biology, organic chemistry, all the things. I'd already accrued enough AP credits for all the Americans listening that I was technically like a second year college student rather than a first year. Oh, yeah. Rather than a first year college student. You want the speedrun college. Oh, yeah. I was ready. Skip the fun years. Get straight to work. Which I kept doing that. I had like a 19 credit semester at some point. I was like, why did I do that? That was such a poor decision. But I mean, I had fun doing it. So I guess, you know, my brand of fun is, I guess, different from other people's brand of fun. And yeah, I was going to do that. And then like three weeks in, there were a couple of things that really bothered me. One, it's not because you're good at something that you like doing it. And I think I conflated the two for a really long time. I mean, to be fair, when you're 18, like, what do you know? But I conflated being really good at memorization with liking memorization. And I don't know if you've ever taken a molecular biology class, but it is a significant amount of memorization. And you're learning the Krebs cycle. So I was going to switch into organic chemistry, into chemical engineering. And to do that, that's a different faculty at McGill. I mean, it's a very big school. There's a faculty of science. There's a faculty of science. There's a faculty of engineering. And I was looking at all of the requirements for first year engineering students. And there was a programming class. There was an intro to computer science. So I swapped out one of my bio courses the next semester with the intro to computer science class. And I told my mom, I swapped it out for a computer science class. Please do not tell dad. Please do not tell him that I swapped it out for a computer science. Like, I don't. I'm scared I'm not going to like it. And then he's going to be sad because he loves this so much. This is like true love for him. And she was like, sure, OK. I'm pretty sure she told him, like, two minutes later. Because that's what my mom does. I love her. But like, man, I think she knew that it was not going to be, you know, a wreck. And sure enough, eventually, like, I tell him I'm taking this class. And three weeks in, I'm like, oh, I really like this. Complete opposite from the molecular biology class, where I was like, oh, I really liked it. It was like, oh, with the power. I'm sure you had this feeling. It's like I only have a computer and I type into it and then it does things. I feel like a god. And then your first job, you did internships. But your first job, which followed the internship, was at Rent the Runway. It was, yeah. It's a pretty well-known company. Thanks to anyone who's an injury manager and will probably know of the book The Manager's Path by Camille Fournier. And she was at Rent the Runway. She was, yeah. And actually, that was a big part of why I wanted to go back after my internship there was because you didn't see very many women in those CTO-type positions at the time. Also, I mean, my mom very nearly became a fashion designer. And so we grew up watching Project Runway from the very beginning. That was the one thing I was allowed to stay up late to watch was Project Runway. And so it combined two things that I loved. Combined, you know, programming, computer science. I was mostly doing front-end stuff at the time. And fashion. I could just rent stuff for dirt cheap and walk around in a Monique Riviere ball gown if I wanted to. It was so cool. I think there were so many women on the team in general, I think partly because of the space that the company was in. But I also just, for folks who've had the opportunity to be in a room with Camille, she just commands this presence. And I had so much respect for that. And I mean, even if I was many levels removed from her, I was like, I want to learn from this woman. I really admire this presence that she has and this confidence. And so it was an easy sell. It was a very easy sell to go back. But then she left a couple months after I came back. But she was still instrumental in helping me land a job at Slack a year later. This is Slack in 2016. So at the time, I vividly remember because in 2016, I joined Uber. We still had HipChat, which was Atlassian's Slack offering or like Slack competitor. Other entity that overarched everything, where you could still cross teams. And that got very complicated very fast. But then the part that became really terrible was everyone across all of IBM, whether they were on the marketing team, the engineering team, whatever, were all querying the enterprise shard for all of their DMs and the big company-wide channels, all of that stuff, all at the same time. And then within that data, when you're booting up Slack, you have your channels that show up in the left sidebar. You'd think that it was just like, cool, get, you know, four user ID, get all channel membership entries. That was not the case. And I think it came from a good place in that DMs seemed like a separate thing from a regular public channel, right? Like, they seem different. And so that was a separate table. And a public channel is a public channel. And so that was a separate table. But then private channels seemed different. And I think, like, from a product perspective, you can totally see it. And from a product security perspective, you can also totally see that. You want more scrutiny when you are querying for private channels. And so that was a separate table as well. And then group DMs came about later, and it seemed like they were just sort of like a private channel, and so they were just glued into the private channel table. So when you were making that query to populate your sidebar, you were actually doing a big, big old join union. So what happened once the team had the idea or figured out that these three tables don't make sense, but now you're talking about a lot of customers, including the likes of IBM, just, you know, putting a bunch of load on it, they're using it. Yeah. How do you migrate from these three tables that are actively being used into one? All the time. Yeah. I think it came about from calculating mentions. So calculating mentions, like to get the badge icon, was also a very complex query whereby you had to join across all these tables, very latency sensitive. And we had gone through the process of doing as much as we possibly could at that time to— Ease the load. And in one of my teammates doing some work on mentions, he was like, We gotta, we gotta join these tables together. It will not—like, there's only so much we can do before we have to, like, really start digging the hole. So I was like, Sure, yeah, I will endeavor to do this very scary thing that no one else wants to do. That seems totally fine. I have so much appreciation for this in hindsight too, in that if you look at blame work on GitHub, which is the PHP framework that Cal Henderson open sourced after writing Flickr, it will explain many things about what Slack was like early on. Yes, I highly recommend. It's a good—I think it's still there. It's still there. It's a good read in terms of like, Oh, this explains so many things. So we had—everything was super procedural, for lack of a better term, in terms of how code was organized. We had sort of like API entry points, and it was like chat.login or whatever, right? And it was like, Okay, this is the API, and this is all the code for the API with some helper functions in there. There were a wild amount of functions that were like many, many, many, many, many lines long, but there was just like straight SQL queries, like embedded inside the code. Like an ORM did not exist in this world, whatever you want to call your data access layer, we did not have one. And there were actually—there was a lot of opinions about whether we should even do anything to hide the SQL queries. And yeah, now, I mean, I totally understand why they had that stance and why we did the super minimal approach that we ended up doing, because it is so easy to— Write a SQL query that will, like, shoot you in the foot over and over and over again, unless you're forced to actually understand what it does in the context that it is being called. But we had, like, 400-something call sites that called into any number of these channel tables. And those call sites, these are SQL instructions in the code. Just like, select star from Teams channels where user ID blank. Yeah, where team ID blank, user ID blank. And so you just copy-paste the pattern that exists, right? That's always the problem with migrations, is, like, you have to make it so clear that, like, this is the old pattern, stop doing this, because there are still, while it's going on, so many examples of it in the code. So I had to write some—I started off by going through, doing some kind of regex approach where I was finding all the places that we were querying for this. Thankfully, the table names were, like, pretty easy to find and identify. Got very lucky there. And then I just, one for one, took the call and just pasted it into another file. Yep, that is all I did. You created a layer that did the same thing. Yeah, just a class, basically. Yeah, I didn't optimize anything. I literally just copy-pasted, and now I was calling into a function instead of straight-up calling SQL. And you had a minimal abstraction layer, if you will. Yeah, which already was, like, a little bit spicy. But I tried to make sure that, like, the function name was, like, real darn clear as to what it was doing. And I got the spicier folks to be okay with this approach, so that felt like a win. And so I migrated everything over. Everything was feature-flagged. Basically, I had—I don't know how many feature flags at some point, but I had so many distinct ones just to make sure that I was doing it as carefully as humanly possible. Even then, you know, I still, like, definitely borked some things along the way. I accidentally—when I thought that we were basically there and ready to start flipping on to prod traffic, I'd run all the backfills because obviously, like, I was moving everything to a new table. I was doing—we were doing double writes for a very long time. For a while, yeah, writing to both places to make sure the data will be up to date. Yes. So that's where we started. We started doing double writes, making sure that that all looked sensible. I was doing a bunch of— Did you also do shadowing? Yes. Shadowing meaning that you're, you know, like you're doing whatever you're doing, and you're doing a new operation and comparing that as the same result, right? Yeah. Yeah, we did a bunch of that as well. But obviously we had to, you know, make sure all the data was there first. So we, like, did the double writing, did all the backfills, and then we were shadow reading and comparing if there were any differences, throwing away the new call, obviously, the new results, and keeping reading from the old ones. I was about to flip all of Slack to reading from the new table. Which at this point you confirmed worked, so that should have been a low-risk change. Yes. Yeah, yeah, yeah. All the data looked right. I had a bunch of data warehouse dashboards that showed me how many differences existed between the two sets of tables within Reason. And then I was about to turn it on to all of Slack. This was, like, Thursdays back when we had gather hour, which was like after-work hours, everyone would sort of hang out if they wanted, have appetizers, whatever. And I turned it on, and then I realized, oh wait, no, something's not quite right. Oh, and we didn't do a gradual rollout, because with these things you don't really do a gradual rollout. Internally, we were, like, not that big at the time. Everyone was at gather hour, you know? I was like, oh, this seems safe. We've been shadow reading. It's fine. Yeah, I turned it on. I was like, oh, there's, like, data isn't— there's some data that's, like, not quite right. And I had been so used to essentially, like, running the script that, like, would delete all of the backing data and rerun the backfill, or at least chunks of it. Because I had been seeing discrepancies, and honestly, it was so much easier to just nuke the data, re-backfill, than it was to try to, like, pinpoint and, like, scalpel fix the call sites. So I forgot that the flag was still on to everyone, and then I promptly deleted— Which flag was on? The flag to have everyone read their channel memberships from the new table. Okay. Yeah. I forgot that it was on for Slack, not everyone externally, like just us. Yeah. And I was like, Oh, there's some discrepancies. Oh, right, but I fixed those. Let me just— Yeah, yeah. So one of the things that would happen to him, right, is that before some of the re-architecture work, whereby you didn't have a bunch of distinct WebSocket connections for each team that you were logged in, Jeff was an admin at IBM. So he was in hundreds of different workspaces. At one point, I think the maximum number of workspaces you could be logged into was around like 30-something. And you're saying he was in this because IBM as an enterprise had this concept of, like, the marketing team, the finance team, the thing. Maybe they had an external customer team, and as the admin, he was in all of— He was in all of them. Yeah. And they had so many teams. Yep. And he was in all of them. And so if you were getting a message in an enterprise-level DM or a channel that is shared across all 30 of those workspaces that he was currently logged into, well, he would get that message 30 times. There was no deduplication of that whatsoever. So his client, his machine, his notifications would be both overwhelmed from maybe a little bit less in processing, that's fine, but, like, from a UI, from a being unable to see what's going on, annoyance point of view. Yeah. I mean, also his client was just dying all the time. Which was not good. Yeah. Yeah. And interesting, like this is the first time I've heard about event floods. Yeah, and they were very user-specific, but they were particularly painful for power users. This happened to power users all the time, and power users were also some of the, like, biggest champions of Slack. And so it's a double-edged sword of, like, I'm sorry you get the absolute worst experience of our product, but also you love it, I guess, and you are trying to, like, make sure that people adopt it more. It's really, really rough. Mo just made the point that the heaviest users of Slack got the worst of the event floods. Today, the heaviest users of Git happen to be your agents that often push code in parallel, which brings us to our season sponsor, Entyre. A few weeks ago, I told you that Entyre was building a Git forge. Well, it's finally here. You can now host private repos directly on Entyre in the U.S., Europe, India, and Australia. And Entyre has just re-implemented pull requests for the agentic era. The problem with PRs is how agents write it faster than how any of us can review it. So Entyre just replaced it with something called trails. A trail collects everything about the work in one place: the intent, the code changes, and the agent sessions. A trail can be spread across multiple repos, so the change to your backend and your web app gets reviewed once. With a trail, you can configure customized evals like code review, security reviews, slop reviews, and others. They look like a heads-up display on an airplane and score your work on every push. And you can see how each agent run passes or fails your user-defined gates and then decide when you merge based on these. The point is that you control when you allow your agents to be allowed to merge code, like merging only after human review or after all gates pass. Oh, and one more thing: Entyre ships continuous memory. This turns your repo into a brain so your agents can search past coding sessions to find the context behind earlier changes, basically answer the why behind the code changes. Memories get shared across the team and across the repo. If you want to see what an AI-native Git forge looks like, head to entyre.io/pragmatic and check out Trails. I'd also like to mention our presenting sponsor. TurboPuffer, who were rebuilding their search engine to make search faster and cheaper at the kind of scale we're talking about in this episode. The new system, called TurboPuffer v3, is not yet in prod, but it already passes CI. And when the rewrite started in September, it was not nearly as performant as the current production version, v2. The team has been perf-maxing query plans until they are as fast or faster than v2, doing all of this in public all the way up to the v3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to several times faster on things like cold vector search, cold full-text search, or counts. The v3 system is still slower in areas like hot full-text search, but only 1 to 2 times slower versus 100 times slower when the perf-maxing started. And the team is sharing exactly how they optimize things in their development log, with things like vectorized query execution, group count aggregation, compressed column encodings, and many more. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is TurboPuffer. Tip of started as a vector database, but it's no longer a vector DB. It has evolved into a high-performance database for search. The v3 discards the vector primary index, and all indexes will share the same columnar storage system. Every query will run on vectorized query engines, which is something that you don't usually see in search databases, but which will both result in very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey, and you can follow along at turbopuffer.com/v3. It's a blast to see how it's being built. And with this, let's get back to Mode and how Slack load tested for a customer expecting to have 500,000 users. And I'm back to load testing. So how did you simulate the load that you would be expecting to get? In this case, you knew that a big customer would be coming with 500,000 or a million users, and you didn't know if Slack could handle it, but you wanted to figure out before testing in production, before they all came in. Did you build a custom load testing tool? How did you go about it? Yeah. So we'd had a couple of, like, half-conceived load testing options to date. Mostly that tested just API throughput and making sure that we could withstand the underlying load, primarily on the database. And so if you are loading up a channel that issues a particular API request, I think it's conversations.view unless they last changed it, and that makes a particular database request, a SQL read on that channel shard now because everything is sharded differently. But it makes a particular request. We wanted to know that we could handle that RPS. The problem with that is, and any of the load testing solutions before this particular moment in time was none of it had active websocket connections. And the two particular cases that we just talked about, the websocket is the problem in some ways. I mean, we've never really had latency issues with sending messages through the websocket. Like those stateful services have—one of the co-founders was, like, the mastermind behind all of these, and, like, the amount of time and energy he sank into making them as performant as humanly possible is commendable. They were never really a bottleneck. It was either how the client handled it or the re-downstream event caused by you seeing and receiving this message that was the further bottleneck. And so we needed a system that had an active websocket connection in order to be able to adequately test the whole flow. It's not just sending, like, as though you're reading an API. Yeah. So we really—and it's not just, like, flooding a database with, like— But when you say you needed an active websocket connection to test for 500,000 load, you needed 500,000 active— We needed 500,000 active websocket connections. Yes, we did. Okay. Yeah, yeah, which is not easy to do. And so we had a solution before this customer came in, which I'm 90% sure it's public knowledge now that it's Amazon, but we had a solution that would spin up Slack as a headless Chrome instance. In, like, a bunch of Kubernetes pods and try to run a tiny little JavaScript script and, like, click buttons, like pretend to be a Slack user. Which, like, was sort of fine. But I don't know if you've tried to wrangle more than, like, 50,000 headless Chrome instances in Kubernetes. I don't think many of us have. I don't think many people listening or viewing have. It is not a good time. I mean, the problem with the headless Chrome instance, right, is, like, just running Slack in, like, 200 Chrome tabs and whatever on your laptop is, like, kind of a time. And now you're running, like, you know, LLMs on top of it. But, like, it's an experience in really pushing your laptop to the limit. And now you're running, like, per pod, like attempting to run, like, 10,000 headless Chrome instances of Slack, which is, like, a very memory-CPU, mostly memory-intensive application. The amount— of space. The space is running out, and we either—well, we both have to figure out how to move to cloud, and we also have to figure out how to do less, because we're doing a bunch of dumb stuff, as everyone does, that we, like, shouldn't be doing, and we can make the system a lot more efficient with what we have to give us more runway. I was like, Ooh, this sounds like a very interesting problem. Would love to work on this problem. But I was also seven months pregnant at the time. And so what that ended up being was I showed up, I was seven months pregnant, which is actually kind of, if you can swing it, a really great time way to join a company, because you sit there and you're like, Cool, I will just drop a bunch of opinions. Like, you know, nothing very strongly held, but I will just drop a bunch of opinions on people. I will own nothing for the next two months, and then I'm going to go have a baby, and then I'll come back. See what happened. See what happened. And in my first couple weeks, I was sort of floating around some spaces and I, having done a bunch of database migrations at Slack and some of the stuff that we didn't talk about, but was moving from this, like, team-sharded, tenant-based model to a more dynamically sharded model with Vitess. Where it turns out, no, a team was too big. So you mentioned Vitess. What is Vitess? Vitess is essentially a wrapper on top of MySQL that lets you shard more dynamically. And so you have a— So instead of just having, like, MySQL run on one database, and when you outgrow it, good luck, you manually move the tables or copy them or whatever. It's a nightmare. It allows you, as it grows and as it outgrows, it will, I don't know, predict, or it will now have a way to more seamlessly move over to multiple machines. And you can reshard, redistribute your data a lot more seamlessly than you would be able to otherwise. And you can choose different sharding keys more easily as well. And so, like, before at Slack, everything was sharded by team. There's still data that is sharded by team, where team is the logical unit that makes the most sense for fetching that data. But it actually turns out that the smallest, most logical unit was channel. And so we ended up resharding a bunch of our data to be fetchable by channel ID rather than by team ID. And then that allows for a much better horizontal sharding structure as well. And the idea being that there is no channel that will ever be big enough. Maybe one day it will be. The bots keep posting in the channels, maybe, that will have enough data associated with it that it can never run on a single ultra-large database, right? It's a much better unit from a scaling perspective. So Vitess has had some of its nascency embedded in GitHub, actually. And it was really interesting because when I was interviewing at GitHub, I spoke with a bunch of the principal engineers that I would later end up working with, and they were sort of trying to figure out, okay, we have Vitess, but I don't think we are actually using it to its potential, to where we should be using it. And so they were asking me a bunch of questions about, like, are Vitess set up and what we were doing, caveating this with I did not ever operate Vitess or deal with the internals. I was more of a, I'm a product platform person. And so I just, like, find that, like, we should not be sharding this thing by this thing anymore. Let's migrate it. And so that was my perspective on it, which was also mostly their perspective on it as well. And I came in and I started poking at the Vitess setup, and I was like, What are we doing? What are we doing here? I was used to thousands of shards per sharding key, so say four channels. We have a bajillion, because there's a bajillion channels, and you want to sort of spread load evenly throughout those. And when I arrived, I was looking at the Vitess setup, and there was, I mean, I don't know if this is still the case, but there were 16 shards for all of pull requests and issues across all of GitHub. Across all of GitHub. Only 16, which is not very many given the amount of data that we know is associated with that whole space. And my first question was like, Wait, why do we only have 16 shards? Oh my gosh, this makes me deeply, deeply nervous. And part of the answer was, we can't have more because we already have so many scatter queries. So those are queries that are coming in— Yeah, what are scatter queries? So those are queries that are coming in where you do not have the primary key in them. So say you are querying for a pull request or an issue, and you don't have the pull request ID, which is how this data is being distributed. Well, you have to find it. So you have to go through, you have to find and match the criteria. So say I'm querying for all pull requests authored by me. Yeah. I don't know that that's an actual scatter, but this could be in this scenario. Let's just assume. And so because I don't have a pull request ID, well, I'm going to hit every single one of those 16 shards. Wow, this will be bad in so many levels. Uh-huh. Because, well, so many people are doing it also. And I understand then why you don't want to have many shards, because if you have more shards, you would need to hit all of them. We're talking like, it's almost like a linear scan. Yeah. Yeah. Oh no, it's quite, quite, quite inefficient. And again, I don't know that this is an actual example of one of the— this is completely hypothetical, but there were still very many significant scatter queries that were hitting almost every single shard before it knew that it could complete the query and return the result. Which, yes, I understand why that was a constraint then. You don't want to add more shards. But then again, you also cannot make the shards much bigger, and they're under very heavy load. And so I was ready to, like, apply the playbook that I had been using for the last several years of, like, well, let's just burn out— burn down the scatters. Let's just get rid of the scatter queries. Don't have scatter queries. Just don't have scatter queries. That can't be that hard. Well, it turns out that can be very hard when you're dealing with a Rails monolith. Yep. And you've got ActiveRecord. And I always sort of thought it was funny, the number— there were a lot of ex-Twitter folks at Slack early on, and everyone was like, ‘Ha ha, ActiveRecord.’ And I was like, ‘What are you—’ Like, before they did the whole, like, Scala-Java thing, right? And I was like, ‘Okay, I guess, like, we're all, you know, joking about ActiveRecord.’ I did not understand until I, like, came face to face with it, and I was like, ‘Oh, I really don't like this. This is really terrible.’ So ActiveRecord is Rails's abstraction layer for the database, right? So you define things in, I guess, code, and it kind of maps nicely to database, except it might be harder to fine-tune your queries, et cetera. Yeah. I mean, it does a very good job at hiding the underlying data model. From you, which again, pros and cons. I came from a world where, like, SQL was, you know, religiously written out by hand for performance reasons. And then I was being thrown into this world where at one point we were trying to figure out, we were doing a bunch of work around domain extraction. And so how do we rip out everything related to, say, pull requests from the Rails monolith and extract that into a separate Go subsystem, take everything that deals with pull requests, just, like, take, make the requests go through, say, the monolith still, but then just, like, immediately send them over to the other system. Really standard sort of migration procedure. We started doing that. We started with Stars, and I think that project was eventually abandoned. But I was working, I was pairing with someone who had, like, a ton of Rails experience, and I was like, okay, you gotta, like— because I, as someone who was so used to reading just code that was very sort of straightforward and didn't have so many levels of abstraction, it's a little bit daunting when you're going into a world where now there's abstractions upon abstractions upon abstractions. And I was working with her to try to rip out more of the Stars-related queries inside of the Rails monolith. We'd found sort of the root caller. Started sampling for cost reasons, but not in one place, which is, again, really easy to have an LLM do because it's like, oh, like, find me a place where we can send fewer traces to save money. And it's like, cool, I'll just, like, sample it right before I send it to Datadog or whatever. No, no, no, don't do that. Please don't do that. Always do it in the same place. The problem with duplication is just, like, so much worse when you're dealing with this. And so one of the first things that I ended up having to do was, like, honestly manually instrument a lot of these flows and actually read through the code myself and understand, okay, this chunk of processing is synchronous. This chunk of processing is asynchronous, and put all of those together. And one of the ways that I started doing that was I had it generate a, like, big sequence diagram of everything that goes into time to first token. So, like, from the time you hit submit to when you receive that first visible token. And it was huge. And I went to Staples and I had them print it out on, like, big blueprint paper because I'm still a really analog person. And then from that, I went through and I was reading the code and I was annotating on my big piece of paper, like, okay, this is correct or this is incorrect. This piece we could probably do something about. This other piece, no, actually definitely needs to happen before this other piece. One of the other things that I was finding that it was doing really painfully wrong is that it was finding sort of the classic things that could— What was doing? What the LLM was doing wrong? Oh, yeah. So what it was doing wrong was finding— well, wrong is kind of the wrong word. Let's be opinionated. I like it. Naively, naively what it was doing was, say you have a bunch of things. You're finding a bunch of tasks that don't depend on each other, right? But they're latency sensitive. What's the classic solution? You parallelize them. However, again, product hat, right? What if basically no one is ever doing all four of those things? And why are you doing all of them? At the same time, like you're wasting resources, latency, you're increasing load for no reason. And so I remember, I think it was, I think it was Opus that was, like, very excited that it, like, found this opportunity to paralyze a bunch of these tasks. And I was like, oh, like on first glance, sure, that seems like a good thing to do, except that the problem, even if you paralyze them, the longest one is always going to be the bottleneck, regardless. And the longest one was always so much longer than the other three, and the other three tasks were being run—such a small, 1% of users were doing any of the other three. And so it ended up being a complete waste of time. Like, this was not the optimization that it was, like, selling to me as being, like, the panacea to improving time to first token. And a lot of the things that were actually quite helpful were really silly things like—I mean, silly—like increasing the TTL on what we stored in Redis where we could within reason. How do you use LLMs? Because so far my sense is, talking through your career, you look at a problem, you understand it, you figure out how to do it, you fail sometimes or make mistakes, you learn from it, you learn from others, including right now in the last few years where AI is increasingly everywhere. But as you're making systems more reliable, more performant, it sounds like you're still doing so much thinking, so much. I sense that you're using the LLM on the side to generate some ideas to say, no, this is wrong. No, that was a good idea. Okay, let me use it. But unlike a lot of software engineers I talk to, even on this podcast, most of them are saying like, oh, so much of my work that I used to do, the LLM does it. I now can do more things. I can focus on higher. I'm not getting that sense from you. Yeah, I mean, I'm spending all day, like, chatting with an LLM in some way, shape, or form. I think, like, I still do a lot of the— Normal engineering stuff, which is like, I'm on call. Sometimes I just have to fix a silly bug. But I do tend to, I do a lot of sort of preliminary exploration, but then verification manually. And I sense that a lot. Like, it sounds like even when you get something from an LLM or whatever, you will verify. You will make sure that you understand that this is right. It sounds like you don't have a judgment if it's going to be right or wrong, but you need to verify it, right? Yeah, yeah. Because I think a lot of the problems that I work on are more of the, like, product platform-y problems. And the whole point is to solve a problem holistically and to create that foundation that you can build on top of, right? And if you only build the foundation to cover, like, 80% of the house, like, what's the point? Because that other 20% that was not built on the foundation is going to be really shaky and impact the rest of the house that you did build on the foundation. And if you already did that over here, well, like, nothing's really stopping you from, say, like, tacking on another 10% off past the foundation, and then that spiraling, right? And so I think especially, especially now that, like, I think AI is so good at just duplicating things in places because that seems fine and it doesn't have a sense of it's not great at going, you know, but why? But why? But, like, 20 levels removed, it leans more towards, I will just tack on, I'll just build an extra bathroom past the foundation. That's fine, right? Like, you asked me to build an extra bathroom, that's where I'm doing it, rather than questioning the underlying fundamental assumptions, which are, should one build a bathroom not on top of a foundation? And those are the questions that I'm asking myself all the time. And I think part of it is because I'm coming from a product platform perspective. I have a sense that the product platform work, the platform work reliably performs, it will only be more important because, again, we'll have systems that have so much load, so much test. What advice would you have for a software engineer who may or may not have much experience, but who is excited about the idea of performance and reliability and would like to get closer to eventually work on platform teams? What are paths that you've seen people make it onto platform teams that you worked on? How did they transition? I'm such a strong advocate for more product folks, more interchange, actually, between product folks and product platform teams. I've seen it so many times where product platform teams are spun up out of product engineers who care a lot about the fundamentals, right? They care about reliability. They care about performance already. They're trying to improve that and inject that into the way that they're building product all the time. And then I think the best product platform folks that I've worked with get to a point where they've seen a pattern enough times in the things that they are building as product engineers, and they're like, I want to dig in and solve this problem across the board. Like this specific problem, I understand really deeply from having implemented it two, three, four times. I want to go solve it once and for all and be done and, like, have this thing that other people can also use, right? They get annoyed with the toil. And then they go and maybe embed with a product platform team, or they, like, spin one up with some other folks, and then they do that. And then a lot of times those folks end up staying on product platform teams and continuing to really like the product platform work. But then they get further and further detached from what product development actually feels like. And I think that's where you see sort of the chasm that ends up being created between product platform teams and product teams is just there's not enough cross-pollination between the teams. And so one of the ways that I've seen that chasm merge back together is having, like, rotation cycles between those teams, sort of always having someone on the product platform team, having a core set of folks who are, like, maintaining the main systems and platforms that they've built, but sort of always having someone on rotation from a product team on the product platform team, bringing, like, Oh, well, I don't know what—