GitHub’s COO Explains Why AI Hasn’t Replaced Developers

Hi, I'm Mike Taylor. I'm the head of tech consulting at Every, and I sat down with Kyle Daigle, the COO of GitHub, and talked to him about what is happening on the front lines of coding agents. We have 17 million pull requests coming in every month to GitHub now. It's growing exponentially, and that puts them at the forefront of what's happening in this new economy. We talked about how that affects their users, as well as how this affects open source maintainers, and we covered a topic which is dear to everyone's hearts. How do I stop my $200 a month coding agent subscription from ballooning into a $2,000 a month usage limit? In this interview, we did something a little bit different, which is I told Kyle I had made an AI clone of him to practice the interview, and he revealed something surprising in return. Here's the conversation. Every is the only subscription you need to stay at the edge of AI. If you care about being on top of the latest models and using the latest tools, you have to subscribe to Every to separate out the signal from the noise. Go to every.to slash subscribe today. Hey, Kyle, thanks for spending some time with me at the conference. Yeah, of course. Yeah, it was good to meet you as well on the day before. And I feel like we already kind of covered a few of these questions, but I think it would be good to, you know, for the wider audience so they can understand what's going on here. Absolutely. Yeah. So first thing I think is, we were talking about, it's really interesting is that the demographics of the customer are changing, right? Like a lot of people who previously maybe never used GitHub or never used developer products before are now using them. So how has that changed the way that you decide the product roadmap? Yeah, yeah. I mean, you know, I think for GitHub in particular, we've always really had this really expansive view of what a developer is. I started as a developer before I would have ever called myself a dev, you know, where I was just like writing code, but it was just for me. And I went personally, like a completely different career path. I didn't go to school for computer science. I was going to art school. I wrote code to pay for art school, which was a very silly decision as an adult now, I guess. But like, at then, you know, that sort of journey of just like, I can create tools with the team and deliver them to people who can have that same experience of like, I just want to build an app that's for me or for my family. Maybe as a startup, maybe as a business. We very much have like serious developer tools, you know, like all the largest businesses are using GitHub. But when I look at something like the GitHub Copilot app, I see just as many, you know, developers that are using AI every day, running multiple projects, like all kinds of agent sessions at the same time. And I see, you know, our legal team at GitHub using the GitHub Copilot app or the finance team. Or I was meeting with a customer today and they were saying the same thing. A lot of the folks that, you know, the industry would call knowledge workers or just, you know, non-by-trade developers are using these tools to build little apps or assets for them. And so while our focus is very much on developers, I think we want to make it easier for people to choose to try to, you know, write some code and make sure there's always an on-ramp into, you know, writing some software now with, you know, things like the GitHub Copilot app. Yeah. And then how do you deal with the, or how do you help developers deal with the burden of all of that extra, you know, there's a flood of PRs now, like open source maintainers I talked to are like drowning. Yeah. How do you, how do you, like what needs to happen to help them? Yeah. I mean, I think for all developers, we're, you know, building tools like the copilot code review. It's now agentic. So it finds a lot more novel vulnerabilities and you can just like comment and the, you know, the agent will take that on and go implement the change if you want to. So I think that code review step is in some ways like overlooked as a really great way to get PRs to a place that are much more easily reviewed. I think that uh the agentic merge in the app is another place where we see a lot of times internally, uh, and in the community, you know, you make comments on something. You might have a code review and you might go through and get it almost all the way there, but then there's all those like manual steps just to finish processing the PR. And instead I can go in and set exactly what I want to allow, you know, GitHub Copilot to do and say, okay, now go merge this PR and wait for CI and wait for policies and all of that. I think that's a big part. On the open source side, it's a unique set of needs because you don't control who's, you know, sending everything in uh or you haven't really historically. And that's been really where we've been focusing is just giving maintainers more tools to decide, well, do you want to accept all of these PRs? Like who do you want to accept them from? You know, how much work do you need to do to to kind of prove that you're going to contribute something that is going to be meaningful to this project? And that's something that we want to provide tools to open source maintainers, but really leave them in control. Every community is choosing a slightly different way to approach the problem. And for GitHub, we've always wanted to leave that in their hands. Like give them tools and enable them. But if a standard comes out of that or most are using a certain practice, we'll lock that in. But we don't really ever want to be the first to create a standard or an approach. Like I think, you know, Mitchell Hashimoto shared like the vouch system that they use. And I was getting questions like, well, why aren't you roll this out to everybody? But there's just as many communities that don't want to use that system because they have their own ideas of how it should work. And so for now, we're focusing on the building blocks of controls for maintainers. And then as we all are kind of learning together and as maintainers send feedback in, we'll, you know, we'll cement an entire system if one emerges. Yeah. And I feel like you have a front row seat to this like new agent economy where, you know, I think you said publicly on Twitter that you've had more pull requests submitted and like per month than you did all last year. Like how are those stats exploding? Yeah. I mean, I mean, we're seeing way more activity on GitHub. You know, we've always been talking about our users, you know, for many, many years and that growth. But this year we're seeing obviously the growth of developers having agents building with them. And so last year in October at GitHub Universe, we shared, you know, there's a billion commits on GitHub for the full year. We're on track to be 14 billion if the growth is linear this year, which it will not be, you know. In March, you know, there were 17 million pull requests that were created by agents. That's just the agent pull request. And so there's so much more code being created. And I think at times everyone goes like, oh, this is all just like slop. This is all just code that's getting pushed up, but and no one cares. That's not like really true. We're all just actually getting to the point where we're no longer in the super early adoption. We're definitely not at the peak, but we're climbing that hill, you know, to see what can we build when it's not just Kyle building, but it's Kyle and one, two to N, you know, agents that are using my skills, using my resources, using my contacts and so on and so forth. And so like we're, you know, investing heavily in preparing for the next wave of growth because it doesn't seem to be kind of like growing and plateauing. It's just going to continue to grow because no matter where you're, you know, building or what tools you're using to build all of that code ends up on GitHub, you know, or that's where you're sharing it with the world. That's where you're collaborating in a PR. And so we need to be able to support everyone's, you know, sort of agent moment and not just, you know, GitHub Copilot. Yeah. Yeah. And how does the business model change? Because I think freemium makes sense, you know, in a human centered world where we go to bed, but the agents are still working while we're asleep now. So does that change like to usage based? Like, is that, you see that kind of where things are going? Yeah. I mean, like, I don't think we know like yet ultimately. Like, I think, you know, we very much right now, like Kyle is going to have a license or Kyle is using GitHub.com for free. And we've always had, you know, API rate limits, you know, and like things like that. And that's usually where folks are seeing the, you know, the agent back pressure, I think. I think the goal is that if you want to be able to do way more, if you want to be able to like Peter Steinberger says, you know, 150 agents are doing everything all at once, you know, that's great. We want to be able to enable that to you. But at the same time, I Of just, you know, experimentation, everyone is building and using these tools, while obviously we're putting most of our energy into our own tools. It's such a blind spot that I think, it's happened to GitHub in the past, where like, when you're doing something and you're doing it well, you really laser focus. And that's what every piece of startup energy says, right? It's like, look down and just keep moving and keep moving fast. And I think that's myopic. You know, I think that while I can't spend every day using every tool, when something comes out, I want to know why this is really great. Why are people having a great experience with this? Not, not only so I can understand, but so I can figure out, okay, well, for our goals, for our goal of developer choice, I don't need this. Like I wanna focus over here, but I wanna know why a dev would pick these tools. And the same thing goes, you know, the same thing goes for our teams. Yeah. And how do you, how do you filter? Because obviously a lot of these ideas are relatively short lived. Enterprise, product development cycles are longer lived. How do you decide? Yeah, I think that like, right now we're in a moment where we're really looking at the like, the short term in capturing the ability to have a multitude of, you know, agent sessions. This idea of just, you know, because that seems quite clear, you know, everyone's doing it, how can we cement it? But it seems clear on the longer term path, models are gonna continue to get better. The prices of tokens, like token economics is gonna be a bigger, bigger factor in like what's models everyone is using. And I do strongly believe that we're not very far off from having serious, the serious ability to use, you know, something above a small language model on a local device, you know, to be able to do some of our work. And so if I assume that I have all this optionality when it comes to tokens, you know, effectively, the thing that I think seems to be true from the beginning to open claw to now is this idea of like personalization or mine or context or fine tuning with context or like, you know, memory. All of these ideas seem to be a truth that's been there since, you know, ChatGPT came out or GitHub Copilot came out. And there's experiments, but not a long-term like vision, I think, for this like across the industry. So I think it's a good example of like where I need to get you to use agents incredibly well, a lot of them, because like if you're into using agents, you're not just gonna be staring at a single agent working, but that's not gonna give you a long-term great experience. Using an agent that you feel like is completing a thought for you will give you that great experience, especially if you did not have to personally codify that thought to your agent. Always remember that I insert thing, you know. That's a lot of work, yeah. 100%. Like, and it, it should be able to intuit that or potentially again, like, you know, post-train or fine tune or frontier tune like a model on that deeply understands me and how I'm using the work. That is the, that is kind of how we're looking at it. It's like, sometimes it's short term and sometimes we got to take a bunch of bites of the apple or a bunch of attempts at the long term to get to something really tangible to help us move forward. Cool. And I heard the term hill climbing like a hundred times yesterday. And I'm a big proponent of that because, you know, experimented with DSPI, Auto Research, a few others. Can you talk a little bit about how that's become a big focus? Yeah, I mean, you know, I think, you know, Satya and Mustafa talk about it a fair bit and Jacob too, uh, leading the co-pilot group. Uh, the biggest thing that we've kind of learned is we need to use the, you know, the use of the tools as a core way to improve the underlying use of the models, our own models, et cetera. In just the evals that are necessary to ensure that we're actually improving from things like, you know, using the thumbs up, thumbs down data that comes in, to using like whether you're accepting it and how much you're accepting, you know, all of that data is enormous to create that magical type of experience. That's not just for you, but for everyone. And so every week we're talking about the hill climbing results, you know, we're looking at the data, we're looking at the improvement, we're looking at both the hard measures and the soft measures because sometimes the hard measures and the evals and rubrics will show that we've made an improvement, but like user sentiment will crash, you know, uh, even with the same latency and performance. Yeah, to overfitting. 100%. And so like being able to, being able to really, um, do that loop incredibly quickly. And then I think the main goal, uh, is, uh, giving everyone one of these hill climbing machines and not have you have to do it the kind of hard way that we've all been doing it, but particularly if you're, you know, in an enterprise and you are using M365, we know so much about that data or we could know so much about that data because of all the assets, all the documents, the chats. And so, uh, being able to, you know, turn on something like frontier tuning and using on the, you know, using a MAI thinking one, uh, as the base model, it shows real results without having to do all that extra work. And it's been interesting because when I first heard about this, I'll be honest, like I was like, this is like a magic parlor trick, you know, that is not going to be real. It can't really work here. Yeah. You know, and I think the reality is that's sometimes, um, uh, where the alpha is, is like where it feels like this is too simple to work, you know, Um, uh, you know, we all have all this data and what are we going to do with it? And we have to do all this effort to make it work. But I think so much has come down the pipe to allow us to just use the data and improve, look at the workflow and improve, and just keep doing the hill climbing. That's why I think we say it so much, is that it's not these moon shots or like a hill climb. It is just climb, climb, improve, new eval, improve, new data, improve, and just keep going to get to the point where, you know, we're able to launch these models, seven models for ourselves and then, you know, allow customers to use the same or similar, you know, tooling to do it. Is that the answer to um, stopping the $200 subscription becoming a $2,000 subscription? I mean, like, I think that, uh, the, you know, $200 subscription to $2,000 is really going to be, uh, not only, you know, making these models, uh, or, you know, uh, uh, frontier tuning these models so they way, they know you better, but I also think it's really, really going to be about how can we, particularly for developers, you know, help you automatically choose the models and potentially either have like a model in that step. Like the model router and GitHub co-pilot. 100%. Exactly. Like the, you know, uh, auto model router with task intent and GitHub. Uh, Microsoft Foundry has a model router as well that can do this, uh, you sort of at an API level. The more and more that we can help you, um, uh, you know, tell us a bit of like where your bars are, like, this is an incredibly hard problem and I'm willing to go all the way to the top, or I don't, I just kind of want to sit, you know, here, uh, and let us help choose the models because there's a lot of times where a lot of the reasons why tokens are expensive is because we're all going and choosing our model of the day or week or hour, you know, and those models are incredibly expensive, but my train of thought is slipping in and out of a hard problem to like a simple problem. I personally, I feel like I'll, you know, I'll get an agent to do like an enormous amount of work and then there's always that last step that is like a smallish thing, you know, like, oh, I don't actually like change all the naming of this to this. Yeah. And that's Like a find and replace. You know what I mean? But I'm, but am I gonna actually go and like, oh, I wanna save tokens right now, so I'm gonna go off of 4.8 or 5.5 and down to haiku or something, you know? Probably not. Uh, but the tools could. And I think that will really help us, uh, uh, particularly in the enterprise, but even for, you know, individual developers and, you know, folks that are building like automations and using their co-pilot, uh, you know, SDK, uh, to power that. It'll, it'll help them too. I did something a little bit weird. I hope you don't find it creepy, but I, I made a AI version of you to practice this interview. No way. Yeah. And it's actually been pretty spot on so far and hopefully you think the questions have been good. That's good. They've been great. Now I wanna see what AI Kyle said. And um, yeah, it's just in the terminal. I didn't