What AI Changes

Hi folks, this is All Things Product with Petra Wille and Teresa Torres. And we're so happy you're here. Petra, right before we started recording, I was on LinkedIn and I wrote this screed about what has not changed because I'm really annoyed by all the people that say everything is changing. A lot is changing. Let's just start there. We'll acknowledge that. But I thought it would be fun for us to dive into this topic about what AI changes and what it doesn't change, because it seems to be the hot topic of the day, and I think we might have some surprising hot takes. Should I just share some of the hot takes that I have been discussing recently? Yeah. Oh, from the top of my head: random coaching conversations, hallway conversations, water cooler conversations. Where do you have hallway and water cooler conversations? Oh, still here in Hamburg at meetups and something like that. So in Hamburg, the product community still tends to hang out with each other. Nice. Yeah, so that's really nice. There are more meetups than ever, by the way, so new ones are popping up even, and I appreciate smaller ones maybe, but still nice and cozy, so I couldn't complain. Yeah, so one thing that I heard, that I was sitting with for a while, was this: Hey, our engineering department is now so much more efficient. Cost of developing something went down so much, but, you know, we still have this massive bottleneck called testing. So now the stuff is piling up at the end of our task boards, and we still struggle so much with testing what we get, which led me to this kind of—and it was just a conversation that I heard. It was not a conversation that I participated in—but I was really like, okay, but is... Was that, tying back to our last episode, by the way, was that an engineering organization that understood what good looks like in the past already? Because how can it be that now everybody is super happily developing features with the help of AI, or even agents do it without the help of the engineer, however they do it, but now everything piles up and is untested? Was this about testing or was it about code review? Both. So they talked about both. Because I have heard a lot of teams, like, reviewing all the code that AI creates. Yeah, like you go through the cycle of, like, engineers are no longer writing the code, but at a lot of organizations, one engineer writes the code before it's released, another engineer reviews the code. And so now with agents writing the code, there's like a pileup of code review until you get agents to do the code. Yeah, it was one aspect, but the other aspect was really like, I was actually wondering if they did test-driven development in the first place, or if there was never a case in this organization and they always sort of retrofitted all their testing after developing the actual features or something like that. So that was an interesting thing. And that's one thing for where AI is not changing any of the problems of that tech organization, obviously. It's just emphasizing the issues that they may have already had. Yeah, this one is really surprising to me because I actually think one of the things that agents are really good at is writing automated testing. It still requires, like, human oversight to make sure the right tests get written. But I'm wondering, like, is this an organization that didn't already have automated testing? To me it sounded like exactly that. Like, do they have QA people that are literally testing every feature? Yes, they were talking about QA people, and that is— It's humbling to some extent to listen to those conversations. Actually, I had the conversation with Anna afterwards because we're currently starting the process of curating next year's product at heart. And then again, I said to Anna, Anna, it's always so interesting to hear these conversations because us belonging so much to this product bubble and this product thought leader bubble, sometimes you forget that the companies are still out there. Yeah, maybe manual testing people are manually testing stuff before it then gets deployed in a weekly release, maybe even, or something like that. Maybe weekly. Yeah. Super interesting. Yeah, I did talk to the team recently. They're actually two of the best product managers that I've worked with, and they started to share, like, some delivery concerns. And a lot of it was rooted around the fact that, like, their team was reluctant to use AI to write code. They had literally no automated testing. What they shared with me was they would do a sprint, they would release new features, but as part of those feature releases, a whole bunch of old stuff would break. Yeah. And so it's like they were reluctant to give their engineers new stuff because it would just break a whole bunch of old stuff. And it would break, like, core functionality in their app. And I remember I asked them, I was like, Wait, do you guys have any automated testing? And they were like, No, we don't do any of that. Interesting. Yeah. So I don't think this is a function of what— So even really good people end up in organizations that don't have— I don't think this is a function of what AI changes. I think maybe AI is exposing what was already broken. Yeah. I would totally subscribe to that. For example, that's another hot take that I heard. And it actually made me gasp for a second. It was somebody saying on the stage that they think that we now finally can ditch all the Scrum that we ever used or something like that. So all the frameworks that we ever used to organize work basically are no longer important. And at first I thought like, oh, this is a bold statement for somebody on a stage in front of you don't know who's in the audience, right? And maybe it's such an organization that is so close to finally have a good rhythm and good rituals for what they actually do, no matter if it's with the help of Scrum or VU, or I couldn't care less. But there always will be this rhythm and rituals that you need. We always need to plan the work. And even in times of AI, we have to. And of course, again, AI can help us to plan the work. But still, somebody has to be aware of planning the work being a thing and reviewing the work being a thing and demoing the work and its impact being a thing and measuring the success of the work forever being a thing. And last but not least, and we just recorded an episode on that as well, is kind of look at your work and your efficiency and then think about how can we improve our work working relationships for the sake of being more efficient in the next. And for example, they no longer talk about iterations. They gave it even a different name. And I was like, oh my God, now we, they still work in one week chunks, but because they just declared that Scrum is no longer a thing, they of course don't call it iteration anymore. They found a new word for that. And this is, for example, something that I think AI isn't changing as well. So this planning the work, reviewing the work, demoing the work, making your work more efficient, no matter if you use a Scrum framework or any other framework for organizing, but somebody has to take care of organizing the work with the help of a large language model maybe, go for it. But still, that will remain unchanged. So they argued Scrum is dead. Let's put aside the Scrum part. Really what they argued was that, like, planning and aligning and setting, like, a line in the sand for what we're going to accomplish and then measuring whether we did that, maybe doing retrospectives, like some of these rituals that we associate with Scrum. Did they argue why? Yeah, to some extent. So they double-clicked a bit on estimation meetings and backlog refinement meetings, and that estimating things was maybe always a bit of an overhead and never a good idea. And so now in times of AI, that's finally over. And with that part, I do agree, because I think all of the cross-functional teams that have a certain level of AI efficiency or literacy work more on the product itself or the software itself these days, and not so much on understanding it and putting it in a specification. But even then, you still will need to be asked to put things on time horizons at least. Yeah, so I don't know. It was wild, but that was what they were arguing with. So finally these rituals are all, they always have been overhead, and now they're no longer necessary because there would be way too much overhead in times of AI and faster development cycles. I mean, I argued 15 years ago that estimates are a waste of time, so I agree with that part. I think that's the part that I agree with as well. But the rest— But I don't. There's a difference between, like, estimating story by story, task by task, which frankly is just— Pure fiction from the beginning. Yes. And estimation for alignment, by the way, is always something that I found helpful. And saying, like, this week we're working on this chunk of work and our goal is to complete it. Like, that is always needed, regardless. Like, I don't think AI has changed that. Like, are we just saying, go do some work and get as far as you get, and we'll talk about what you did later? Like, yeah, I don't—I'm not sure I get it. Like, how do you even work with other people if you're not having a conversation about who's doing what by when? Yes, exactly. So that is why I always, when people are saying, like, Scrum is dead or SAFE is dead, and would I hope SAFE to be dead forever? Yes. But still, you need a process. Even as a product person, I think we need a process to organize the work, and we need to agree on that to some extent. And it can be a very small nucleus of an agreement and a process for one team, and it's only this one team that works by that. And yeah, maybe all the teams are shrinking. I get the idea. Maybe there are fewer engineers in a team. Maybe the teams are smaller. Then it's super, because then only six people have to align on how they work, and maybe they don't need three-hour-long retrospectives every other week. Cool. But I don't think Scrum, SAFE, or any process-defining framework is dead. What I think is funny about this one is that for people that are using agents to build, they're like, we're seeing this cycle where, like, early on when people were building with agents, people started talking about planning documents and first plan with the agent, and then let the agent—once the plan is finalized, let the agent build, and you'll get better quality. I even wrote a blog post about this, right? Like, then we get to the point where you have multiple agents building, and so people talk about, like, there's this... If you hear about people talking about loops and graphs, like these are all ways to communicate requirements to an agent, so the agent knows when it's done. And like with a graph, it's a way to let multiple agents coordinate their work so the agents don't step on each other's toes. Which, by the way, all of this is the same as the rituals and ceremonies we do with our teams to, like, build software. We're just now doing it for agents. So for someone to say, like, this isn't needed or it's dead, it's actually more needed because while the number of people involved might be going down, the number of agents involved is going up and the throughput of work is going up, which means, like, for anybody to have any idea what is happening, you need some sort of way of saying who did what by when. Yep. And yeah, yeah, it's actually even supercharged. You're right. I haven't thought about it from that angle. Yeah. So stating all these work agreements are dead, how we organize work is dead, that's here to stay, I think, as well. Yeah. Okay. So mine are all around discovery habits, right? Like I just hear people say, you know, we don't need to do any discovery anymore because delivery is free. We can just launch our product and hope it works. Or I hear, like, just the other day I read an article about how story-based interviewing doesn't work anymore. We can't ask people about their past behavior because AI is brand new and no one's used it before. This is, like, such a, like, when I read these things— Even my mom has very decent AI experience. Yeah. Like, when I read these things, what it tells me is that people, like, misunderstood the habit in the first place. So yes, you can build something and just put it in the world. But when it falls short of your expectations, you're not going to know why, because you didn't do the work to enumerate the assumptions it depended upon. You didn't systematically test those assumptions. So literally all you learn is it didn't work. You don't learn how to fix it. You don't learn how to iterate on it. You don't learn what to do differently next time. You're just throwing spaghetti at the wall. Yeah. And same with interviewing. Like, story-based interviewing was never, Tell me about my product. It's not about the product. It's not about the thing that you're building. It's about the customer. Tell me about the last time you experienced this problem, this need. And it doesn't matter that the solution is brand new. Your customer is not brand new. They're still a human. They still have past lived experiences. And real life experiences. To learn about those things. So, and then the worst one, how you would— because this, you know, this customer delight moments, these aha moments, these features that you have never thought about, but then you see them struggling or then you hear them share a story, and then the entire team has this, Huh, this is actually something that we could solve for. I don't know where people want to get those ones from if they're not talking to humans. They just want to ask the LLM. Yeah, so convenient. Yeah. I mean, here's what I'm glad about. Like, fine, go ask your LLM. Use that to build your product. Good luck. We're going through a period of time right now, I'm just thinking about the fact that, like, Bending Spoons just bought Airtable. They just announced they're buying Miro. Yeah. Like, I, a week or two ago from the time of our recording, wrote a post on LinkedIn about how I don't think SaaS is dying because of AI. If your SaaS company is dying, it's because you're making bad decisions for your customers, and that has nothing to do with AI, right? And so I wrote about Airtable in particular. I didn't name Airtable on LinkedIn, but here I am. I am now naming them on our podcast. I have been a very happy Airtable customer for years. They are the heart of my knowledge management system in my business. About two years ago, the performance cratered. I wait—I'm not exaggerating—45 seconds to wait for a view to load in my browser. I emailed them. They said it's because I have large tables, but this is not true. If I have a table with 200 things, 20 things, it still takes 45 seconds to load. They, like, lost focus. They got distracted. They started building a lovable, a Replit, an AI agent that builds things for you, and they stopped investing in their core product, and their core product deteriorated very quickly. This is not AI killing SaaS. This is people at SaaS companies making stupid decisions. And now Benning Spoons will be killing their product. Well, now Benning Spoons, I fully expect them to quadruple or 10x my price, so I am furiously working to get off of Airtable. Yay, thank you. Yeah, you know, I just did that with Harvest, so welcome to the club. Yeah, it's so easy to blame whatever is new. AI didn't kill Airtable. The Airtable team getting distracted from their core value killed Airtable. I can make another argument: raising way too much venture capital killed Airtable. Yes, yeah, yeah, because, like, that happened to so many startups in the first place. They were a very successful growing company. The problem is they weren't growing at a rate that justified the value they raised money at. Yep. This is a problem they created for themselves. And I can make the same argument for Miro. Miro brings in $600 million a year in revenue. That's a very successful business. And if you talk to the customers, lots of them are very happy. Yeah. So why did they have to sell? Because they raised way too much money at a 17.3 billion dollar valuation. How are you valued at 17.3 billion dollars when you have 600 million dollars in revenue? Let me do the math for you. That means you think this company is going to make at least 600 million dollars a year for the next 34 years. Years. Like, come on, we're like, this is getting absurd. And we will see the exact same thing happening with most of the niche AI products. Yeah, and I don't think this is AI killing. Yeah, all of them have raised way too much money as well, and then they will be... This is not AI killing SaaS. I think it's the venture capital mindset of one winner takes the market, which, by the way, if you look at any market, that is not true. The only market I can think of right now where that's even remotely true is search. And you know what? Google has faced a lot of litigation for antitrust issues over that. But, like, let's look at Airtable's business. We have Airtable, we have Coda, we have Smartsheet. All three companies, we're doing perfectly fine. Miro, we have Miro, we have Miro, we have FigJam, we had Jamboard, right? Like, all those companies were doing fine. It is not the VCs create this ecosystem where they convince founders one company will win the market and everybody else will fail. Yeah, because it's in their favor. It dramatically reduces the chances that company will be successful. Yeah, and I find this wild in these times especially because the consumer will be less tolerant of one mediocre solution, and with more players in the market, everybody was able to find the one solution that fitted their needs best. But now with AI in the picture, and Julie Su just wrote a very lovely newsletter about it and how she is doing hyper-personalized software for her business, and so do I with my Harvest solution. So my entire CRM invoicing estimate stuff is all heavily personalized now with the help of Claude Code. And of course, are we the tech bubble? Are we the forefront? Are we the early adopters? Sure. But will people become more and more capable of doing that and building stuff for themselves? Yes. And then a market where you try to have one player cater for everybody's needs, it's just a wild bet, isn't it? Yeah. Okay, so you just opened a giant can of worms because you're starting to make the argument that AI is— And we're already at 20 minutes. I know. No, no, no, I'm not arguing that it does that at a large scale. Well, I do think, like what you said is a trend. I do think technologists, technology companies will build more single-point solutions internal because they will benefit from them being completely customized. Yeah. I don't think the average consumer is going to do this. I don't think the average business is going to do this. I think what's more likely to happen is we're going to have lots of smaller players building much more targeted solutions for niche customer segments. Yes, hopefully. Right? Yeah. That's what I would love to see. That's what I would love to see too. But then the bet is a weird one, the bet of all these shareholders saying, like, there is one, the winner takes all in every market, basically. Yeah, I mean, I'm sort of hoping that venture capital just falls apart, but we'll see what happens. Which opens the can of worms on capitalism, but we're not going there today. Not tonight. No, I would not say venture capital falling apart opens the can of worms of capitalism. Capitalism is fine. I think venture capital is a whole different beast. Here's what I will say. I do think— This is going to change the way software is built, and that if you're building software, one thing to think about is, can you build your product for hyper-personalization? So I do think AI might unlock some new ideas around personalization in platforms, in products, rather than, like, I don't think the average restaurant owner is going to build their own point-of-sale system because they can do it with cloud code. Agreed. But I think a point-of-sale provider can now almost, like Notion. I actually think Notion is a huge leading-edge innovator in this space. Notion was built around this idea of building blocks. You have a database as a building block. You have a wiki page as a building block. I don't use Notion, so I don't know all their building blocks, but I've heard about the way they built their product was around this idea of building blocks, and that allows their customer to mix and match those building blocks in infinitely customizable ways. I think this is a pattern that we're going to see way more often. Yeah, hopefully. And we will see SaaS providers figure out how to offer hyper-personalization because AI makes it just now possible. Yeah, that is what changes. And as we called the episode, what will remain unchanged in times of AI, I think this is all the things we've learned in the past. So this is like, get your processes in order, get your rhythms in order, have your rituals still going. Maybe change them, maybe shorten them, maybe shorter cycles, maybe ready for hyper-personalization. But a lot of the basics are not changing, at least not in the ways of working. Keep talking to your customers. Keep talking to your customers. Start with that. You said you're retiring from saying this. I know. I can't. It turns out. I know. It's a reflex. So thank you, Teresa. Thanks, Petra. That was a spicy one. Ja.