Otto Hilska: What we see with a lot of companies is this AI paradox, which is individual developers 10x and the organization gets 10%. And that’s almost the whole discussion with a lot of companies right now — figuring out where is this coming from? What are the bottlenecks? And the exciting part is that the ceiling has moved up. So everyone knows that there is a productivity gain here somewhere. We just haven’t fully unlocked it yet.

Erin Backlund: I’m Erin Backlund, and this is a special edition of Engineering Unblocked. Engineering Unblocked is brought to you by Swarmia, the engineering intelligence platform trusted by some of the world’s best engineering organizations, from startups like Lovable, scale-ups like Miro and Bolt, and companies from the Fortune 500.

You might be wondering what warrants a special episode of this podcast, and I’m excited to share that this time we’re welcoming back our former host, Rebecca Murphey, alongside Swarmia CEO Otto Hilska to unpack the thinking and the process behind publishing the third edition of Build: Elements of an Effective Software Organization.

So welcome back, Rebecca, and welcome to you, Otto.

Otto: Hello.

Rebecca Murphey: Thank you, Erin.

Erin: It’s great to have you both here. Obviously we’re here to talk about the book, but it is a very exciting thing that we have Rebecca back in the room. So tell us, how have you been going? How is life there on the other side?

Rebecca: It’s a life-comes-at-you-fast kind of situation, I have to say. I joined Honeycomb at the end of May, and a week later I became interim manager of the engineering enablement team because the manager left. So I have not been doing the job that I exactly thought I would be doing, but I am still very much in the eng enablement space, and right now that means all AI all the time.

Erin: So the book has been printed for about a month now. How are we feeling? What are your initial thoughts about having this out in the world right now?

Rebecca: I think one thing I’ve learned in my brief time away from Swarmia is how relevant it all still is today. There are still problems that organizations really have, and AI is changing the shape of those problems, but it’s not changing the fundamental nature of those problems.

Erin: What about you, Otto? How do you feel having a third edition of your book out?

Otto: I think it was well needed. Clearly things had changed in a couple of years, and especially this year, the pace has been accelerating. We just had to decide a point when to actually publish the book, and I continue thinking about many of these topics. I think the way you use AI agents and give them more and more independence, that whole thing has been happening in the past couple of months, and I would have loved to include some more of that in the book. But maybe in the fourth edition.

Rebecca: I remember when we started in December of ’25, thinking that this will be a little nip and tuck, and then we can move forward. And then Claude Christmas happened, and so many things happened in the time between the “oh, this can’t be that hard” and when we actually had something that was ready to be shared with the world.

Erin: So what’s the one thing that you think would be missing from the book now, Rebecca? If for Otto it’s more guidance around agents, is there anything that you would have loved to, looking back, have included in the book?

Rebecca: I mean, of course a new book will be needed in two years. I’m not saying that this is the perfect book forever. But being out in the world, as it were, I find, just like I said at the beginning, the fundamental problems and the fundamental concepts that you need to know are still the same.

The theory of constraints is still real. It used to be that the constraint was around the production of code, the creation of code, and now the constraint has moved in both directions to, number one, code review, which is a really pressing problem — or challenge, let’s say — for us at Honeycomb. And on the other hand, can we generate enough things worth experimenting with to keep the engineers fed? And then finally, can the go-to-market team sell this stuff as fast as we’re producing it?

So I think if there’s anything I might have liked to add to the book based on what I know today, it is how this problem of engineering effectiveness is no longer an engineering problem. It’s about organizational effectiveness now. And I’m telling engineers to go talk to BDRs, and talking about whether we can send a tiger team to the BDR team to just eliminate some quantity of work. So I think that that might be what the next edition is about: this evolution that we can’t just think about the R&D organization anymore. We have to think about the whole business.

Otto: I agree with that. And we always included product in engineering effectiveness, but maybe it goes even beyond that now.

Rebecca: Yeah, because I’ve gotten to spend a little bit of time around our go-to-market team, not very much, but it’s very clear that they need enablement just as much as engineers need enablement around thinking about how to alter their processes now that they have these tools available.

Erin: I guess this feeds into the idea of non-engineers contributing code now, because it’s not now product managers with an engineering twist that are contributing code, but basically anybody, from salespeople setting up a go-to-market repo on GitHub. Talk me through how you feel like this fits into the effectiveness or the productivity conversation, if at all.

Otto: Yeah, we’re definitely seeing our product managers, designers, support, everyone creating a bunch of pull requests. Marketing updates the websites, and there’s some pretty big migrations there these days. So that’s obviously happening. At the same time, it’s important to remember the engineering side of things, so the same dynamics still apply.

Who is going to fix bugs if something arises from this change? And who’s going to proactively think about the architecture of this so that it doesn’t grow out of control? So the engineering job definitely still exists, and I think a lot of organizations are struggling when they see the visible results of doing something amazing and unlocking that innovation, and then the engineer comes and tells them that, well, it actually doesn’t work for 90% of the customers, so you still have to do the rest of the work somewhere.

So the collaboration patterns are going to change quite a bit. I think everyone needs to be open for this, because indeed, one of the biggest bottlenecks for engineering is going to be: do we know what we want to build and why? So if we can get support for that from the rest of the organization, amazing. But we also need to hear the engineering side of the story and make sure that the work gets prioritized. So we still need to empower the teams to make these prioritization decisions. And if we’re all very optimistic based on what we read on LinkedIn about AI’s possibilities, I think we might be missing half of the story.

Rebecca: I do think that a new job emerging for engineers is making it safer for those non-engineering roles to build things. So giving them environments where it’s really hard to screw up. Giving them environments where they’re adding a plugin, not creating a whole app from scratch.

When I got to Honeycomb, we had just shipped, or we were just about to ship, Autobots for code authoring. We had an Autobot for code review, but we were about to ship an Autobot for code authoring. And our field engineering team and our CS team are some of the biggest users of that tool, because suddenly Liz Fong-Jones on the field engineering team can be talking to a customer and making a pull request from her phone all at the same time, by just talking to Linear or talking to Slack.

So I think creating those environments — a platform, if you will, where those efforts can be successful — is going to be a real need for non-engineering. Now, Liz is an amazing engineer. She doesn’t need us to talk to her about security and stuff. She’s fine. But the world of “I have 50 personal Claude projects” is not sustainable and not scalable. So putting organization around that I think is going to be a really interesting next thing.

Otto: Yeah. We’ve definitely seen that, and we implemented this go-to-market operating system repository to be the platform for what the business side is doing with AI. And I think it’s pretty great to see, even when we just do normal deep research about a certain topic, like when we’re planning a product feature. When you have all the context about how we position against our competitors and what our customers are saying and asking and all of that, the results you get from a simple deep research are just so much better when all of that context exists. So building a couple of these enabling things really sets you up for success.

And similarly, for internal tools, we have a separate platform where we can securely deploy those small tools. So if we need to take something in from an API and push it to another API, that’s a super simple thing for almost anyone to do.

Erin: So if the book is Elements of an Effective Software Organization, do you think that this new enablement piece is one of these elements now, and has it always been? Because obviously in the book we break it up into business outcomes, developer productivity, and developer experience. Where does this enablement piece fit in, if at all?

Rebecca: I think, like so many things in AI land, this problem always existed. You’re in marketing, Erin, don’t you wish you had three engineers working for you? Of course you do. And you always have. But you don’t. So I think that is not a new problem. I think what the new problem is, is that your ability to do something very, very, very wrong has just gone off the charts. Before, you just wouldn’t do it. And now there’s a real risk of you doing something very, very bad.

Otto: And the element of developer experience has existed for a long time, and it is really about: is this an environment where it’s easy to get my job done? Do we have the platforms and infrastructure to support that? I think many of these AI topics actually just go back to building platforms that make sense for your organization.

So I think it’s very much the same thing as we see throughout the book. The same concepts always existed, and the best companies always just looked at their problems and designed a scalable solution that takes into account organizational aspects and technology and customer value and everything else. So in that sense, I think, again, the fundamentals are very much the same.

Erin: One question that I do have is, if we are saying that the fundamentals have not changed, a skeptic would hear that and think, well, why change anything? We don’t have to change anything. So what does the time that we live in now demand of a leader that is new, that is not just the fundamentals?

Otto: I think it requires a much deeper understanding of what those fundamentals actually mean. I would argue that in a way nothing’s changed in the fundamentals, and on the surface level, how things look and what you should be doing has changed completely. So the funny thing is that yes, there is a connection between cycle time and work in progress. That hasn’t changed. That’s the kind of dynamic that exists. However, there’s now new information on how the system works, and the bottleneck’s in a different place. So you still have to model your organization in a way. But yeah, you definitely cannot think that things are the same, because they are not.

Rebecca: And I would add that a big thing that I’m seeing among software engineers, I’m often trying to remind them that the fundamentals are the same and this is new technology. And it has always been your job to understand how to use new technology to deliver value to the business. That’s not new.

What I think is new is that to effectively use this technology to provide value to the business, you better be paying a whole lot more attention to the business than maybe you were in the past. Because things are changing, you need to be aware of the context in which you’re operating in a way that I don’t think was as necessary five years ago.

I think five years ago, certainly 10 years ago, you could take tickets, and you could have a healthy team that was cross-functional, but fundamentally people were taking tickets. And the robots take the tickets now. That’s not the interesting part. But again, that doesn’t mean that the rules have changed. The rules are just a different shape, but they follow the same fundamentals.

Otto: Yeah. And I would argue that even before, the best companies tried to find engineers who are very outcome-oriented, who are very good at managing their work, who understand customers. And even the performance management process has often been around impact to the customers and to the company. So that has existed, but I think no company really managed to hire 100% that kind of profile.

And I think that makes sense, because you want to hire people who are excellent database architects, and you want to hire security folks, et cetera. People just have different shapes of profiles, and you put teams together so that you have all these shapes represented. But now, if you just rely on a product manager generating tasks for your team, and everyone’s able to consume those tasks very quickly, and mostly they just offload them to an agent, that is simply not sustainable and something needs to change. So indeed, I think that’s a great example of something where the way you need to compose your teams is actually quite different.

Erin: It’s a good point to make now, I guess, because in chapter one we talk about teams and how teams should be structured and how teams look. So what do you think that teams should be structured as now, with the way that we are working?

Rebecca: I’m very interested to hear from Otto. I know as I was leaving Swarmia, you were talking about the new way, and it’s been three months or so since I left. I’m very interested in how that’s working, because that has been kind of my North Star. So maybe you can talk a little bit about that.

Otto: Yeah. So when we started talking about the new way, we had this realization that if you’re working on a problem where most of the work actually happens outside of coding — coordinating with customers, defining what you’re trying to build, prioritizing with the team, reviewing code, all of that stuff — AI is actually not going to help you as much. So you have to figure out how to turn this whole thing around and design the process and system so that you get the most benefit. And the big pitch for this is that developers’ job used to be building the product, and now developers’ job is to build the system that builds the product.

So that’s the idea. And that means you’re going to build a lot of automations. You’re going to build custom review agents. You’re going to build custom agents that do some things completely automatically, like if there’s a production exception or slow queries happening in the database. Someone should probably investigate those proactively so that you don’t have to go through the coordination challenges. So that’s the underlying idea.

And with the teams, it just means that you need to have — I wouldn’t change anything about the structure itself, but I think the size of the team and the developer-to-PM ratio is definitely something that’s changing. So now, if even a small development team is able to be extremely productive, you still want to have a handful of people so that you get to share knowledge, and you don’t have this single person knowing everything about a subsystem, because it’s still challenging if that happens and then that person leaves.

But otherwise, I don’t think the structures change as much. Then there’s the cultural aspect that we just talked about, which is: how much ownership do you need to take of your product work, and how interested should a developer be in the customers? I think this is where organizations also have to realize that if someone’s worked taking tickets for the past five years at the company, and suddenly you are trying to empower them to make all of their own decisions, they might not have the customer connection. If they were never allowed to talk to a customer before, they might not be in the place where they can make great prioritization decisions right now. So you have to also build towards that capability so that you can build that culture.

Rebecca: That very much lines up. I think we still need people thinking about product. We still need people thinking about user experience, and we still need people thinking about engineering. Those three roles still exist.

But I’m definitely seeing what you just talked about, Otto, that there are people who not only do not have the experience to do this new way of working, they may not have the desire to work this way. They may have liked that they got to be heads down writing code all the time, and that may have been part of their identity. And I think that we’re seeing that in the industry very, very much, that engineers and even engineering leaders are really having to reckon with what is my job anymore.

And of course, this also opens up the whole conversation about where are the juniors, and who’s growing the next generation of software engineers? I think it’s very hard to get the industry as a whole to suddenly pay attention to that. But whoever starts solving that problem, the industry should be very grateful to in about five years.

Otto: The jobs are definitely changing, and I feel managers need to become more hands-on, and you need to start playing with AI to understand the possibilities of that. And you can actually be a highly productive manager who, because of the context that you’ve built, is able to address tasks that would be very difficult for your team to take on. So that’s the manager side.

But then developers need to almost become more like managers at the same time. So you are suddenly allocating capital to different projects. You might have this migration that you can do with $20,000 of tokens and not too much of your time, and you have to figure out, is this a good use of company money, and do we want to do this? And you start orchestrating it. You start focusing more on how do I collaborate with all the other teams, how do I prioritize these things. So everyone’s job is changing, and it’s not always exactly what people expected to do with their career. So a lot of people are definitely looking for their new work identity as part of this.

Erin: But I think, at least to me, the three table stakes that we mention in the first chapter of the book — empowered teams, rapid feedback, and outcomes over output — it sounds a lot like what we are talking about now. It’s just that the individual role of engineers is moving more towards this outcomes and product sense, basically, is what I’m hearing.

So for an engineering leader who picks up this book and they want to take the first step towards this new way, what do you think the first thing they should do is? Maybe, Otto, you have an idea about this.

Otto: One thing to be careful with is that when you read someone else’s stories, you read how they solved their biggest problems. And even when we wrote about the new way, we started talking about concepts like campfires, which is a meeting type where the team comes together to build a feature. It’s meant to solve the problem of not having all the knowledge you need to build something really complex. So you bring everyone together in the same room, and instead of a plan, you produce a working prototype, and you all talk about it while someone’s prompting your coding agent.

But that’s a very specific solution for a very specific type of issue. Although I do believe that practically every organization is struggling with figuring out how exactly are we building and what are we building. So in that sense, that’s quite useful for a lot of folks. But many of these solutions might be more platform specific.

Like we figured out as a business that we have to build a lot of integrations, because we have to get people’s data into our engineering intelligence platform so that we can help you analyze what’s happening in your organization. And integrations used to take two months to build, which made it a very difficult prioritization decision. Now the team has been building skills and platform support so that we actually shipped our first basically two-day integration using AI agents. And that’s an amazing capability, and that’s probably one of the best ways to move this business forward, because each new integration means that we support all the customers who use that tool. So it’s an amazing pipeline builder for us as a business.

And I think similarly, every company needs to adapt it to their own use. I would say that building the system that builds the product is the key. I think practically everyone will struggle with code review. Everyone will struggle with product management and decision-making, so those are probably pretty good starting points, and that’s what we’re writing about in the new way. But there’s going to be some platforms that are very specific to you, and that’s something that you need to figure out on your own.

Rebecca: Yeah. I think I’m going to go in a different direction with that question, and just say the first thing that somebody needs to do after they read this book is talk to their people. Talk to their people. How are they feeling? What challenges are they seeing? What are they struggling with?

There’s a team at Honeycomb, the core services team, that actually have scheduled time to sit down with their manager and learn all about it, because they have been really excellent at iterating through how the team is going to work as a team using AI, and what feels good and what feels bad and how they’re going to maintain human connection and all of these things.

So I think talking to your team. We ran a survey using Swarmia and got feedback that was very helpful in supporting my spidey sense about what was wrong. Code review was identified as the number one bottleneck in a free-form, open text question. People could’ve said anything, and mostly they said code review.

But I do think that talking to your people, surveying your people — not every day, but talking to your people, surveying your people, and understanding what’s hard for them. And I think we got so excited about how much code we could produce that some people kind of forgot about everything else that software engineering involves, and that the business involves.

So the best teams, the healthiest teams that I’m seeing, are talking to each other about it, running little experiments, trying things and seeing what works. And sometimes they work, sometimes they don’t. But saying, for a week, let’s only review these pull requests and just click merge on these, or whatever. So yeah, I think that in order to use the book, you can’t just use the book. You have to talk to your team, the people around you.

Erin: Something you said just now, Rebecca, reminded me of something I’ve heard you say recently, Otto, which is about this individual versus organizational productivity, and it’s something that we talk about in the book as well. So it sounds like this individual productivity piece is that they’re getting faster and faster and faster, and then the first roadblock that we hit is code review. So maybe let’s talk a little bit about this individual versus organizational productivity, or speed, as it were.

Otto: There’s a lot to be learned from your best individuals in how they use AI. We are seeing that the output is very different when you look at the top performers. So obviously, and we talked about this already last year, AI is an amplifier. So there’s a very important aspect still, which is this performance management aspect and who are my best people. How can I understand why they are successful? How can I help everyone else get there? What can I do to get them a similar context and things like that.

At the same time, the organizational level outcomes are really dominated by the organizational level stuff. So if your organization is not ready to review that volume of pull requests, then they’re going to be just left open, and actually that person’s productivity doesn’t really do any good for the company. And in fact, if someone’s really just constantly plowing forward and everyone else is struggling to keep up, that might be really bad for the organization, and people will lose their motivation.

So again, things have changed significantly. What we see with a lot of companies is this AI paradox, which is individual developers 10x and the organization gets 10%. And that’s almost the whole discussion with a lot of companies right now, figuring out where is this coming from? What are the bottlenecks? And the exciting part is that the ceiling has moved up, so everyone knows that there is a productivity gain here somewhere. We just haven’t fully unlocked it yet.

And so suddenly the value of solving your code review bottleneck or your product management bottleneck is immense, because you know that you can have a multiple of output from your current investment. So that’s a pretty amazing result.

Rebecca: I would tag onto that, that senior, especially very senior engineers need to really, really, really think about what their job is, because that is a pattern that I have seen where senior engineers are suddenly very prolific and their team is drowning, just like you said. The team doesn’t need code from that person. The team needs leadership from that person, and mentoring, and coaching, et cetera.

So I think that is the job of the senior engineer, and they should have always been doing those things. And I’m talking again about staff plus engineers. There are staff plus engineers who get this so intuitively, so all of this is so fascinating and exciting and so much stuff that you can get done, and you can build your own personal platform to get things done. It’s amazing. Not the job. Like, the job, especially right now.

A thing that I also see is that the same people are talking about the cutting edge all the time, right? But where are the people who are not part of those conversations? Where are they getting their information? Do we expect them to drink the same fire hose that I’m drinking about all things AI? I don’t think that is, to Otto’s earlier point, it’s really hard to hire 100% of people who want their job to be drinking from the AI fire hose and building personal platforms.

And so I think a big challenge that I’m addressing at Honeycomb is how do we propagate that knowledge intentionally, regularly, in a predictable place or places? What format? Is that video? Is that blogs? Is that stand-up comedy? I don’t know. But thinking about how do we find those folks who aren’t AI-pilled. We don’t need to AI-pill them, but we need to get them the tools and the knowledge that they need to do their job.

Erin: It sounds a lot like that is the thing that, if you achieve it, will get you up from, maybe not from 10% to that 10x, but at least further along that way.

Otto: Yeah, it will certainly help with the AI tool usage. And there’s a big difference between someone who uses Cursor with autocomplete versus someone who uses coding agents locally, versus someone who has a bunch of Slack threads with a cloud agent ongoing to fix smaller things as they get escalated from customers. And then finally, someone who can complete huge migration projects with a lot of agents running at the same time, and that’s the way you spend tens of thousands of dollars on tokens very, very quickly.

So indeed, there’s this leverage scale that you get from these agents, and if you’re the one telling it to do every single step, then you are the bottleneck and you’re not fully there. So there’s definitely this side of the problem that gets solved by teams sharing knowledge about AI tool usage. And we have really good examples from our own teams where the team proactively decided to have Friday AI knowledge lunch type sessions, and then everyone was sharing what skills they are using and what new tools they are testing. And that team got really quickly to a really high-performing position.

But then you still have that other side of this, which is: do we as a team control everything that’s affecting the organizational performance? We might not control dependencies to other teams at all. We might not control how the company expects us to review code, because we have SOC 2 Type 2 that dictates how things need to happen. So there’s definitely two sides of this, but of course, the AI enablement is an important one that you need to do first before you even hit the problems of the second stage.

Erin: If you don’t mind, I’d like to ask a little bit about chapter three, developer productivity. We spoke about it a little bit before, that most of these engineering leaders, if you want to do something with this book, you should have a good idea about all of the elements in this book. But I wanted to ask about the classic productivity challenges that we cover in the book and how relevant they are now. Things like working in larger increments, or multitasking, or cross-team dependencies or sequencing. What’s your take on this now? How much of this can we keep for the AI world?

Rebecca: I think people used to hear developer productivity and that translates to code. The most naive translation is how many lines of code are people producing? And that is an industrial mindset that has been with us from the beginning. But as we talk about in the book, it was never about that. It was still about whether you have the tools, and the knowledge, and the context that you need in order to do your job successfully.

I haven’t read the book since it came out. I read it when it got here, but then I haven’t read it again. Sorry. It’s on my desk though. So I don’t know how much we talked about that exactly, but I think that the shift that has to happen is not that the developer productivity chapter is wrong and bad and should go away, but that it was never about the code, and now it’s especially not about the code.

And we need to think again about the theory of constraints. Everybody needs to understand Goldratt’s theory of constraints. Everybody should read Reinertsen’s The Principles of Product Development Flow, because nothing about those books is wrong. Still read that. The theory, nothing about it has changed, nothing about it has suddenly become wrong.

What everyone wishes is that there was one number, that at the end of the developer productivity chapter we had said, “and here’s how you measure this.” And that also persists, that desire for the single number. I was thinking the other day, there’s Drake’s equation, which is a probabilistic estimate of the number of planets that contain life in the entire universe, and the purpose of that equation wasn’t to come up with a number. It was to list all the things that we would need to know in order to come up with a number. And I feel like this productivity question is the same. How do we boil this down to, like, the number of aliens in the universe? There’s too much that we need to know that we simply don’t know or simply can’t know in time.

I think any measurement of productivity has to include business outcomes. Did it sell? Did we get more revenue for this? Did we retain a customer that we were going to lose? Lots of reasons that work is valuable, but we don’t know until months or maybe even a year later what impact the work we did had on the business. And so any number that doesn’t include how the business is impacted is missing the most important part of this calculation.

I wish I had put the Drake equation part in the book, because I think that is a really good analogy for the challenges of measuring this. And again, in my first weeks at Honeycomb, we’re still having the “wouldn’t it be great if we could just measure productivity” conversation.

Erin: And we think about frameworks in the book for thinking about productivity. That’s what we talk about, frameworks for thinking about productivity. Of course, the classics, DORA and SPACE. And now I’m seeing more and more people thinking about DORA as a quality thing, as much as it is or has been a productivity thing. Perhaps we can talk a little bit about that, because we have a new section in the book in the developer productivity chapter which is all about productivity and quality, and how these two topics interrelate with each other. So how do they? How are we thinking in 2026 about productivity and quality?

Otto: It’s almost like this historical battle between speed and quality, and you have to choose which one you want, and that’s been the debate in a lot of engineering organizations. And what DORA brings up, and how we see it as well, is that ultimately a lot of productivity work is quality work, because your ability to ship something to production is dictated by how afraid you are to deploy to production. Because if you’re afraid that you’re going to break everything, then you’re going to slow down, try to do some extra testing, and that’s what’s really slowing down a lot of organizations.

And if you know that you can trust that if CI passes, then production is going to work — maybe the button is going to be misaligned on the screen for some devices, that can still happen, but I know that I’m not breaking everything by doing this. So investing in that side of quality is super important. But I also feel like DORA is maybe quite focused on the CI/CD side and the deployment pipelines.

And of course, there’s another aspect of quality which goes hand in hand with the product management discussion we just had. Does this actually do the right things? Are we making good decisions about the products that we’re building? And that, on the other hand, requires the ability to get early prototypes to customers and test with production data and things like that. So again, it goes to building the capability there.

So I think a lot of companies are realizing this, but maybe they don’t realize how much investment it would require to get from their current place to that optimal place. And maybe they also don’t realize the total cost of working on a codebase that is very miserable to work with, and where you hit every single problem every time you try to make a change.

So I feel like this is going to be the discussion that dominates the next phase of this AI era, because you can definitely build a prototype and keep adding features to it for a while, but there is going to be a moment when you realize that 80% of your time goes to doing ad hoc bug fixes, where you’re basically fixing this bug back and forth because both implementations are actually wrong in some ways. So a lot of companies are going to find that out very, very quickly.

Rebecca: So many times, it’s still all the same. It was always bad if you were scared to ship to production. That was bad in 2020. That was bad in 1998. It was bad if you were afraid to ship to production, but companies didn’t do anything about it because they couldn’t. Assuming the best of intentions, they didn’t understand the impact that was having on their overall ability to deliver value to customers. Now, with the flood of code, that has become so clear. These things that we avoided for years, that we patched over with manual QA teams — massive manual QA teams and UAT and all of that — those were always problems. Now they are problems that we have no choice to deal with. We must.

Otto said that quality is going to be a big focus of this next era, phase, whatever we want to call it. I also think cost is going to be the other piece of this. Never before has it been so easy to attach dollar signs to a pull request. Never. And that raises a lot of really interesting questions. It raises a lot of questions about who should have access to Fable, and who can get along just fine with Haiku. You want to be the engineer who only gets Haiku? No, you don’t want to be. Don’t worry. We’re not going to just limit engineers to Haiku.

But I think two things about the cost question. Number one, setting aside all ethical and moral dilemmas about AI and the climate, I think quality is the only thing that is still a reasonable concern. It’s a reasonable concern that a year from now we will be in a quality crisis as an industry. That is, I think, a legitimate concern, that we can’t predict the future, what’s going to happen. Yes, the models have gotten so much better. Yes, we can put in all sorts of quality constraints. But we don’t know what’s going to happen to our ability to operate our systems.

And that plus the cost question. It’s going to be real interesting what happens in the next 12 to 18 months, because we just went through time to renew our Anthropic contract, and those numbers are not the same as they used to be. And we’re having real conversations about how do we, do we tolerate the spend? This is engineers’ worth of spend. Is that good? And these are questions that we have, because we’ve never been able to pinpoint, well, Joe is worth this much money and spent this much time, and therefore this pull request cost us this much money. Like, no. But when the robots do it, the robots will tell us exactly how much it cost. And I think that is the other interesting thing that’s going to happen in this next period.

Otto: I think one dynamic with quality is that if you go in the wrong direction for a year, it will probably take someone else at least five years to fix what you just did. And I think the same dynamic applies even if you’re using AI to fix this thing. You went in one direction for a year and you spent the labor cost plus the tokens to get there. And now even if you’re able to give it to a future AI model that’s super capable and is able to solve these problems, there’s still probably 5x the amount of work required to then do something about it.

So indeed, to your point, there’s real cost to it. So I think starting to watch the total cost spent on things, both how much did we spend on developer salaries, how much did we spend on tokens, and then also trying to map these work streams so you understand this is the initial spend I had on this project, and this is how much I ended up fixing it afterwards. I think that’s another important lens to have.

Because I think a lot of companies right now are probably making decisions about buying SaaS software, and you’re able to build a prototype of Jira in 10 minutes. And then you realize it doesn’t have the reporting, and you realize it doesn’t integrate with any of the other tools, and you start adding up, and you realize that actually the total cost was something totally different. So this cost and ROI thinking definitely needs to become a standard in every single engineering organization, because an individual developer can deploy $20,000 on tokens just like that.

Erin: Do you think this is where the discussion has moved? At least, well, for you, Otto, you talk to engineering leaders every day. Do you think that the focus has shifted from people coming to us at Swarmia to buy our product to understand things like how fast are we going, what’s our cycle time like, is the flow of work moving properly? Do you see that the topic has changed to this cost equation of how much is going where, and then what should I do about it?

Otto: Absolutely. And it’s funny, because we ran a product discovery process in December to understand the cost side and what people think about it, and 100% of people told us that they’re not interested in cost at all. They just want people to adopt AI tools and maximize token spend. And then they got what they wanted, and it was maybe four months later when the opinion started shifting. And now practically every conversation we have involves some element of cost and the ROI math for AI tools.

So obviously there’s some pricing changes behind this. If you were just spending $20 or $100 on a subscription, that was a very different story than if you’re now actually paying for the tokens. And so far, models getting better has happened at the cost of more expensive models. So it’s not as if the top model stayed the same, but you’re constantly spending more and more money on this. And you should remember that when you’re planning your strategy for fixing the quality problems next year with the better model, because it will likely not be cheap.

Erin: One final question on developer productivity before we move on. So what happens to metrics now, because a lot of the work is not being done by people anymore? At least, if an agent opens a pull request and then an engineer merges it, then whose work is it? And then in the same vein, is it still worth tracking things like cycle time or similar metrics?

Rebecca: One of the first things I did when I got to Honeycomb was to connect the Swarmia MCP, the Linear MCP, and the GitHub MCP and start asking questions about DORA metrics, still. Because what’s interesting is not necessarily — Honeycomb is a pretty healthy, mature engineering organization. We don’t have a crisis around cycle time at all. But it was still really interesting to see where the differences were in individual teams, how different individual teams were spending their time on code review versus code authoring versus whatever. You could also very plainly see that waiting-to-review number just zoop, right?

And so I do think that metrics are still really important. We just, again, have to get out of our head that code is the thing that we’re trying to measure. We’re trying to understand where processes are breaking down. I did that work. I could see very clearly code review was a bottleneck from that work that I did, and then a month later we did the survey, and people told us exactly the same thing.

So I think having those metrics and having access to tools that can assemble those metrics is still really, really valuable. We just might not care so much about certain metrics that we used to care about, but new metrics will come. And that one around waiting for code review, which is part of cycle time, again, was so obvious that that was the next thing that the engineering team needed to go tackle. And I wouldn’t know that. I would intuit it, but I wouldn’t know it without metrics.

Otto: I don’t think it matters that agents are creating more of the code, because these were never metrics about tracking individual performance. Cycle time is about how quickly work gets through the system, and that work can be initiated by an agent, but it’s still going to go through some kind of checks and someone’s going to have a look at that. So if you already have a code review bottleneck, now you just have a worse one.

And I think a lot of companies actually prior to AI got to a very stable situation where if your team is roughly the same, and you’re not hiring a lot and everyone’s familiar with the company, you might solve your cycle time problem one time, and then it doesn’t really get much worse after that because people know how to deal with that.

But right now, the pace of change is such that the agents are doing something new all the time, and there’s more and more pull requests every single month, and whatever worked last month doesn’t work anymore. And again, if you have this kind of bottleneck in a critical part of your funnel, you’re never going to ship anything out to production if you don’t survive the code review phase. So in that sense, it is more important than before to understand where the bottlenecks are.

I think you’re going to have to be a little bit more creative than before in solving that, because historically, all it really took was, well, let’s make the pull request more visible, and let’s enable some Slack notifications, and let’s do a couple of things like that. Now, you might have a much more challenging problem with culture of: am I pushing out code that I haven’t even read myself? Because that’s going to annoy my colleagues quite a bit, and they are not going to read the next pull request coming from me after that. So there’s a lot more of these discussions that need to happen, but the metrics actually tell a very good story about how well that is going. And it is currently an industry-wide issue that everyone has a problem with code review, and you have to be very intentional about solving that.

Rebecca: I think one of the metrics that we are going to see become more and more important — and Erin, like you said, it’s already in DORA to some extent — but metrics around quality, and metrics around risk. What is the risk of how we are operating right now? I think that those are going to become necessary. Mean time to remediation is great, but I think that it really just scratches the surface of the things that we need to be concerned about now with AI. And I am so curious to see what the next DORA report has to say, and what they abandon entirely.

I will get in trouble if I don’t say the word observability once, because that’s what Honeycomb does. So I wanted to say I really think of this state that we are in as having gone from a people and process challenge to an observability challenge, and being able to understand — you can ask the agents what have you been doing today, but they might lie to you. You don’t know. And you don’t know how much they’re costing you, any of these things. And so I think this becomes not a people management problem and not a process management problem, but it becomes a systems management problem. And what’s really cool is that it’s so much easier to interrogate those systems than it was to interrogate systems before.

Erin: So we’ve talked a lot about code review being the bottleneck. And I’d like to hear, Rebecca, if you have any stories or anything that you can tell us about what you’ve found through your research in your new job.

Rebecca: Funny that you ask. So yes, I show up at my new job. I know that I’m going to be a manager for a little bit. But I’m basically managing the team that I would be working with all the time anyway, the engineering enablement team. And so I focused upfront on what are our bottlenecks? Where are we getting stuck? And code review is very obviously one of them. And of course every industry’s talking about it too, but my data that I could point at, and that was cool.

So the next thing that I did — and I found out later that one of the people on my team did a similar but different experiment and arrived at the same-ish conclusion — I looked at, and I’ll write up the methodology somewhere else, but I looked at a few thousand pull requests at Honeycomb. I looked at pull requests that got merged and pull requests that got closed unmerged. And I looked at whether a human comment had materially changed the direction of the change.

Thankfully, somebody on my team had, about a month or two before, worked with security to agree that we could still be SOC 2 compliant if our reviewer was automated, and that we didn’t necessarily need to have humans doing the review on every single thing. We first applied that to Dependabot. We were flooded with Dependabot pull requests, and we classified them and decided that some set of these are really quite fine and we don’t need a human to look at them. And of course, Dependabot alarms are going off far more than they used to because AI is finding bugs.

So I came to the conclusion from looking at this and learning some stats, that somewhere in the order of 90% of pull requests were unaffected by human review. We can argue is it 92, is it 87, is it 75? But some significant portion, and my teammate found essentially the same number. Some significant portion of pull requests were not benefiting from the time they were spending waiting.

Erin: So are you then, Rebecca, in the camp of humans don’t need to review code anymore?

Rebecca: I am in the camp of humans do not need to review all code anymore, 100%. And so we actually have a goal, the team I’m on nominally, we set a goal that by the end of the year we want to have 25% of pull requests being merged without human review. And the general consensus is that that feels okay to people. That doesn’t seem crazy to people.

We did a lot of talking and sharing of feelings about why we value code review, and that goes back to the campfires and the stuff that Otto was talking about elsewhere. The biggest thing that people were concerned about if we stopped doing code review was all the side effects of code review: staying familiar with the codebase, agreeing on the architecture. All these things that we tended to do during code review. We don’t need to do those during code review. We can do those when we open the ticket, when we start the ticket, when we pull the ticket in and put it in progress. We can have those conversations with the team.

I’m not recommending waterfally stuff, but I’m saying if you’re doing something substantially large enough that the team needs to know about it, that doesn’t mean you need a human pull request review necessarily. That just means the team needs to agree upfront on how something’s going to get done.

Do I love this? There are camps out there that say blocking code review was never a good idea. Bryan Finster is very much a believer that blocking code review was never a good idea. So this isn’t a radical new concept. It’s maybe radical, but it’s not new. And I think that whether you like it or not, the economics of the SDLC demand that you do something about code review. We can’t keep doing what we were doing before.

Erin: What about you, Otto? Which camp are you in, code reviews or no code reviews?

Otto: I’m similarly in the middle. So I think code review plays a very, very important role. I’m old enough to have built software before code review really existed, and I do remember those times when we were all looking at code and saying, who wrote this, and how did this ever get to the product? That was a very, very common discussion back then.

And at the same time, doing nothing is not really an option. I have 14 pull requests currently open, and so I’m a victim of this same situation, and I’m part of the problem, because I’m not driving these pull requests forward. And juggling 14 things at the same time is probably not the right number.

So I would think about it from the perspective of what do we need to be able to review less, or a smaller percentage of the pull requests. And for that, you’re going to need, first of all, a generic AI reviewer bot. And actually, I think those bots are pretty good at finding bugs. So I would argue that they’re better than humans in finding bugs where this variable wasn’t used correctly here. They do a super thorough job with that kind of thing.

Then you should have something to enforce your own policies that are specific to your own company, and you can build these own reviewer bots with a simple GitHub action that just runs Claude and checks whether we follow the patterns that we have documented for ourselves. Because even if you have a skill for doing something, it’s 50/50 whether the agent and the person invoking the agent actually realized that this skill should be applied here. But we have, for example, one around reviewing migrations, because migrations are the easiest way to take down the production environment for us, because we have a very, very big Postgres instance that we run this with.

So you need to build these things, and then it gets interesting, because if you’re saying that not all code needs to be reviewed, there’s going to be a line somewhere, and people are going to be debating where that line is. And of course, from the individual’s point of view, the less time you have to spend on code review, the happier you are and the faster you’re able to move. While at the systemic level, it could be that these individuals are making some pretty catastrophic choices that will be very expensive to pay back later. So you have to be willing to have this conversation constantly, and have ownership of these things so that the team actually feels comfortable challenging each other and keeps doing pull requests in the areas where it actually matters.

Rebecca: One thing that we are talking about at Honeycomb is the importance of teams being able to set their own thresholds. We live in a monorepo, but we understand that certain teams are going to have different thresholds for what’s okay to auto-review. Our storage team, for example, has a lot of feelings about what code comes into it, because our bread and butter is storing the events that you send us. And if we can’t do that, then that’s a big problem. So that’s how we’re thinking about solving that problem.

I think you also have opportunities before any code gets written to build a system, build a platform, build a harness, whatever you want to call it — build a system that lets AI make better decisions in the code. So you might re-architect something and document it really well, and now it’s very clear how this should be done in the future, to humans and to AI. But I think that is going to be part of the work, because platform teams are more important than ever. It’s the platform team who’s going to build the systems that prevent the bad thing from happening in the first place. And I think that is going to become a really essential part of your engineering team.

Otto: I think it’s also important to remember that different teams have very different blast radius. And in our case, the platform team is maybe less AI-pilled than some of the others. And I think that’s simply a good thing, because they’ve built an amazing platform that helps everyone else build their products a lot better. And then when we have teams whose focus is on something more customer facing, they get this super fast iteration speed from the good platform, and also from being a little bit more forward leaning with using all the latest AI stuff. Whereas if you manage to create a huge mess in the core of your product, again, that can be very expensive to fix, because everyone’s building on top of that.

Erin: So we get to the end of the book, we reach chapter five, putting it all together. And if I were an engineering leader and I had put it all together, I’d read the book and I’d done everything, and things didn’t get better, what would you bet went wrong for me?

Rebecca: I think there’s a couple of things that typically go wrong here. One is if you aren’t placed properly in the organization, this stuff can be really, really, really, really hard to change. Reading this book as a line manager is going to help you improve your team. It’s going to help you talk upwards about these challenges. But as a line manager, you do not have the levers that are probably needed to make the organizational changes that might be required.

So you as a line manager can’t say, “I refuse to use manual QA,” if it is the policy of your organization that everything goes through manual QA. Even though our productivity conversation is done as soon as you say that everything goes through manual QA. We’re out. So I think that’s one really common failure mode: you just aren’t positioned in the organization and don’t have senior enough sponsorship to be doing the work that needs to be done.

I think the other way, and there’s a great talk, I can’t remember his name, but there’s a great talk by somebody who used to work at eBay, about how they did it. They did everything in the book. Did a great job. But the rest of the organization had its own challenges, and that prevented the organization from succeeding even though engineering was shipping everything they were asked to on time. Everything was great. Great platform, great everything, but the business surrounding the engineering organization wasn’t capable of turning that work into revenue.

Otto: I have some very similar thoughts. I think when the agile hype was at its peak, people focused a lot on the team level, agile and ceremonies and that kind of stuff, and then people started realizing that, okay, well, there’s maybe a level above. You have to look at the whole organization.

I think, and we touched on this earlier in the conversation, that there really needs to be a focus on how does this business run overall, what does this company want to achieve? And then this needs to start from the top. If you are indeed that one person managing one of these teams, it is important to read this book, because it’s going to uplevel the conversations you’re able to have with everyone. You’re going to be able to move from code reviews or no code reviews to understanding the nuance of the problem, and you might have some models and frameworks worth talking about things. And you’ll have a book to give to the other person you are talking to so that they will maybe understand what you’re doing. But if you don’t get the buy-in to change as an organization, it is really difficult to change it as an individual. So initiating the conversation is the thing that you need to do, but finishing it will take the whole team.

Erin: And then I’m going to close with a final question for you both, and I’m very interested to hear the answer. So what would be the next book that you would write in this space or elsewhere?

Otto: We added this new section about quality in this book, and we’ve been writing a couple of blog posts on the topic recently and been adding some quality metrics to our Build framework. So it’s definitely a topic that has been under-discussed in the industry so far. Obviously, everyone understands that quality plays a big role, but how do you structure the conversation around that? What are the different elements? What are the metrics that you’re able to look at? I think that’s a really interesting topic, so I would like to dive deeper into that.

Erin: And what about you, Rebecca? The next book that Rebecca Murphey would write.

Rebecca: I don’t have any plans to write another book right now. But if I did, I think it would be about what we talked about up at the top, about how it’s about an effective business, not just an effective software organization anymore.

There’s a great book by Tanya Reilly, The Staff Engineer’s Path. It’s a book about what it means to be staff plus. And I think I would love to write something about what it means to be an engineer today, and what engineers need to know about the whole business. Because, again, I say BDR to a random engineer and they don’t even know what those letters are. Business development representative or something. But one of the things that was so beneficial to me at Swarmia was working across the business and learning how the entirety of the business fit together, and I think that’s going to become a really critical skill for software engineers. I would love to write about that, and how AI is changing how businesses need to run. I don’t think I know enough to write that book, but I would read that book. And maybe someday I’ll know enough to write it.

Erin: I would read that book, too. I would read both of those books. So I’ll be waiting with bated breath until I see a new release from the both of you.

But for now, we’ve come to the end of our session today, and thank you so much to you both for being here. It has been great to talk to you both, and we’ll see you on the next episode.

Rebecca: Take care. Thanks for having me.

Otto: Thanks.

Erin: If you’re keen to get your hands on a copy of Build, you’ve got a few options. You can read it for free on our website at swarmia.com/build. You can buy the paperback or Kindle edition on Amazon, or get in touch with us and we’ll send you out a copy. Thanks for listening.