The state of DORA and developer productivity

Rebecca: I'm Rebecca Murphey, and this is Engineering Unblocked. Engineering Unblocked is brought to you by Swarmia, where we're rethinking what it means when we talk about developer productivity. Today, I'm here with Nathen Harvey, and he leads the DORA group at Google Cloud. So if you have questions about DORA, you're going to get your answers here.

Thank you so much, Nathen, for being here. Yeah, tell me a little bit, like how do you get this job? How does this happen?

Nathen: Yeah, yeah. That's a really good question. First and foremost, thanks so much for having me, Rebecca. I'm excited to be here. How did I get this job? That's a really good question.

So, I joined Google Cloud about six years ago, and about a month and a half after I joined Google Cloud, Google Cloud acquired DORA. I'd like to say that that was because I told them that they should do so, and that's how I got the job – you just go buy the job that you want.

Uh, that's not very practical advice, I'm afraid. And, of course, I had very little – absolutely nothing – to do with the acquisition of DORA. But I had worked with the team that founded DORA, and came over as part of the acquisition. In particular, it was Doctor Nicole Forsgren, Jez Humble and Soo Choi all came over as part of the acquisition.

I'd worked with all of them at Chef Software prior to them forming DORA and prior to me joining Google Cloud. And so it was a nice little reunion that happened. And then, over time, those folks have moved on to other things and Google Cloud has kept the door research program going. It felt like a natural fit for me to step in and take on a good, strong leadership role to help drive that program.

Rebecca: Google can do anything it wants. But what's the connection? Why is DORA at Google Cloud?

Nathen: Yeah, that's also a good question. I think that Google is an organization that's pretty well-known for having great engineering practices, software delivery practices, and so forth. DORA is, or was, an organization that was well-known for researching those practices across the industry. And, when you think about how Google does things – how Google does engineering, how Google does software delivery – there are a lot of really great things there. A lot of really great things that are unique to Google.

Actually, if you think about the way Swarmia does things, or the way, I don't know, American Airlines does things, or insert any organization name here, there's great things, and they're all unique to that context. Google has a long history of being sort of data-driven and has a ton of research that happens. Certainly inside of Google, there's a ton of research that happens around engineering productivity.

DORA's research has always focused on the world, right? Organizations of every shape and size around the world. And so there's a really nice compliment there. I also think that, today, what we're doing with DORA is we're using this research to help drive and inform some of our product decisions when it comes to what are we building out?

DORA helps us understand what's happening out in the real world outside of Google, how real customers are putting these technical practices and so forth into practice. How can that inform the product decisions that we make so that we build products that will enable teams to get those great DORA outcomes, if you will?

Rebecca: If people know of DORA, they probably know of the annual report that comes out. So it just came out a couple of weeks ago. What were any surprises first, anything that you didn't expect to see you enjoyed?

Nathen: Yeah, for sure. This was our 10th annual DORA report. So that's a big milestone. That's pretty exciting. And you're right. It did come out a few weeks ago. It was late October when it came out. It's also super refreshing. Now that the report is released, when people ask me, “When will it come out?” I can give them a definitive answer. Up until that moment, I could not. So, that's really, really helpful.

But this year, in a surprise to no one, we had a deep dive into artificial intelligence. We also had some of the sort of traditional things that you might think of from DORA – software delivery performance, things along those lines. We also looked at platform engineering and the overall developer experience, as well as transformational leadership.

There's so many great topics in the report. Maybe I'll apologize in advance to anyone who hasn't read the report yet. It's about 120 pages long. It'll take you a minute to read it.

Rebecca: There's a great executive summary, though!

Nathen: Yeah, there is a pretty good executive summary. I would definitely recommend you start there and then decide where you want to dig in deeper. I think there were some really interesting and somewhat counterintuitive findings this year, specifically when it comes to things like artificial intelligence.

We're seeing, no surprise, a growth in the adoption of AI. And we're seeing some really good impacts of AI and that adoption, specifically for individuals. You're seeing things like Individuals reporting more time in flow, higher productivity, higher job satisfaction. That's all really good. And even from looking at an application level, we're seeing things like higher quality documentation, less code complexity, things along those lines. Even faster change approvals. All of that is really good.

And when we think about how do those capabilities or conditions impact software delivery performance? A team's ability to deliver change? Historically, we've seen that all of those things together make for great conditions for high-quality and fast software delivery. Unfortunately, that's not what we're finding with AI, right? So, with further adoption of AI, we're seeing software delivery performance drop off a little bit, both in terms of throughput – so how much change are you able to move through the system – but more significantly instability, stability of those changes.

So when you push a change through, are you likely to have what I call an “Oh, expletive” moment? You know, we've all had those before. You push to production and someone shouts out an expletive. Yeah.

Rebecca: Yeah. So, does the report get into at all why? What's happening there? And I'm also curious, we say AI and there's everything from AI bots that will pre-approve your pull requests to AI that's writing code for you and everything in between. Claude and I are best friends. Right? So, but yeah, I'm curious, when we're talking about AI, are we getting any more specific than just AI? And, especially I'm interested in the throughput. The quality or the stability is not a surprise to me. I think that's not unexpected in my head. But the throughputs really interesting. So what are you finding about why?

Nathen: Yeah, so in terms of what do we actually investigate. we ask a bunch of questions about artificial intelligence, how you're using it. At a very high level, there's kind of two ways that you might think about AI and how we asked about them.

One: are you using it to complete your daily work, right? So are you using it? And we are finding, of course, lots and lots of teams are doing that. The top two ways that people are sort of relying on AI – again, in a surprise to no one – summarizing information and generating code. Right? So that's not really a surprise. Of course, there are other reasons why people are using it and other areas in which they're using it.

But the other sort of high-level category is are you building AI experiences into your applications, right? So something that maybe a customer of your application or a user of your applications may be interacting with a bot or some AI capability.

We kind of look at those as two distinct things. And we tended to focus more of our questions on the “how are you using AI in your daily work?” And in terms of why does stability go down? Why does throughput go down? What did we learn there? Unfortunately, our data can't really give us enough, right? And just speaking of our data, let me tell you a little bit about how we collect our data each year.

So each year, we go out and we run a survey. And we get thousands of practitioners and leaders from around the world, from organizations of every shape and size, to participate in our survey. That's our primary method of collecting data. But this year, in addition to that, we also have a qualitative researcher on the team who sat down and did a ton of interviews with people from organizations around the world. Those interviews really help us get a lot more context and color, if you will, than we can get from survey questions alone.

So while we can't really say – even with those qualitative interviews – we can't really say why stability might be dropping off or why throughput might be slowing down, but we have a bunch of different hypotheses. And this idea of a hypothesis is really interesting because, to me, it's great that we published the report. It's even more exciting when we hear feedback and we can share and test hypotheses within the community following that.

So some of our hypotheses are: one – if we just think about what do we know about how to improve software delivery performance? We know that if you ship smaller changes all the time, you tend to have better software delivery performance. It's easier to move a small change through your pipeline. If you have one of those “oh, expletive’ moments, it's easier to reason about that change and roll it back or push forward a hot fix, et cetera.

Well, as an industry, and as we see in our data, generating code is one of those top two things that we're using AI for. So if, all of a sudden, we're generating more code, perhaps what we're doing is going from small changes to larger changes and trying to push those through. And when that happens, like you said, we shouldn't be surprised. Throughput is going to slow down. Stability is going to fall off, et cetera.

Now we can't say definitively that that is why. There may be other reasons. Another hypothesis that we have is specifically around that quality or the stability, maybe some of the changes that you're making. And this, frankly, could impact throughput as well as I'm thinking about it, but maybe some of the changes that you're making within your code base that AI has suggested to you, perhaps the models don't have a good understanding of your organization's norms and policies and practices and sort of how you write code and maybe even what APIs are available to you within your organization. It's giving you more generic suggestions. So those generic suggestions maybe feel like they work, but on review, maybe they get kicked back, which would slow down that throughput, things like that.

Rebecca: Yeah, that's what I was wondering if it's just that you'd have people reviewing more bad code. And you know, AI code is AI code. I have had AI make me whole little mini-applications. That's because they're for me. But yeah, I can also imagine. I would never want anyone to code review any of the code that I have written with AI, because… “make me a React app!” Very easy and I don't ever even want to look at the code that it used to do it. I'm sure it's concerning in a variety of ways. So yeah, I can definitely imagine that that code review step is suffering.

I'm curious, so those hypotheses seem easy to test with quantitative data. You mostly have access to qualitative data. How do you go about testing hypotheses like that?

Nathen: Yeah. So a lot of the way that we do it– first, we may, in the next year, add additional questions about the size of your changes that go through. And while we're not pulling out a ruler, we're asking a human, what is the size of your change? We know that humans are pretty good. People aren't showing up to the survey to lie to us, right? And we care a lot about their lived experience, right? And that is something that matters.

At the same time, even if we were able to measure something… what is a large change? Is a single line of code a small change? Arguably, yes, but obviously, not necessarily. It could have tremendous ramifications on the code that you're moving through your pipeline. So it's a really difficult question to answer. We also, because we're part of Google, we can also look at what's happening within Google.

And as more AI Adoption is happening within Google, is the size of a change list that's moving through Google system changing? I don't know the exact answer to that question, so I can't say anything about it. But that is another way that we could test that. And then, finally, my role as the lead of DORA is to go work with customers and help them take this research and put it into their context. And so through those conversations and conversations with the DORA community, that's really how we start to test out some of these theories.

Rebecca: So I'm also curious, you're talking about small batch sizes. And after talking about all of this endlessly for the last two years, I keep coming back to Little's Law of how queues work. And this all seems very mathematically straightforward to me. That small change – you do all the things that we say, and you're basically optimizing a queue and its workers, right? That usually resonates with engineering leaders. When you use that language, engineers get it. “Yes. No, I would not have a worker that was working on three things at once. That would be ridiculous.”

But how have you found success translating that to the non-technical stakeholders inside of businesses to help them understand the interruptions and big changes and, you know, all that?

Nathen: I think it's very difficult to do because we work in complex systems with lots of stakeholders, our customers, the business stakeholders that we're working with and so forth. And everyone wants more. And we have great ideas and we want to ship those great ideas as fast as possible. And unfortunately, within organizations, we have this tendency to try to push more things simultaneously, right? Whereas we know from research, we know from experience, that if we can minimize the amount of work we have in process and really prioritize flow through the system, like completion of this item before we pick up another item.

In a larger organization, though, we aren't necessarily organized in that way or incentivized in that way. Right? A developer’s job is to develop features, the release engineer’s job is to release features, and then the operator’s job is to keep those systems running stably. And we try to disconnect those things, and that's where we lead into some of this friction. We've got product managers always coming up with new features and ideas, handing those to engineering who's, you know, and it goes on and on.

Rebecca: Of course.

Nathen: So I think that we, as an industry, as organizations, really do need to have some critical thinking about this and try to really dig into: how do we prioritize the work that we're doing, and set the right amount of work for the teams that we have and the goals that we have? Some of that comes down to visibility of work, and making that work visible. There's also a really interesting thing that we tested this year as well when we were looking into what is developer experience like, or user experience like, what's your experience like as an employee?

One of the things that we looked at was the shifting priorities of organizations, right? It seems like every organization is shifting priorities nearly weekly it feels like sometimes, right? We're always constantly shifting priorities. What impact does that have on people within your organization? And what our research shows is that it has certainly an impact on productivity. So, as you're constantly shifting priorities, productivity goes down. But even worse, it has a stronger impact on burnout, right? So if I show up to work today, and I don't know if my priorities are the same as they were yesterday, that constant shifting is driving additional burnout.

But Rebecca, you know what? There's something interesting. AI helps with this.

Rebecca: Interesting…

Nathen: Yeah. Well, okay. Maybe I oversold that a little bit. It's not necessarily that AI is “helping” with this. But here's one of the things that we did find. Many organizations are prioritizing AI and the incorporation of AI as a top priority. So not that the LLMs are helping us with our priorities, but this AI thing is helping us leader. AI is where we're focused.

Rebecca: And also it's so much more exciting to work on AI than to work on security, right? No offense to my security friends, but yeah.

Nathen: Yeah, because that's the other thing I've found. And again, I hear this a lot in the community. When I'm working with AI to help generate some code, it's kind of fun. It's fun! And it takes some of those coding challenges that maybe were just out of reach and makes them a little bit more accessible to me. And we see this reflected in our data. As AI adoption improves, so does your job satisfaction. So it's all lining up there.

Rebecca: Yeah. That's a really interesting angle because yeah, I find myself enjoying writing. I hardly ever write code anymore, except for fun, but I find myself enjoying it now because all the hard parts I can just make go away and it's just a creative endeavor instead of a mechanical endeavor, which is kind of cool.

Nathen: Right. And I mean, if we can, pulling on that creative endeavor versus mechanical endeavor moment is I think really important. Because I think that a lot of the current conversation around developer productivity kind of ignores the fact that developing code and engineering is, in fact, a creative endeavor. And so we can't just replace that with machines that just do the thing for us. I'm not a great engineer because I understand the syntax of the language. That matters. But what really matters is I have the domain expertise, and I understand who my users are, and I understand how to take a large problem and break it down into a small problem. It's going to deliver value, right? And do that over and over and over again. And this, again, sort of, reflects some of our findings. We find that the teams that are user-centric, that really understand why are they building this particular application or this particular feature, those teams tend to have the best outcomes as well.

Rebecca: So this is really the crux of my eternal pursuit. We know all of this. We have like ten years of data showing us all of this – that outcomes over outputs, focus on the user, small changes delivered frequently. We know all this and yet it's really hard for some companies to actually adapt to this. When you tell a company that you can't have three top priorities, that's hard for some of them. They've always had three top priorities.

And I think this is a really interesting thing about “developer productivity” too, is that so much of what affects productivity is happening at that cultural and leadership level. And not at the boots on the ground level.

Nathen: Absolutely.

Rebecca: I can't choose to only work on one thing unless you've given me permission to only work on one thing. Right? My team can't limit its work in progress if you're not giving me the air cover to say no. So, I'm curious. I don't know what my question is here, but what's the disconnect in your mind between lots of proven data and still people behaving badly?

Nathen: Yeah. I mean, I think a lot of this, as you were kind of alluding to, gets back to incentives and certainly leadership. And it's a constant reminder to us all that while, as practitioners on the ground, as engineers, we know the right things to do. And there are small pockets that we can improve. We can make small iterative improvements to the way that we do our daily work. But what we can't do is change how does the organization set up incentives? How do they prioritize work? How do they help us with that?

And so again, we look into leadership. We find consistently that things like transformational leadership really matters. And a transformational leader is one who sets a good vision, has personal recognition, has a number of other components. But what they don't do is tell you how to do your job or tell you what to work on. There's a lot of team autonomy and there's alignment around the right goals and incentives and so forth.

And I think this is just really, really difficult. And it becomes even more difficult, maybe, as an organization becomes more “data-driven.” Because it's very difficult to measure things like I'm the developer who's responsible for pushing the button that will deploy the change. How confident do I feel in pushing that button? How safe do I feel? What are going to be the implications if I push that button and we have one of those “oh, expletive” moments where something goes bad. What's going to happen to me? We can't measure that.

We can only get there and answer that question through experience. And that is, again, a place where leadership and culture really, really matter. And it's so important. Now we know this, again, to your point – we know this! So, why can't we change it? Why is it? Why is it still, why is it constantly year in and year out, we're talking about these things as the major challenges?

I think it's kind of human nature. I think that change is difficult and changing things that are very slow to change and very hard to measure, those things become even more difficult to change. And frankly, as an industry, for the past number of years, not only are we ever-shifting priorities, but we're also shifting a lot of who are the players on this team. Whether that's through attrition or layoffs or executives change – all of it, there's constant churn of the people on our team as well. And I think that has a big impact.

Rebecca: Yeah. As far as measuring, but also just as far as propagating culture.

Nathen: Exactly. A new leader comes in and wants to make their stamp and change things. And, look, I certainly don't want us to be in the place where, “well, this is how we've always done things. So we're going to keep doing them this way.” We have to push against the status quo, but we also can't swap out the status quo every six months.

Rebecca: Well, you can. You just can’t then ask, “why aren't we productive?”

Nathen: Right. Oh, of course. Of course.

Rebecca: And I think that's the thing too, is sometimes it is going to be the right thing for the organization to do something incredibly disruptive.

Nathen: Yes.

Rebecca: We just got to understand that we're doing it right. And that the estimates that people made when they had ten people on the team are probably going to be a little different now. So I think leaders often underestimate how long that impact lasts. That it's not just a week and everybody's better, especially in the case of layoffs.

Nathen: That's right. And there may also be something to– the impact takes a long time, as does the feedback cycle. Right? You don't find out tomorrow that that was a bad change. It takes time, right? And frankly, if you're the one that's made the decision, you kind of have this inherent disposition that it was the right decision. And so it takes even more feedback for you to understand “oh, maybe. Maybe that was a mistake.”

Rebecca: Oh, we could– yeah, six more podcast episodes here. We can just talk about reorgs for 40 minutes. But I wanted to come back to talk about something that has been all over LinkedIn, and maybe other social media as well. I think this is also this interesting cultural thing that we didn't really talk about when money was free and suddenly we're like, who are the engineers who aren't working?

And number one, insistence that there are engineers who aren't working and number two, efforts to kind of figure out who they are. And so, in the last week or two, this person posted undocumented research – to be clear – saying that they found that 9.5 percent of engineers are ghost engineers. And there's no evidence of their contributions whatsoever.

So this raises all sorts of interesting topics in the developer productivity space, because certainly you could read that and be like, “I'm scared they're going to think I'm a ghost developer.” And I didn't know people were out looking for ghost developers, but now, you're bringing in this new thing called Swarmia and like, what's next? But also, I'm curious: what is productivity? And can you get to the point where you can say this person is not productive?

I think I'm personally skeptical. Yes. I do believe that there are people who are collecting a paycheck, doing very little work. I believe that that exists and I want to be very careful about firing them. Because there's so many different shapes of contributions, but yeah. How does DORA think about: what is a valuable act by a developer?

Nathen: Yeah, I think it's a very difficult question. And I think that the “9.5 percent” of ghost developers, or ghost engineers… it also kind of gets back to just that. What is the measure of success? And maybe what you have is 9.5 percent of coders are ghost coders, right? And we know that because they haven't contributed any code. But the job of an engineer is not simply to sit down and write code, right? And so that's where it gets really sort of interesting and maybe amorphous. And, again, a thing that we need to think about. Your senior engineers, the principal engineers, they might write zero code throughout an entire quarter potentially, right? Are they having no impact on anything? Of course not. Of course not.

And so I think it really comes down to this idea of understanding what do we mean by productivity? And are we measuring things at the individual level, at the team level, at the system level, and so forth? And with DORA, most of our metrics – certainly the software delivery performance metrics, those four keys that people know and love – those are measured at the level of an application or service. So it's not even not even necessarily at a team level. It's certainly not at an individual level.

Now, to your point, can we have people on the team who are showing up taking a paycheck and not actually doing any work? Of course, that can happen. And I suspect that it will be very difficult to identify those people correctly through telemetry and system data alone. And this is where we really need to lean on local managers, right? That individual's manager or leader and teammates would be able to identify that, right?

Putting a bunch of data into a system and figuring out who are my low performers, you're probably looking at the wrong data. You are going to make mistakes. And frankly, maybe sometimes we're okay making mistakes. Which can be scary, especially if you happen to fall on the wrong side of that line.

Rebecca: I am thinking of somebody who I obviously will not name. Their existence at the company is incredibly valuable. The things they know about why things have happened and what that line of code does. And, you know, “don't do that. We tried that before.” And I think that's another missing piece, especially at the more senior levels is I might pay that person just to not leave. And to show up to meetings when I ask them to, right? And to be available to troubleshoot an incident if one comes up. And so I think that's another interesting thing is that obviously it's very easy to put telemetry around, like, concrete actions.

But it's hard to put telemetry around the value that the individual is actually providing. So interesting because a lot of the layoffs affected middle-level, like mid-level managers who might have been in the best position to notice these things. Because I agree that if you have somebody who is doing that and the manager doesn't know, that's not the person's fault. That's the manager's fault. It's kind of what I feel.

Nathen: Absolutely. And for that principal engineer that you were mentioning, or that very senior person who seems to know everything, part of their job should be to remove themselves as that bottleneck, right? And we should be encouraging that, right? Because having all of that sort of institutional knowledge and tribal knowledge locked up in their head is not sufficient, right? And it's going to lead to bad outcomes for them and the organization.

Rebecca: Yeah, I'm certainly not celebrating the hero who you have to call, you know, once a month to flip the right switch. I think that's bad. And there is also like the knowledge of “we tried that before,” that I think is a lot harder. Well, I don't know. Now we're getting into like the accessibility of documentation for engineering decision-making.

Nathen: I will say, this was the fourth year that we looked into internal documentation and the impact of having better quality internal documentation. And every year, I just get more and more astounded. It's so, so important to have good internal documentation to document things like what you just described. Having those good decisions documented somewhere where we can use them. But just written down, it doesn't make it “quality” documentation, right?

It's also interesting because one of the things that we found, the increased use of AI improves documentation quality. But there's a really interesting way to look at that, though. Because the first the first question might be, well, what do we mean when we say documentation quality? Right? How do you know that a document is good?

And I'm just going to pull up just a couple of questions. So the way that we assess documentation in our survey… We put statements in front of the participant and we ask them to rate on a scale from strongly disagree to strongly agree. And these are the four statements that we use this year:

I can rely on our technical documentation when I need to use or work with the services or applications I work on.

It's easy to find the right technical document when I need to understand something about the services or applications I work on.

Technical documentation is updated as changes are made.

And when there's an incident or problem that needs troubleshooting, I reach for the documentation.

So none of those are about quantity of documentation, right? We're asking things about, can I rely on it? Is it easy to find? Is it up to date? And do I reach for it, right? So that's how we assess quality documentation. And we say that, as AI adoption improves, so too, does your documentation quality.

So then you have this follow-up question, is the AI improving the actual underlying quality of the documentation? Or is it just improving those characteristics? It's easier to find. It's more accessible. I rely on it more.

At the end of the day, the answer is probably yes. And also, it doesn't matter. It doesn't matter the mechanisms by which it's improving. What matters is that it's improving. And then it has knock-on effects. You know, it helps, it helps accelerate and amplify other technical capabilities like continuous integration and automated testing, but it also helps with things like being more user-centric and understanding who are the users of this application.

It has tremendous value and benefit across the organization. So that's my little sidebar on documentation.

Rebecca: Well, I'm going to keep going. Last question, I swear. It was interesting to me. How did – and maybe you know, maybe you don't – how does quality documentation correlate with size or age of a company?

Nathen: Uh, that's a good question. I don't think that we looked at that specifically. The way that we model our data, it will have been taken into account. And if there were differences, they would have kind of popped in the model. We didn't report on that, so I can't say definitively that nothing popped, but I don't think–

Rebecca: You would have recorded it if there was something.

Nathen: If there was a big thing, we would have reported that for sure.

Rebecca: So, what do you say to an engineer who don't want this coming near their world, because they're afraid it's going to be used as a performance management tool or similar? And they could be right! It's hard to know. But yeah, I'm just curious, the idea of measuring engineers has become less toxic among non-engineers, but still is perhaps a point of resistance among engineers. So, yeah, I'm curious what you see about that and what you say about that.

Nathen: Yeah. I think that there's a couple of different things. So first, if we're speaking with the engineer and they're concerned about measurements and how those measurements might be used against them. I think the first thing to do is to understand exactly, how did they reach that conclusion? Maybe they have experience within this organization of measurements being used against them. All right. Well, I can't argue that. I can't tell you that that's not your lived experience.

But also, I think that when it comes to some measures, the measures are actually there to help the engineers and, and maybe even those measures are only meant for the engineers. They aren't meant for leaders and so forth to see, right? And you might even argue that about DORA's software delivery metrics. Does your CEO care what your deployment frequency is? They probably shouldn't. What they care about are the things that that can lead to, right? Better customer satisfaction, more customers, more revenue, more profit, et cetera. Those business metrics that really matter.

And what DORA has shown us over the years is that as software delivery performance improves, so do those organizational outcomes. But I would argue you shouldn't trust DORA. Go prove that yourself. Prove it out within your organization. And when I say that, I say that to mean, as a team, you should understand what are the business goals we're looking for and how is our software delivery performance going or other metrics? Are they moving together? And you want to be able to visualize them next to each other so that you can see that and really understand that.

And then you can go to your executives, your leadership team, and say, “Hey, tell me what business metrics you care about. I'll help relate how our engineering excellence is driving those metrics” and have the conversation at that level. So one, I think those measures are for the team for self-guidance to drive their own internal improvements that they want to see on the team.

Second, I think, and we've alluded to this a couple of times already, measuring an individual is probably the place where you're going to run into the most reason for fear and probably the least accurate measurements. We have to move to a team level or an application level to really understand what's happening.

Rebecca: So you said, no, the CEO probably shouldn't be looking at DORA metrics. Let's imagine for a moment that you have a CEO who is looking at DORA metrics because this has happened to me. Not exactly DORA metrics, but same thing. Maybe they're listening. What's the right cadence to be looking at these metrics, at the team level, at the organization level? And is it every week?

Nathen: That's a great question. Okay, so first, I think that you have a couple of pitfalls that you've kind of alluded to there that I just want to point out. First, DORA measures those software delivery performance metrics. And just to be clear, DORA has tons of metrics. A lot of times, we talk about the four. So when we talk about those four software delivery performance metrics, we always measure those at the application or service. In fact, the survey question starts with “for the primary application or service that you work on.”

So when an organization tries to roll those up and report organizationally, “here are our four-door measures.” Don't do stop. Stop doing that. That's not how they're designed. You're mixing apples and oranges. It's not okay. If you want to roll something up, or maybe worse than rolling them up, is doing team comparisons. Team A has these four; Team B has those four. A is clearly better. No, stop that also.

If you want to compare teams, here's my suggestion for comparing teams. You really care about the change over time. So, team A has improved their DORA metrics by 15 percent in the last quarter. Team B has improved their DORA metrics by 23 percent in the last quarter. Team B has improved more. That's great. Now, at the start, Team B maybe was deploying once a month, and Team A was deploying once a week. You'd think Team A is better, but Team B has improved more, and that's the thing that we should be celebrating. DORA really leans into and wants to promote a practice and a culture of continuous improvement. We want to help teams get better at getting better. That's the goal. That's the goal.

At the same time, I have seen, I'll give you one example of rolling up metrics that actually was helpful, not harmful. So, in an organization where they have a bunch of different sort of pillars across their organization that all kind of work semi-independently, they were able to roll up some DORA metrics and say, “look, this team over here, they're deploying more frequently and are more stable. This team over here is deploying less frequently and is less stable. So that tells us that even in our organization, we can probably deploy faster and improve our stability.”

All right, great. With that in mind, and not picking on any one team or one business unit or whatever, we can then go to that business unit and say, all right, let's create a program that's going to help you improve that. We're not going to define exactly what that is. You know what you need to do to improve it. But that's a healthy way I've seen those used.

Rebecca: So probably not a good idea to tell all your teams they need a cycle time of one day, right?

Nathen: Right.

Rebecca: How do you set goals when you can't set a goal like that?

Nathen: Right. I think, we have on dora.dev, we have an article – I think it's in the transformational leadership article – there was a CIO at a big financial institution, a big bank. And this CIO, or maybe CTO, I don't remember which middle letter it was, but this particular individual set a goal for all of their engineering teams. Their goal was to double half and quarter. Double the frequency with which you release, cut in half the number of low-priority incidents, and cut in quarter the number of high-priority incidents that you have.

Now that singular goal can be used for the mainframe team that's responsible for the mainframe applications and for the retail banking website team. The numbers that they're both shooting for are going to be very different, right? But when you set proportional goals like that, now we can all be working towards the same goal. And what matters is the progress that we're making, not the actual sort of raw metric that we hit.

Rebecca: I think that goes to another point of when your cycle time is one day, that may not be a thing that you need to be spending a whole lot of time on. Or if it's five hours, there's some number where it's maybe not the right thing to spend time on anymore. How do you think about that?

Nathen: Yeah. So the way I think about that, I think the DORA metrics provide this really nice compass for sort of how we're doing and where we're headed. And when those numbers aren't where you want them to be, you don't get better at them by thinking about those numbers and focusing on them. You have to look at the capabilities and conditions that lead to better performance. And that's what a lot of our research is about, right? Why we research documentation and AI and continuous integration and so forth, and culture. Go look at how do you improve those things.

But the whole reason DORA exists and the reason that we focus on software delivery performance is because oftentimes, in an organization, the ability to get change into production is a bottleneck. Now, if you're working on an application team who can deploy on demand and those changes land stably most of the time, maybe software delivery performance isn't your bottleneck anymore. So maybe stop thinking about– not stop thinking about that, keep those practices going, but maybe it's time to turn your attention to somewhere else.

Maybe you're not hitting your business goals because of a reliability problem with the running services. Maybe you're not hitting your business goals because there's not a product market fit with the application that you're building. Spend your resources, spend your capacity investing in those things that are the bottlenecks. This has always been DORA's message. It leans a lot on the theory of constraints, look at your system, find the thing that's holding you back, go alleviate that constraint and then congratulations, you've got a new constraint. Go work on that one now.

Rebecca: Forever and ever.

Nathen: And ever. Yes. And you're never done. You're never done. I'm sorry. It is a journey of continuous improvement.

Rebecca: Yes. Well, Nathen, this has been, gosh, I could keep talking to you for quite a while about all sorts of topics in this space. I'm going to go give the report another read because you've put on me some interesting things I want to dig into.

Nathen: Well, thank you so much, Rebecca. This has been super fun. And I've alluded to it a couple of times, but these conversations continue throughout the year at dora.community. So if you haven't already, please do join us over there. We've got researchers, practitioners, leaders, all trying to answer these same questions that we're struggling with.

Rebecca: It's an amazing resource, and the report is an amazing resource as well. So yeah, thanks for doing it. I know they pay you, so it's probably not a hardship to do it, but still, thank you, Google. I'm glad it's happening.

Nathen: Yes, indeed. Thank you, Google.

Rebecca: Yes, thank you. All right, Nathan. Thanks a bunch again. I hope we can catch up again soon.

Nathen: Sounds great. Bye bye.

Rebecca: And that's the show. Engineering Unblocked is brought to you by Swarmia, where we are rethinking developer productivity. Catch you next time!