Growing up DevProd: 5 years of improving efficiency at Stripe with Ainsley Escorce-Jones

Rebecca: Hey Ains, how's it going?

Ainsley: Doing pretty well. How about you?

Rebecca: I'm excellent. I'm super excited to have you on the podcast. Just to talk about your experiences in, not the early days of Stripe, but in the earlier days of Stripe when the whole developer productivity initiative really got its start there.

So yeah, just really interested to talk about that, your part in that and, lessons that you've learned over the years in those adventures. So tell me a little bit about what you do today, but then I'd love to hear how you ended up in this role of tech-leading the developer productivity organization.

Ainsley: Yeah, for sure. So today I am the tech lead of Developer Infrastructure at Stripe, which was a very recent rename. For all intents and purposes, it's still mostly Developer Productivity is what we do and what we focus on and I've been doing that for a bit over the last two years. It's been a long journey. I've been at Stripe for five years. I've pretty much reported to the same person and been on Developer Productivity for pretty much that entire time. And it's grown from pretty much we all sat at the same lunch table in San Francisco to like geographically distributed. I'm gonna go and hang out with some people in Europe next month. Just all the time zones, all of the different concerns that we have now.

But yeah, to go back to when I joined Stripe, which was 2018, I actually almost ended up on a product team at Stripe, working on Connect. That was kind of the first team that I had a manager chat with to decide where I was going to land. And it just so happened that through a friend, I found out that a person that I knew of from Facebook, Paul Tarjan, was working on Sorbet, which is our Ruby type checker at Stripe. And I was like that sounds like there were a lot cooler problems in that domain, than perhaps there might be… Not to talk ill of any product… Nothing against product people.

Rebecca: Nothing against product. We're glad they exist too.

Ainsley: But also, at the time I was working at a startup in London where I’d de facto become the developer productivity team. And so at the time, that entire company was probably in about the hundred person range or a hundred engineer range, which it's funny now thinking that that's the size of the entire developed productivity team, plus or minus, at Stripe today.

But it was very much like I was there also working on product. And at the time I just realized that I work with a bunch of really, really smart, really, really awesome people. And we would spend like 20 minutes or something waiting on a build, waiting on some feedback. Our deploys were super, super slow.

We didn't have code search back then at the company, and I was just like, if I carve off some extra time in my day to just tweak this build, I just unlocked productivity for the other, like 99 people that are sitting around me. And that seems like a pretty good use of my time.

So I kind of just became the build guy. I started talking to Sourcegraph to get us Sourcegraph internally for code search. A few months after that I think we had like three or four people dedicated full-time to what we called Engineering Velocity there.

Rebecca: You did an internship at Facebook, right?

Ainsley: Yep.

Rebecca: How much do you feel like that kind of… I know they were huge when you were there and they're huge now, but how much did that experience inform your thinking about productivity?

Ainsley: It definitely massively shaped my expectations, I think, of the infrastructure for engineers to be able to move fast, right? Like they kind of coined the whole move fast and break things term. But I think it was really a core part of the DNA of how things were done. And I think the thing that surprised me, or actually I think surprises me now, really Facebook was my first experience of like Big Engineering.

Before that I was a bedroom software developer, right? The idea of I'm going to like SSH into the box and edit these files, or like FTP some PHP files into like the ether was like my deploy strategy. So I moved from that to Big Engineering, but it didn't actually click to me until I'd seen a couple of other different companies, how the fact that I didn't feel a massive drop off in my productivity when I joined Facebook, there was a bunch of work that went into that. To me it just felt like this is how things are. I'm an intern here and I think on my first day before onboarding had finished I had a change up for my manager.

Because I was bored during onboarding and I sent him this PR and he was like, “Oh, I didn't know they'd gotten to the part where they taught you how to use those tools yet.” And I was like, “Oh no, they didn't. I just wanted to fix this.” And I just took all my intuitions from like, it's just me or like four people working on this. And I started doing productive work at a company that had thousands of people distributed across the globe. And so that had unfairly perhaps set my expectations for what engineering was going to be like.

But yeah, I think both a really smart colleague of mine at Stripe, called Danne, she kind of coined this concept of like L2 weeks when thinking about like, if you've got a well scoped problem, add a button to the dashboard, maybe it's connected to the API and calculates some number and sends it back to the client, right?

That should be scoped honestly to an L2 — so at Stripe that is after new grad, but before we expect you to start scoping out your own things and making your own design documents, just very much someone who's competent in execution — that you should be able to measure tasks in like how many hours, how many days, how many weeks does it take someone who is capable of independently executing to do something.

And I think Facebook just did a really good job of so many things fell into like the L2 days category. And so they were able to utilize me quite well as an intern. And I've always held that as like my gold standard of what it should feel like.

Rebecca: It's interesting, and I don't know if we even have time to talk about this, but it's interesting how if you invest in productivity, you can have L2s be highly effective. If you don't invest in productivity, those L2s might struggle and you might find yourself just hiring L3s instead because they have the context and the experience. But L3s cost more than L2s, and don't necessarily want to be doing work that really an L2 should be able to do.

Ainsley: I think it is also, I guess just very true that you actually get more out of your L3s and L4s. I don't necessarily think there is some hard cap to productivity where you become an L5 and you're just finished being more productive and there's nothing we could possibly do.

I really do think the question you should also ask yourself is, “OK, we’ve hired someone as a senior engineer, staff engineer at our company: How quickly can they become productive? How quickly can they gain that context and start making the changes that they want to make?” They might be making broader changes. They might be making more like abstractions or something, but I think there's a valid question of it. It is almost certainly true that if you make it very easy for your L2s to get stuck — your L5s need to ramp up too, and you're paying them a lot more while they're ramping up. You might want to really figure out how to use them effectively.

Rebecca: So when you got started with this at Stripe just give me a lay of the land. I know there's a large number of engineers at Stripe today and I know there were a lot fewer back in 2018. What does Stripe look like in 2018 and how… Was this an urgent problem? Was this an opportunistic thing? How did it start?

Ainsley: I think it was important at Stripe at the time, because Stripe had hired a lot of good talent from Facebook, specifically in, this was after the team was called Product Infrastructure. It was actually called Developer Productivity at the time, but the same group that was focusing on both abstractions — which is, you know, hey, we should write some libraries in this language. Stripe primarily writes Ruby to make it easier for people to do simple things. Save thing to database with validation, et cetera.

And then kind of the higher level things on top of that: we should have remote development environments. Stripe back then was a user of like VMs on the laptop, Vagrant, you know, type vagrant up, wait some long period of time you get a VM. And we wanted to move that to more like, you SSH into some box in AWS cloud that has all the things running and you're synchronizing your code between those two machines.

But because at the time, the real effective lead of developer productivity in many ways was Paul, he came with that same attitude of, this is important because so many decisions at the time were made optimizing for engineering ergonomics.

And so the team was small and the company was relatively small, like definitely sub 1000 engineers. Pretty much everyone in one contiguous building in San Francisco at the time. But the idea that we could enable a bunch of these product teams to move faster by dedicating ourselves to solving the friction for them was just deeply embedded in the DNA of the team.

And not long after I joined, Scott MacVicar also joined from Facebook to lead Developer Productivity. And so that has stayed a core part of the DNA to say that we can build tooling and abstractions at Stripe that multiply the effectiveness of engineers.

Rebecca: And you were probably yourself a lowly L2 back in these days, right?

Ainsley: I was.

Rebecca: Nothing against L2s. We love them. But so you may not have been privy to kind of conversations going around, but like, do you know anything about how it was decided, like how much you invest in this versus like, you could always be building product instead. So yeah. I'm just curious. Were there, conversations that you were around for, about how Stripe would invest?

Ainsley: On the one side, I definitely was not privy to like the swath of conversations that I'm privy to today. I'm very much more deeply involved in making those decisions today, and I'm sure that there were conversations like that happening at the time. But like I was saying before, one of the reasons I joined Stripe is because I saw them writing. At the time they had like two blog posts out that I'd seen: one was about Sorbet and one was Nelson Elhage had written a post about how we parallelize our test running across, at the time it was like 40 machines. It’s way more than that now. But it was clear to me that not only did this company invest in this area that I had managed to convince the previous place I worked at to invest in, but they were also willing to talk about it publicly.

I'd love to go back and calculate as a percentage of the company, how big developer productivity was. I actually think it was quite big as a percentage of the company back then. And it just seemed by both how the company talked about itself publicly and also the kind of projects we were working on internally it was a pretty major priority.

Rebecca: So we worked together at Stripe. That's why sometimes I'm going to say “we” by mistake here. I don't work there anymore, but we worked together and I had worked at a company that size, but it was a little bit of an older company. And that's one of the really interesting things to me in this space is like how the moment in time that critical decisions were being made, like how that shapes you know, 10 years later you are living with the consequences of those decisions and undoing them can be really hard. So I'm curious, going back to you're the new L2 in 2018, what sort of things were you working on?

Ainsley: When I joined Stripe, I joined a team called the Code Team, which didn't exist for a while and exists again now, but fundamentally our remit at the time was code storage, code intelligence, code search. So that was really like how are engineers at Stripe making commits, getting code reviews, how are those reviews getting to the right person?

And also when you're new, you've like cloned the codebase for the first time. Like how are you able to explore that if you see some kind of error message, some kind of stack trace, how do you know where the right piece of code is that you need to do to make a change? So I came in in an L2 frenzy of just going in and starting to make pull requests to CI bot, which orchestrates a bunch of processes around code changes, including kicking off builds, making sure the reviews are assigned, et cetera.

I went off — so we use Livegrep another tool, open source, actually by Nelson, which is our internal code search tooling. So I started off making some changes to that, which was to try and marry that with Sorbet so that people could get kind of code intelligence with their code search.

Did a bunch of build optimizations at the time. So I was very deeply in like IDE tooling. We use … extending GitHub Enterprise to make it easier for people to navigate and get their code reviewed. And then code search stuff as well.

Rebecca: It's so interesting to me again, this whole time thing and how it affects the decisions that you make: Like are there things that you worked on then that you're like, “Oh my god, I would never… I would just buy something to do that today.”

Ainsley: Oh, 100%. So I’d just come off of a Sourcegraph contract negotiation at the place I worked when I lived in the UK, called Improbable, left that company, came to Stripe, and then immediately walked over to Sourcegraph's office in San Francisco.

And I was like, “Uh, hey, we built our own code search tooling.” But there were certain things, so for example the way that we have Livegrep set up at Stripe, it would periodically index code, which meant there's like a 12–24-hour delay between you writing something and it being searchable very easily by everyone else in the company. And Sourcegraph have an entire team dedicated to just like “this should be instantaneous.” They had an entire team dedicated to code intelligence should be built into the product. You should be able to hover over a piece of code and get all the context. That kind of important.

I think there is something to be said, like did these things exist at the time? And obviously VS Code is what we've all standardized on pretty much now in terms of like open source editors that we kind of call extensible, but it is very easy to forget that the reason that they didn't all use VS Code at the time is because it just didn't exist.

And so similarly, the reason that we have Livegrep at Stripe is like, there just weren't great code search tools that you could get off the shelf at the time. The downside is just, this is life of a growing company. It's going to be true of companies founded today in 10 years time.

It's just the decisions you make then sometimes are hard to change. Or even then the products that come off the shelf are just targeting a different segment to you. And so someone comes and they make a code search product or they make a new IDE. And it's great for the startups that are starting in that present day.

But you are coming along with your tens of millions of lines of existing code. And you ask this company that maybe at that time has like 20 people and you're like, “Hey, we would love to switch over to your product.” And they're like, “Absolutely no way.”

Rebecca: “Please do not.”

Ainsley: Yeah. So I definitely think that's one of the things that we did. It took us quite some time to standardize around VS Code as the way that we wanted to, the platform that we want to build experiences on top of, we're always going to have domain specific experiences. That's just, every company has its own special domain, but even deciding like how much time do you want to spend writing about things that actually affect your own domain versus just like generic IDE things or generic code search things.

Rebecca: A lot of these things that you're talking about, I was there. I understand how they are so impactful over an engineering organization the size of Stripe’s. And I'm also now in my current role talking to a lot of companies that have a hundred engineers like Improbable back in the day.

When do you think like this sort of investment… I think we can agree that if you have an engineering team of 10 this is probably not a thing to dedicate one person to. How do you think about that? What's your sort of calculation for when this becomes a thing worth doing? A thing worth not building product for? Basically, like they could have made you build product and I want to meet Ainsley, the product developer someday. I think that would be fascinating.

Ainsley: I'm sure some of this stuff still exists on the internet somewhere.

Rebecca: But how do you think about when it's worth doing this kind of improvements?

Ainsley: Yeah, so my first thought here is actually I think you can get a long way by embedding in the culture of your company at a certain size that there isn't such a clean delineation between like productivity work and product work. I think if you accidentally set that expectation of the norm, you're gonna need a productivity team way sooner than if you actually say like, “Hey, if you are building this thing, or if you're building this piece of your product? Are you building it in… “ Once again, I don't want people to go too far into like, “let me make everything an abstraction.”

But, “Is this somewhat of a reasonable component?” Like, “Have I contributed to a tragedy of the commons and now I'm going to walk away and move on to the next thing? Or am I going to just like actually think a little bit about the longer term ramifications of my change?”

So I do think there is some really early on cultural changes that you can make where you can just say like, “Hey, the productivity of the people around me isn't someone else's problem. It's all of ours collectively.”

That being said, as your company grows I do agree. At the size we're at now, we definitely tend to think in terms of fixed percentages of engineering working on productivity. But there is a lower bound where that probably ceases to be an easy calculation that you can make.

But my sense is once you are crossing, honestly, like 50 engineers kind of range of a company — definitely before you reach a hundred, seemingly after you reach 10 — there are a bunch of things and honestly, I think there's also a question of when does it become a full-time job?

Because I think, as we were saying before, there are these kind of SaaS solutions that you can pull off the shelf that do things that took people to build like five years ago. And so instant previews… I think Heroku obviously had this for quite some time, but now it's built into Vercel, it's built into Cloudflare Pages, et cetera. You know, CircleCI like makes, you just drop a YAML file in your repo and generally it works for the most part. There are a bunch of startups now that actually do really cool orchestration of like the PR workflow, you declaratively define how things should be signed off or whatever, and then some bot appears and starts commenting on your PR, right?

All of that stuff is stuff that we had to build. And it sucks because now we have to maintain it forever. But I would say it probably now requires less investment than it did in in the past to get some of those benefits. But I still think having clear… At some point, someone who actually cares a lot about that stuff and is willing to be like, “Hey I just spun up this Docker container with this open source product that allows us to get instant previews,” or whatever.

I think yes, if you haven't got someone who cares about that stuff by the time you're in the 50 person company range, I think you're missing out on potentially buckets of productivity. I think you do have to get much bigger now in like just the present environment before you have to start really custom building out things. There's just so much you can get away with by just doing a bit of research for what seems to fit with your workflow.

My answer here is probably a bit out of date because I'm so deeply immersed in the things we do and don't have at Stripe. And I definitely know that there is so much out there that has changed and that is available now that just wasn't. But I have the sense for what I would do if I were like starting a company or what I would do if I had like… Well, I do have side projects, so I kind of know the things that I use there.

Rebecca: Yeah, it’s wild. When I started at Swarmia… just yesterday, I opened a pull request for a blog post that I wrote and boom 10-30 seconds later, there's a link to the Cloudflare pages preview. And then, Renovate is running and doing automatic pull requests, they upgraded the whole repo to Node 18 the other day.

And at Stripe, we had a person spend a long time doing those sorts of things, but of course we did because it's millions of lines of code, not something nice and tidy and small.

Ainsley: Right? I think there's a public blog post on like the Flow to TypeScript migration.

It’s like so much work if you started using Flow years ago and have millions of lines of code written in it. But yeah, there's so much cool tooling now. Like in one of my side projects, I have this bot that just comes along and just sends you a PR bumping the version of certain things.

And if you have that it's just this really cool thing of like, sure, you're getting every kind of minor version bump just automatically coming into your repo. And then you get this slight like major version bump and you have to make these small changes. But it's built into your workflow that you're doing this incrementally and that it's happening regularly.

And if you start with that from day one, it's much harder for you to get into the stage where you're like, “Oh god. They just released like, you know, Ruby 3.X and they changed everything.”

Rebecca: Stripe was the first place I ever worked with a monorepo. And it's so funny because when I was at Indeed there were a bunch of ex-Googlers who were like, “we should have a monorepo, we should have a monorepo.” And you know, people who had never worked with a monorepo, including me, were like, “I don't get it.”

Why would you do that? And now last I heard Indeed is still talking about maybe we should figure out how to have a monorepo. And I was talking to some of my former colleagues there and saying like, “I've lived it now. And these are the wonderful things and these are the, like… have you ever had git pull take two minutes?”

Let's talk about that. But yeah, so that's an interesting example of a decision that… Was it a decision or was it just a thing that was kind of decided by, no one deciding? Curious to talk about those sorts of… especially at a 10-12-year old company. Can you talk about like how those formative decisions have really influenced the paths that are available to you today and the problems that you have to deal with today?

Ainsley: I will start off with just acknowledging that if someone is to join Stripe after listening to this podcast, that we don't actually have a monorepo per se. We have several very large repos. Not a huge amount. And directionally, what we actually do want at Stripe is a single like actual monorepo as opposed to a small number of mega repos.

But I don't want anyone to get the wrong expectation of what they will see if they turn up.

Rebecca: Fair, fair.

Ainsley: But yeah, to start with my milquetoast rendition of this answer, which is that honestly, it doesn't matter that much, be it a polyrepo or monorepo strategy if you invest sufficiently in the tooling to make that experience good.

I've seen some companies that have made an intentional polyrepo choice which necessarily gets you either code sharing or schema sharing kind of issues that come along with those. Or like, “How do I atomically update this library across all of these different repositories?” et cetera.

Or how do I keep these versions of these libraries in sync? And I've seen some companies not build great tooling around that, and people do not enjoy the experience of the polyrepo. And sometimes they come and say, “The reason that I want a monorepo is so that I can make these like large cross-cutting changes.”

Rebecca: Right? Like the TypeScript change would've been a very different beast not in a megarepo.

Ainsley: Yeah, pretty big repo. And similarly, though, as you said, you have the people that are in the monorepo shops, and if you don't invest properly in the tooling there, you get the classic like cloning is slow. Pulling is slow. Every operation with source control management is slow. The builds are slow. Just general sluggishness. Even though I have this amazing power that I can basically like get this one repository at Stripe and I can just do sed across the entire repo, I want to change this word to that word.

It's done in one commit. It's a big commit, but it's just so easy and I walk away. The downside is what are the day-to-day operations? I just want a new branch, et cetera. And so really like there are inherent trade-offs here and there is no free lunch, kind of classic.

That being said, to give my actual take on the issue, I am a very intentionally, strongly in the monorepo camp. I think there are trade-offs, but I think the trade-offs do end up looking better on the side of monorepo because I do think being able to enforce invariants very easily across like a large codebase or even a large number of disparate codebases, especially because they are inherently actually related to each other, it's very rare that something inside of your company is in no way related to anything else in your company.

Sometimes it does happen. My sense is like the Mac provisioning scripts that IT uses are like probably not that useful to someone working on web stuff. But really, you're making a mistake if you think your JavaScript web frontend stuff is materially different and should not interact with your backend. They are going to interact, and that point of interaction is going to become a source of friction if you try and separate them out.

Now you do need to have boundaries between those things. You do need to have a dependency graph, which can be understood by your tooling, because the downside is if you say like, “These are all fundamentally the same thing and there are no barriers between them,” then any piece of tooling that you build is going to have to read all of it.

And as your company grows, that's going to be a lot of stuff. There will be a point in your company's history where people cannot and should not check out the entire thing. I should be able to make big cross-cutting changes, but I'm probably just not going to have all of that on my laptop.

And so with Stripe, going back to what you were asking, because yeah, Google and Facebook and a couple of other really big tech giants are large monorepos effectively. And they've done things to allow them to do that. It's going back to the investment. We hired a lot of those people and so directionally they came in and said like, “Hey, we do want things to be a monorepo at Stripe.”

That is still true today. I think directionally the things that are not in our largest monorepo, but are logically connected to components in them, we do want to pull in, we want to collapse them in. But we are cautious to do that because every incremental change that we make towards monorepoing is remaking that trade-off decision, right?

And we want to make the right decision at the right time, which means that we could go and take this like bundle of Java over here and mash it into this bundle of Ruby over here before we make that decision. I always ask, “What are we going to gain by mashing these two things together?”

Maybe you can share a schema easily between the Java and the Ruby. What are we going to lose? These two repos separately already have a bunch of performance issues operating in them. We're going to worsen those. So we need to first sort out those source control management issues and then we should merge those together.

But the long term North Star has always been monorepo.

Rebecca: Are there other decisions, that were in early and formative that you feel like you're living with now for better or for worse?

Ainsley: There are two decisions that are top of mind right now. One that we didn't make that I only recently realized the second order effects of. So: consistent code formatting. We have so many tools, right? Now we have Prettier, we have Black for Python, et cetera. It feels quite recent I think in the industry that people have just gone to this like no config code formatting tools where they’re just like the author of the formatting tool has already decided what's good for you?

Zero opinions. And I love this shift in the industry. Like I'm very much now against the YAML files that allow people to choose how the code is formatted.

But we did not do that for Ruby at Stripe early on. And it took us a really, really long time to get there. Now we have that. Well, an employee at Stripe in the open source community built a Ruby formatter called Ruby Format — so that's Fable — and it had like a couple of quirks in it that meant that it didn't work fully on our Ruby codebase. We as Stripe went in and fixed all of those. A person called Reese at Stripe went in, fixed a bunch of bugs in it, and we adopted it.

We formatted the entire repo. Like it was a really big migration. It's now 100% auto-formatted, which is great. But now I actually get to compare the world of like before auto-formatting and after, and honestly? I hit save and my code kind of just pops into the right place is nice.

But one of the things I realized is, now people can make, like doing a code mod, for example, right? Like we have some really old, janky API. And someone would think about fixing it before and then they'd be like, honestly, this is gonna mess like…. I might even be able to get this syntactically correct, but it's gonna cause stuff to go over line boundaries — I just don't want to deal with that. And now they're like, obviously don't do huge code mods with regex if you can avoid it, but you can now and then you just run the formatter over the codebase and it just like puts things in the right place. And just removing an entire class of reasons why people didn't do something because it seemed like a bunch of additional work.

Rebecca: So a thing that I talked about with with my first boss at Stripe was how changes like that Ruby format change can change how people work. Not just make them incrementally faster, but really change the choices they make and how they think about the work that they're going to have to do.

I'm curious if you've seen other examples of that, because I think that was a really powerful thought to me that, that this isn't just about find 10 seconds here and 10 seconds there, but you can actually materially change workflows. So yeah, is that something that has been a theme for you?

Ainsley: For sure. So I think my biggest hypothesis here that we haven't actually 100% proved out at Stripe, because we haven't finished this project yet, something that's ongoing, is the power of stacked pull requests. And so once again, I think there's this concept of like, no one truly knows how people will react once we land stacked pull requests.

Some people are just like, oh, maybe people will just make multiple pull requests. Like something that would've been one pull request will be multiple pull requests. But I just think there are so many added on effects to this. I think one of the reasons that people make large pull requests today is because the tooling around splitting those up, or even like the mental burden of thinking of how I want to split this up, is just something that no one really wants to go through, because the tooling doesn't really support it in a first class way. And so they don't. So you'll get a big pull request that has like some JavaScript and some Ruby and some Java change and some like proto schema change and it's kind of bundled together.

And if the person's really nice, they'll say, “You can review this pull request commit by commit.” Which is like not once again a first class citizen of the GitHub review UI. It's just like a thing that you can do if you want to go through it.

But I've really been pushing this idea that if we make it easy for people to make many pull requests stacked on top of each other for one single logical change, but to like naturally split that up, I think they're gonna make smaller pull requests, which will get reviewed quicker because no one's faced with the daunting task of a +1000-line PR. It's gonna be easier to spot issues in that smaller pull request. You're gonna have the engaged attention of the reviewer for a longer percentage of the time that they're looking at that page.

You're probably gonna end up with fewer incidents because someone's not trying to splat everything in one single pull request. And that also changes like how people think of changes.

Going back to my story of being a Facebook intern. Definitely in one of my first five pull- requests my intern manager at the time when I sent it to him, I think he just outright rejected it within minutes and said, “Send me a smaller pull request.” That was the feedback and I'm sure the approach was fine. It was just like, don't ever do it again.

And not only did it make me go back and make a smaller pull request, but I actually changed the, smaller pull requests were actually smaller in total aggregate number of lines than the big thing I made because I actually had to think about what part of the thing am I trying to change in this thing? And how do I make this like naturally layer on top of each other?

So it's just like, that is something where I'm convinced personally. I have zero evidence, right? There is no metric I can bring out. No anything that I can bring out to prove this. But I'm deeply convinced that this will not only make people work faster, but fundamentally change how they do engineering.

Rebecca: That is exciting. I know that was always a topic for people who had come from places where they had stacked pull requests and found it very frustrating to not have them. So that's awesome.

I am here to vouch for some stuff that you do in this space is based on your experience and your gut. And it's going be really hard to measure it or prove — maybe you can prove that it was good, but you can't prove how good it was, those sorts of things.

So yeah. How do you personally, and how does a developer productivity organization think about what should we do next? Especially… I’m assuming the opportunity space is effectively infinite. So how do you figure out where to spend finite time?

Ainsley: I will say over the intervening five years that I've been on developer productivity at Stripe, we've definitely gotten like less vibes-based on prioritizing projects.

I will say early on, definitely slightly vibes or… I don't actually think they're bad. I think you hire people for their expertise and their knowledge and like generally speaking, if someone has had a good career in productivity at a bunch of productive companies, they're gonna have quite good vibes on the things that you're missing that you really should be building, as we said, stacked PRs or dev boxes even.

It’s much harder to just be like, this is going to improve productivity by n percent. But one of my first projects actually as a tech lead of developer productivity at Stripe was measuring what we now call time spent waiting on machines. So we used to have… sporadically people would add some instrumentation to their tools and it would end up in different places, some in a data warehouse here or someone would just use like a different metrics pipeline to emit log lines to Splunk or something. And you could ask individual teams like, “Hey, how are things looking with your code generation tool?” But like there was no holistic way of asking the question, like, “How much time do people spend on this?” And can I compare it like apples to apples with another thing that might be taking time?

So one of the things that I wanted to do was to… uniform data pipeline. If you have a tool that humans will wait on, you will instrument it in this one way, and it will have this unified schema, and we can now make queries so they're just like down to the individual level. You can actually just be like, how long did Ainsley spend waiting on tools yesterday, and which tools did you spend waiting on? How much of that time was spent waiting on Git versus builds versus like IDE doing the spinning thing, like type checking or something like that.

Like all of those are queryable on an individual level. You can aggregate that to teams. You can put people into personas and be like, “Okay, look, these are our JavaScript engineers. Here's what their breakdown looks like.” That was a superpower. It turned into an entire half [of work] once we had that data of just like, “That's bad” or like, “That's fine.”

But we could finally not just be like, “I feel like this is slower than it needs to be,” but fundamentally, people are spending this much in the way of time per day waiting on this. And it's very easy as a business to convert time into to money. They pay us a salary. We work some number of hours a week. You can quite easily come up with a formula to be like, saving this saves this much money, fundamentally.

Rebecca: So if you have a build that takes 30 minutes, that's a pretty obvious target. Let's make that take less time. But I know one thing that we talked about a lot was when we were working together was what about the smaller tasks that just take 10 seconds, but you have to do them 30 times an hour? Or maybe 10… I don't know where the threshold is.

But how do you think about that? How do you think about like, you go after the 30 minute build, but actually for whatever reason, people only experience that 30 minutes once a day. So I'm just curious how you think about that from a prioritization standpoint?

Ainsley: For sure. I'm still very much in the weeds of figuring out where I fall on a lot of this stuff, because as you were saying with the build times, I've been thinking an awful lot recently about what is the distribution of that kind of metric, right?

Like how many people experience the p90 of build durations a day. Because you just end up in really interesting conversations where you're asking yourself the question, like, okay, when we normally look at the p50, the p90 of event we're doing on like a per event basis. What is the half of all builds take above how much time? That actually doesn't necessarily tell you how that converts into human experience partially because — sometimes people write us friction docs another great way of telling what to do. And maybe we'll talk about like the non metric things after, because there are many ways that you should be deciding what you're doing that aren't numerical — but you know, someone will come with a friction doc and they'll be like, “Hey, I feel like every single build I've done this week lasted 40 minutes.”

And my initial intuition was, “You're wrong.” Because this is just not how percentiles work.

Rebecca: “Sounds untrue.”

Ainsley: Sounds inaccurate. And then I would go into our tooling and I'd type in their name and I'd be like, actually, my bad, you know? It turns out that actually yes, the type of work that you have to do every day involves perhaps changing a set of files, which triggers like a pathological case of our build system.

And there is no way for you to avoid this as a human being because it is very much a distinct part of the team that you work on, the codebase that they have. But in the metrics that we traditionally track, p50 build time, your experience is not being captured at all.

And so I've been thinking a lot about how we actually do incorporate… Is this on a per build basis? Is this on a per person basis? Like, how do we want to balance out those two concerns? Because I think neither of those are wrong ways to look at the problem. So that's one thing on the, how your point of how do I think about the longer things vs the shorter things. It's just a completely not the same event experience by different people, but like this is a completely different event with a different kind of interaction.

I've been thinking a bit about like how we bucket the events in terms of effectively user expectation. So that is to say if I click a new file in my IDE and I open it up, I don't want to wait 30 minutes for highlighting to kick in. It's probably a different expectation that I have for that bucket of event. And so I think thinking in terms of things as like what is interactive? Like hard interactive, like there's some kind of a real constraint that we should be putting there.

What is, I would say soft interactive. So for example, go to definition or find references in my IDE. Yes, I want it to be snappy, but four seconds is not going to kill me. There's a lot of information to process there and my expectations are different there.

And then there is what is almost entirely asynchronous, right? Like I run a very, very big build. I run a load test or something like that.

And then the last thing we used to talk a lot about this back when you were at Stripe, is like there is feedback that we think is asynchronous, but we actually, can we move that feedback left in the development cycle? In theory, you could tell me, “Hey, we ran this linter on your code 12 minutes into my CI build.“ I would much rather have that linter running in the background constantly so I can get that feedback sooner.

So I've been thinking a lot recently about like, yes, how do we bucket the types of metric that we're getting into the system, and not just measure them necessarily against each other, but kind of measure them against the expectation of that type of event for the user.

Rebecca: You also mentioned though, things that maybe metrics don't tell you. And I know one of the things that really drew me to Stripe was the surveys that they do on a regular basis.

And you and I worked on a project to do more frequent engagement with engineers. You mentioned friction docs which are also very Stripe-y. What about ideas that can't back up with, you know, a graph in the tools?

Ainsley: Yeah. We do two huge surveys every year. Every half we do what's called StripeSat, where we ask employees a lot of questions about what it's like to work at Stripe, and specifically within that we also ask them what it's it like to be like an engineer and a productive engineer at Stripe. How productive do you feel, what do you think about different types of tooling? We ask the entire population that twice a year. We ask segments of the population subsets of those questions every month. That we get as kind of a more regular lower resolution update around those kinds of questions.

Then we get friction docs. We talk to engineers, we shadow them. We still do that. We still go to people and they're like, let me just watch you do a task for a bit. It's nothing to do with your performance review, I'm just watching for my own purposes. And it is really interesting starting those conversations because it's really fun to say to someone like, “Honestly, if you are unsuccessful in doing this task or it takes you time, that is actually a bad performance review for me, not for you.” Like I am doing my job poorly if you are not feeling productive right now. And so don't worry if you have to deal with this tool.

Our goal or my goal, the people I work with, we have to synthesize all of that, like non-numeric data. And we have to form a narrative. We have to say like, “What are different types of Stripe engineers trying to do, and why is that hard?”

And where are the places where we can measure that? You know, if someone is blocked, a lot of people come to us, they say like, “Hey, we're blocked on code review. Someone needs to review my code. No one's doing it.” Okay. Can we quantify that? If they say “I'm blocked on someone else's on-call person, they need to action some ticket.” Can we quantify that?

We try our best to quantify some of these things, but fundamentally we also sometimes just say like, a pretty large percentage of your engineering population is having this issue and it's something that we need to investigate more and that we need to invest time in.

That teaches us what we need to be measuring. But also sometimes it puts us onto things that it would take way more effort to measure than it was worth just like fixing the problem in some sense.

Rebecca: That definitely has been a lesson for me that, especially starting from metrics to discover what you should do is… it’s a tactic. But a pile of metrics is not necessarily going to tell you where your biggest opportunity is. And talking to people is often where you learn about these opportunities that you didn't know to look for because you're not doing the work that they're doing the work that they’re doing every day.

And that's an interesting thing too, like how do you stay… you mentioned shadowing, but how do you stay in touch with what the lived experience is for that L2 doing their software engineering?

Ainsley: So as much as I sometimes get way off into like my thought-leadering space, I do actually write a non-trivial amount of code at Stripe. And I need to — to be good at my job, I genuinely do need to.

This happens all the way up to the CTO, honestly. So the CTO also sometimes does these things called “engineerications” where he’ll just join a team, write a product, write a document that's just like, here's the friction that I kind of encountered going along with this task.

So for me personally, it's a mix of writing code. I also similarly encourage people within the general developer productivity team at Stripe to also, if they can, rotate to teams outside of developer productivity, because we do sometimes engineer in fundamentally different ways.

A while back when I was talking about like having a monorepo, but like still wanting to have some structure within that monorepo that splits stuff up. Like we're in a corner of that monorepo and actually our corner of the monorepo is quite nice.

We rake our front lawns and keep our gutters clear and it's actually quite nice. But that is not necessarily representative of the regular product engineer at Stripe. They may have made different trade-offs in the team. There are new frameworks, maybe they're not using them. Should we be migrating people off that framework? The only way to find out is to hang out with them for a little while. So making sure those rotations are happening.

And then I'm in the office like not infrequently. And so that well, all of the offices, honestly, you know up here in Seattle, in San Francisco, back in London, and I get like lunch with people.

I have this really fun story. I went to the Dublin office once and I was having lunch with someone that told me that code search sucks at Stripe. And they're like, “It's so slow.” And I was like, “There's one thing that Livegrep is, it is not slow. I don't understand.” And they're like, “No, it definitely is.”

And so I opened it up on my laptop and I was like, “Oh, this is so strange. It's quite slow.” Found out that actually LiveGrep, I told you it does periodic indexing. It was designed to index while we were sleeping in America. Which is while they were awake in Dublin and there was only so much Disk I/O to go around.

And so there was an entire office of people that were just like, “I don't use this tool. It's quite bad.”

Needless to say, we just had to give the disk more IOPS. But that was just a perfect example of an entire experience that I was not privy to without talking to someone.

Rebecca: Oh, we could talk about remote or like distributed offices because I used to go to Tokyo when I was working at Indeed. And the same thing: Nobody would ever tell you these things until you were sitting next to them. It was never important enough for them to tell you. Or they felt like they didn't know who to tell or they felt like nobody would do anything about it. So yeah, talking to people and making yourself available is so huge. I think you do a really good job of it.

Well, Ains, we could absolutely keep talking and maybe we'll do a round two of this sometime and talk about all the other things we could talk about, but this has been a treat to get to talk to you again about this stuff. That was a favorite part of my job at Stripe, so glad to get to live that again a little bit. So thank you so much.

Ainsley: No worries. It's been my pleasure too. Yeah, it's been great. If you ever wanna do this again you know…

Rebecca: We can make a weekly thing out of it. No, that's probably ambitious. But yeah, thank you so much. It's been great. And we'll talk again soon.