Rising to engineering leadership challenges at Honeycomb

Rebecca: Today, I'm talking to Emily Nakashima. She's the Vice President of Engineering at Honeycomb. We're talking about the hard choices an engineering leader has to make about how to structure and sometimes restructure their engineering organization. We talk cross team collaboration, emphasizing engineering as an equal partner to other functions, and mistakes that seemed like such a good idea at the time.

Emily, it is so great to see you again. How are you?

Emily: I'm doing great. Very happy to be here. Thank you for having me.

Rebecca: Oh, I'm so glad. We talked a couple of weeks ago, just off camera, and it was so fun. I wanted to get you on camera to talk about some of the same things. So, thank you so much for being here. And let's just start by, who are you? Why am I so excited to talk to you?

Emily: Well, I can tell you who I am and then you can decide how you feel! So my name is Emily; she/her; I'm Honeycomb's VP of engineering. I'll say a little bit unique for a VP of engineering. I come from a front-end engineering background, which is how we originally met.

Rebecca: Yes. Ten years ago!

Emily: Yeah! Ten years or maybe even more at this point. Gosh. So I've been at Honeycomb for about seven years now and have been VP of engineering for about the last four.

Rebecca: Okay. So yeah, what was your Honeycomb journey? How does one ‘one day’ end up being the VP of engineering at Honeycomb?

Emily: This is a long and winding road. I certainly didn't join expecting to take on this role. I really wasn't even looking for a new job, but I just met the team, and I met what they were working on. I thought that's a great team, that product should exist in the world. And so I just thought, I have to go try this.

I had managed a couple of teams before, but I certainly hadn't been in any sort of executive leadership role. And I jumped over to Honeycomb with the expectation of: I would be their first front-end engineer, which was fun. And it's a team of folks with like backend and operational experience and there was just such a reverence for front end, and so that in and of itself is so exciting.

But I set the expectation with the founders that I really enjoyed being a manager. I'd be happy to go back to that when it made sense. And one of the founders, Charity Majors, she was like, “I don't think we'll need that for a long time. But you know, well, we can see.” And then of course, like three or four months in, she was like, “wait, wait. Remember what I said? Scratch that.”

So, happily, you know, she and I, I think, are fairly philosophically aligned, but we have very different approaches to things. So, in the early years of the company, it was just us kind of load-balancing all of the engineering leadership. And I do feel really lucky to be someplace where the founders are willing to see how much you can grow. I think so many startups have that reputation for being innovative, but they actually want to just hire someone who's done it three times before.

Rebecca: That’s right.

Emily: And so part of it was, I feel just really lucky to have been someplace where they were like, “All right, let's see. I mean, you can manage this. You can manage that. Well, let's see how far you can go before we need to hire someone over you.”

Rebecca: And here you are. Not the CEO yet! I guess that's the–

Emily: Don't want to be! That's one thing I've learned in this journey: that job is hard!

Rebecca: Don't need that job. I should have started by getting you to say what Honeycomb even is, in case anyone doesn't know. I've used it in past jobs, and one of the things that really stands out about it is that it's a beautiful product. It's clear that you've invested in front end and really being thoughtful about that. But yeah, maybe tell our audience what Honeycomb actually does.

Emily: Thank you. Yes, I should remember. Sorry to our marketing team. I will remember to do that every time from now on. Yeah, so Honeycomb is an observability platform, and we are sort of the category creators of observability to an extent. So we came into the world where people were really talking more about monitoring, and then there were different tools in that space from logging and distributed tracing. And we bring all those things together in one platform.

And I'll say the way the product looks was something that actually drew me to the company. Cause I– especially if you remember back in the kind of 2010s, developer tools used to all have such a Star Wars vibe to them. I think the prevailing design style was very like dark and very – not to gender things but – masculine, kind of, and honeycomb was brightly colored and kind of rainbow. And I was like, “yes, our developer tools should look friendly and be approachable and warm.” And so I really love that they went for that approach.

Rebecca: Yeah. You either had the dark/scary or the Jenkins. I don't know if you've ever used Jenkins in its full glory… Yeah, not so beautiful, but did the job once upon a time.

So talk to me about this observability versus monitoring. I know this is a huge part of Honeycomb's reason for existing and it's a major, major philosophy that drives the product. So can you just talk a little bit about the difference between those two things and what is observability as a category?

Emily: Yeah, of course. So, the definition of observability actually comes from control theory. And the idea from control theory is that you can understand the state of a system without needing to change anything about it. So just being able to look at the outputs of the system. I think you can see how that could translate to the world of software, but I think sometimes it is a little more aspirational and you're in a little bit more of an ‘instrument, deploy, look, instrument, deploy’ workflow sometimes.

But the sort of core of that that we really want to embrace at Honeycomb is making sure that people feel like they can ask and answer any question about their systems. And the connection Charity always makes is: if you have a dashboard full of graphs based on pre-aggregated metrics, you can answer all the questions you already knew you wanted to ask, but in the world of observability, you should be able to realize you don't know something and then go get the answer relatively quickly, versus being stuck because you didn't think to add your graph to the dashboard two months ago.

Rebecca: Right. And I'm going to fangirl a little bit because we used this product, we use Honeycomb at Stripe for monitoring all of our DevOps and DevProd systems. So we didn't use it for production stuff, but we used it to great utility for understanding how our engineers were working and where they were hitting bottlenecks. And exactly like you said – this is the fan girl part – exactly like you said, you could come up with arbitrary questions and pretty much have to do nothing to the existing systems in order to answer those questions. And it was really, it was fun too. It was fun to put together the queries and see what the answers were.

Emily: Nice!

Rebecca: So yeah, loved it. Don't need it so much in my current job. So, sorry about that, but it was huge, huge for us in my last role.

Emily: I love that use case so much, though. I feel like so many companies forget that, fundamentally, everything is software now. Your tools are software! You can actually use your production tools in other places besides production and it can be just as effective or more. So we have a lot of customers who do that same thing where they're using Honeycomb in their CICD pipeline or in their BI stuff. And the impact is the same, even if the problem space is different.

Rebecca: Yeah. And especially once you get to the point that you have a thousand engineers or something like that. That's a solid user base to have a lot of questions about. And they're trying to do their job. And it was a great tool for understanding where they were getting stuck. People would come to us and be like, “This build took two hours.” And we were like, “That is not possible.”

Emily: Oh no.

Rebecca: That is not possible. And Honeycomb helped us see, well, yeah, in your one particular super pathological case, you are doing a very bad thing every day and that's why it's taking two hours. So anyway, yeah. Love it as a tool. That's my little personal plug. I just think it's really great.

Today, I wanted to talk to you a little bit just about your VPE role, and kind of understand a little bit more about what you're responsible and accountable for, and just talk about some of the challenges that you've run into along the way. I know every job like this is extremely unique, and, generally, the stories that you get to tell afterwards are interesting to lots of people, even though the context is so unique to Honeycomb.

So, to start out, what is your job? What does it mean to be VPE at Honeycomb? And what's the size and scope of the role?

Emily: Yeah, that's a great question. I think that, on paper, the answer is that I am the departmental leader for engineering. And, to me, that describes actually the least interesting part of the role. I love that part of the job. You know, I love working with our engineering directors and our engineering teams, but it's the part of the role that I understood the best coming into it.

And one thing I didn't realize when I was taking that step up from director to VP, I was lucky enough to be Honeycomb's first director of engineering and then first VP of engineering. And so it was kind of cool, cause I had a little bit of time to kind of grow into the role and figure out what it was, but it also meant I hadn't seen anyone else do the role.

And so, at first, I thought, “great, it's the same job, but I just have this different title.” And I didn't realize that, you know, in part of that job, I would be joining our exec team, and that's a whole different world, right?

Rebecca: A whole new game.

Emily: Yep, yep. And so, you know, I think a lot of the most interesting parts of the job really are just being a partner to other executives at the company and making those strategic decisions, basically helping to run the company. There's always a balancing act where you want to make sure that that doesn't distract you from running things really well day-to-day in engineering.

But especially as a developer tools company, people are so interested in who are software engineers? How do they work? How do they make decisions? How do they buy stuff? And so it is really fun to be in the room with our executive team and help work on strategy with them. And I think our founders have been quite smart in that they have some people like me who have grown up with the company. And then we've, of course, supplemented that with, you know, many of our go-to-market leaders are quite seasoned, that have come from doing this before multiple times to other companies. And so it's really fun to just be in the room and have all those perspectives together as we think through what are the big strategic decisions we're going to make?

Rebecca: Yeah. I remember when we talked previously, you mentioned that something that's unique about Honeycomb is the amount of cross-functional engagement that happens. That it's not engineering is this ivory tower, and everyone else has to come beg for things from engineering, which is how it works in some cases.

Can you talk a little bit about that, about, what would you say, “I work with stakeholders and other departments”, but what does that mean? And are you a shit umbrella or are you really coordinating, driving people below you to do that same sort of collaboration?

Emily: Ooh, yeah. You know, I wish I could take more credit for that than I think I can. I think, wonderfully, a lot of that actually comes from our founders and I think there's two things they've really emphasized that makes such a huge difference there.

Number one is transparency. I think Charity in particular, she's the person who, at past jobs, she was a key leader maybe within the company, but because she wasn't an exec, she felt like she maybe didn't get all the information that was actually useful to her job and just didn't get to understand exactly what was happening within the company, to her detriment sometimes. And she really had this commitment from the start to be quite radically transparent with employees at the company.

At a certain point, she realized “maybe I don't need to take everyone through all the twists and turns of fundraising”, for example, because that's really distracting. And not everyone needs to experience the emotional ups and downs of that. But really, I think, from the top down, we are very transparent with people about the business. And so I think that naturally cultivates this curiosity of, you just hear all the time about what's happening in our different go-to-market teams and what's happening in even the people team and finance. And so, that sort of cross-functional curiosity, I think is really cultivated through that.

I think also Charity and Christine, our founders, both have such respect for the disciplines that they don't come from. So both of them come from software engineering backgrounds. Christine is more of a software engineer; Charity has been a DBA and a Sys Admin and all those operational roles, like pre and post-DevOps. And I think they understand that maybe the less they understand a part of the business, the more they should bring a healthy respect to that. And so I feel like that's always flowed through our culture.

So I know that I can bring my expertise as a software engineer and as someone who buys these sorts of tools sometimes. But there's such a healthy respect. We both have to bring our knowledge to the table to be successful.

Rebecca: How does that affect hiring? How does that affect the kind of people that you hire?

Emily: Yeah, we have quite a– I think we hire maybe a different shape of software engineer than many companies might look for. And I think sometimes people are confused as to “I was such an incredible principal in this engineer at this company. I keep applying to Honeycomb. They won't hire me.” And it is really– we're looking for that cross-functional respect. We're looking for really strong communication skills, cause we know that to make all that work, you need to be able to really build strong partnerships with all your adjacent functions.

And we do triad-based development within our teams. So there's a designer, product manager, and engineers all kind of at the table together in their teams. And so we're looking for someone who can collaborate with those roles. And then, because we practice service ownership as well, we are also looking for someone who has some degree of operational experience or, for less senior roles, someone who's okay with being on call. Maybe if they haven't done it before, they're willing to learn.

So we're certainly not looking for the widget engineer who's going to turn out code. And it makes it a little harder, but luckily we have a strong enough engineering brand that we're still able to do it. See how long that lasts for.

Rebecca: Ballpark, I don't know how specific you want to be on live TV, but how many people are in the engineering organization at Honeycomb in general?

Emily: Yeah, I think luckily with our approach to transparency, I can just say all the stuff live, which is great. I think, within engineering, we're probably about 75 today and the company is maybe 225 total.

Rebecca: Got it. Got it. So it's pretty healthy, engineering is a significant portion of the company and that's cool. So yeah, let's talk a little bit more about, you talked about being responsible for the, you know, you run the engineering team. But down in the weeds a little bit, what are the things that you are having to pay attention to this week, this month, this summer, this quarter?

Emily: Yeah. That's a great question. I would say there's a real difference between what my stakeholders think I am responsible for and what I'm paying attention to. And if you ask a lot of them, it's like “velocity, velocity, velocity.” And you know, the reality is that within engineering, it's always a balancing act between all the different things that you could spend time on.

So obviously, making sure that we are moving with high velocity and shipping value to customers is something that we think about quite a lot. That's why we have Swarmia part of it; we want to understand how people are working together and make sure it's efficient. And there's also just so much time I spend thinking about all the other pieces: reliability, security, performance, quality, accessibility. And I think the hard thing about those is sometimes the better you do at them, the less grace you get from the organization to invest in them, right?

People are like, “oh, you never talked like that. That's fine.”

Rebecca: “Performance is fine!”

Emily: Yeah, exactly. And people can forget that behind the scenes, it turns out that it's fine because you spent, you know, 18 months on an incredible effort to make that good.

Rebecca: Right. Right. So how’s that group of 75 people structured today? And how has that evolved over time, too? Because once upon a time, there weren't 75. So, how does it work today, and how did it grow up to be that way?

Emily: Yeah. So today we are broken into three major, what I would call Domains or product domains in some cases, maybe engineering domains in others.

We've got platform engineering. That includes our SRE function, our core services team, which is operating these high throughput backend services that are challenging to operate. And then it also includes a team called app enablement that's brand new that I'm so excited about, that’s thinking about developer productivity and experience. And it also includes our storage team, which is the core differentiated technology of Honeycomb.

So within platform, those are generally backend or more infrastructure-leading teams. Those teams don't actually have an embedded product manager and designer, although they do have to work closely with our product function. And I think that one is the natural slice that is replicated at other startups of the same size everywhere.

The two pieces that are maybe a little bit different for us. We are broken into– The other domain is that we have is Product Engineering. And that is people doing web development, except that because we believe in service ownership and putting software engineers on call, those are actually quite deep teams with ownership of some fairly intricate and interesting backend services as well. And so staffing that can be quite interesting. You have a team of five people. You've got to have a couple of people who are good front-end experts, and then a couple of people who are back-end experts, and at least a couple of people who've been on call before. So a lot going on in product engineering, even though I still think it's a great approach, especially for trying to understand what we're asking our customers to do and kind of build that empathy.

And then the final domain that we have is what we call DevEx engineering. And that's really thinking about external developer experience for all of Honeycomb's customers. So it's all of our open telemetry investments. It's some of the software that we create that customers run on prem. So we have a sampling proxy called refinery, and then we also contribute quite heavily to the open telemetry collector. So it's folks who are working on that as well.

And I think, you know, mostly we landed on those three domains to kind of mirror the structure we had in the product organization. So we really want to kind of structure ourselves along the lines of what provides value to the customer. We will probably continue to evolve it, but it's working okay for now. Before this, we did a dynamic reteaming model. And there's good and bad in that. It worked up to a certain point, and you could probably guess exactly why it fell apart.

I think up until maybe about half the size we are now, so maybe 35 engineers, we would dynamically re-team every quarter or two quarters and it was great for churning out product surface area as fast as possible. And of course, the downside is, who maintains that product surface area? You know? Yeah. You're calling upon people to be good citizens and go to quietly fix it in the background. But it did not optimize for investing in your code and your tools over the long term. So it was the right time to make the switch.

Rebecca: The one person who worked on that one thing six months ago to figure out how to fix it.

Emily: Exactly. Yeah. A lot of digging and get history, trying to understand who has been here before me?

Rebecca: So, how does collaboration work among those teams? That's one of the big challenges that companies face as they scale up is that you used to have this tight-knit group of people who all knew each other and could self-organize to do good work.

And then, at some point, you get to a number where that just doesn't work as well. So, how do you see collaboration, especially cross-team collaboration? Do you encourage it? Do you actually try to prevent it? How do you think about that?

Rebecca: Yeah, candidly, that is definitely something we are working on updating right now. And it is for the reason you described. We realized we needed to put a little more structure around it because it was really driven by cross-functional relationships that people had built across teams that just happened to be really strong. But if you happen to have teams collaborating on lines where there wasn't one of those pre-existing relationships, all of a sudden things became maybe a little more challenging.

And so what we're trying now is to have three explicit different models for how you can do that cross-team collaboration and give the teams the opportunity to align on which one they were going to pick based on the project. So, part of the way we've structured our product domains is the product manager often has a relatively ambitious remit that follows a customer journey, but that means that it's very common for that team to have to potentially go touch some piece of code in someone else's portfolio to address that customer need. Rather than just maintain these five services, we're trying to solve ambitious problems for the business. Which is great, but it means almost every major project, you might work with two or three teams to ship what you're going to ship by the end.

And so the three models people can choose from are: there is the obvious one of “we are just going to do this little bit of work in your portfolio, but we're going to talk to you about it first. And we're going to make sure that you code review it and sign off.” I think that one is often kind of the norm that everyone defaults to.

And there is also the ability for that second team to say, “Hey, either this is a really challenging part of our code base to work in, or we think that we want to refactor it and make some change. And we want to, rather than have you go work in it, we really want to own making this change because there's some larger initiative that it needs to roll up into.” And so that team can negotiate and say, “No, we're going to own this. And then this is the timeline it's going to happen on.” There's a little back and forth to make that work.

And then there's also a model where the team can say, “Hey, we really want to work on this. And we think we should adopt this piece of your portfolio. You should just give this to us.” And that one, of course, is the least common, but I think that opening the door to let the teams have any of those three conversations has actually been really helpful. And I think sometimes just talking through those options, even if you're going to land on that first one – that's the most common one – can really help people unpack the issues that they're concerned about.

Rebecca: You answered the question I was going to ask. Now I'll ask the harder question, which is what about model four, where team A needs something from team B and team B says, “No.” Or becomes a perceived blocker to getting something done?

Emily: Yeah. That's a hard problem. So it's less of a hard problem for me because I have directors, so it's a hard problem for my directors. I think that is the point where it becomes a negotiation between, typically, the engineering manager, the product manager on each of those two teams, and then sometimes those people both roll up to the same directors, or sometimes there's multiple engineering directors in the mix. And we spent a lot of time doing in-depth relationship-building at that layer, because those are the people who resolve all of those conflicts.

Rebecca: I'm just remembering at Stripe, we had the ‘unblocking’ process. Do you have a process when things can't be figured out at a lower level? I always put these words in air quotes because we're all on the same team. But there is an org structure. So what's the process when people can't agree?

Emily: You know, we don't have anything super set in stone or kind of baked. I think, realistically, we are trying to get as close to consensus as we can. Because even if you have a process for getting a green light, the effectiveness of the answer really depends on how bought-in both of the teams are. So, I wouldn't be surprised if we end up having to create an official process in the next six months or a year, but right now, just try to bring everyone along as best as you can.

Rebecca: Just talk to each other. Yes. Love it.

So we've talked in generalities about how all of this is structured and how it works, but maybe you can tell me a story about a specific cross-team collaboration, and how it went and what was challenging about it and what went well?

Emily: Ooh. You know, we have a fairly ambitious project right now that is around logging. So we have always allowed people to send logs to Honeycomb, but if you look at the really early messaging from the company, it was very negative about logs. And it was like, “Why would you do that with logs when you can have events and traces or whatever?” And it turns out that we even looked at what we were doing, and we were like, “Wait, everyone has logs. They stick around forever. Sometimes, you do want to go look at them.” And so we had to, you know, moderate that message a little bit.

But as we have looked at the needs that our customers have, especially folks who are bigger than us, who can't maybe transition tools as quickly, we went “oh, right. We need to just get closer to matching the logging workflows that they're already used to.” And so we're starting to build some of that stuff into the product.

And we had an interesting situation where we put a particular team in charge of that initiative that had a great PM, a great engineering manager, a great designer, a great engineering lead, great tech lead. And we were like, “This is the team. These are the best people, they’re going to go solve this problem.” And all our people are great, but we thought we had put together a really strong team. And we were like, “they're going to go solve the problem.”

And then as they were getting into it, they realized that none of the work that they needed to do was in their team's portfolio. They just came up with this giant laundry list of work for other teams. And, you know, that's great; they did a real discovery process and really understood what the challenges were for customers. So they did exactly what we asked them to do. And then they had this awkward situation where they rolled up with this giant list of, “here's all these things these other teams need to do.” And we're actually still figuring out what to do exactly with that situation.

I think what we will end up doing is creating a little more official separation between “this is the remit of the triad and the product manager on the team” from “these are the portfolio pieces that the team owns” and be clearer that it's okay that those things can actually diverge quite a lot. The product manager can really be spending their time – like 90% of their time – on different features and problems than the team, as long as everyone is getting what they need from those relationships.

Rebecca: That sounds like interesting ownership challenges where you have people, like one person wants to change something, and somebody else owns it and probably isn't practical for them to own all that.

How do you think about ownership? Do you structure them around problems? Do you structure them around systems that they own? How do you think about that? And what do you optimize for there

Cause you could optimize for, “let's reduce cross-team collab as much as possible and let's try and get everything under one team's roof so they can do everything they need” or not that. So, yeah, how do you think about that kind of ownership boundary question?

Emily: That's a great question, ‘cause it gets me to admit that we totally did it wrong and now we're switching to something else!

So, for a while, we really had this idea that our product surface area was too small to really have clear product domains. So if you think about Honeycomb the product, there's this query and storage engine at the center of it, and then there's all these views built on top of that, that all just call down to the same underlying piece of technology. That is the secret sauce. It drives all the graphs, all the visualizations. And the entire value proposition is there's one source of data, rather than having to jump between five tools that have five different sources of data. Which is great, except that when you try to divide up product domains for people, everyone's like, “we’re all kind of doing the same thing. We can just visualize it in lots and lots of different ways.” And so we had the idea that we couldn't really have true product domains because everyone was working on such an overlapping set of customer problems.

And so what we did instead was we had these teams that owned portfolios of features. And I think the mistake that we made was saying that the product manager and designer on that team also just own this portfolio of features. And it turns out that when you were trying to solve the most exciting business problems that you can, taking your five features and polishing them as much as possible is not actually the best way to have incredible impact to the business.

So that's how we– you've just reverse-engineered that whole situation that I just described, where we're in the process of moving to this model where we really want teams to own a business or customer problem rather than own a portfolio of features. And that process has been partly driven by the teams themselves. We're like, “look at what you have today and then you help us figure out what is the right sort of team or customer mission based on the skills you have, based on the technologies you own, based on where your expertise is.” And that process is 90% done, but I feel optimistic that it will give our teams a more exciting remit and help them.

Especially, I think such a hard thing can be, as a startup, distinguishing what's truly important from all the other things that seem like good ideas to do. And once you can tie it to that kind of bigger business goal, I think it makes it much easier to– well, it's never easy, but it makes it slightly easier to set the priority of the 200 things on your to-do list that you'd like to do.

Rebecca: You just said the magic word, which is goal. So talk to me about goals. What's a good goal in engineering at Honeycomb and how does Honeycomb set goals?

Emily: Ooh, we have a OKR process and we do both OKRs and KPIs, which I like. I like that we make a clear distinction between those. We have KPIs that are just sort of health metrics that we track across all the different areas of the business. So everyone should have some underlying data that we're looking at all the time. We also do those for process things like PR cycle time, all the Dora metrics, that kind of stuff.

I won't spend too much time promoting any particular tool, but we also do a quarterly OKR process where we pick the specific highest priority goals for each team department. And then we also do them at the company level as well, so hopefully they cascade nicely down.

Rebecca: What do those goals look like? Because goals can be a to-do list; goals can be outcomes; goals can be anything in between those two things. So what do those goals actually look like, especially at the org level?

Emily: Yeah. So there's the theory and the practice. And the theory is that they are quantitative, that they're measurable. Ideally, there's some percentage. So ‘move this business metric from this percent to this percent.’ You know, ‘have 35% of customers get data in within the first hour rather than 31%.’ And the reality is that very often, I think a hard thing for us is that the quarter timescale is great for almost everyone else in the rest of the business. And in engineering and product and design, it takes you a quarter to build and ship the thing. And then you need a quarter to measure it and see if it actually had the impact that you want. And so that's actually something we're still trying to figure out the right process for.

I quietly look behind the scenes at how those metrics are still doing a quarter later, but I would really love to bake it into our process such that, you know, I think, at this point, the teams are pretty good about probably half of the KRs tend to be like, “yeah, we just have to ship the thing or hit this milestone” and then half of them are, are generally relatively quantitative, like ‘move this business metric by this percent.’

So we've at least got a mix, but I would love to have a clearer and more official process for the ones that are like ‘ship the thing.’ There should be some long tail where you keep looking at the metric and that's an unofficial thing right now and not an official thing.

We do have a sticker that commemorates this, that says “my OKRs say ship the roadmap,” which I laugh at every time I see.

Rebecca: You showed me last time. I love it. Love it. How does goal setting actually happen? Do the teams kind of receive company-wide wisdom and then turn that into their goals? Or how much agency do the teams themselves have in figuring out what they'll work on in a quarter?

Emily: Yeah. So the joy and sorrow of engineering is that it's both top down and bottom up, which is either theoretically great or actually horrible, depending on who you ask. We have a company-wide process. Ideally, if we have our lives together, that process is like a couple of weeks before we're asking the teams to write their own OKRs.

Realistically, it's probably kind of concurrent or a few days before. But there are always company ones. Happily, the company ones, the objectives, we try to make them year-long. So hopefully it's not too shocking or surprising to see what comes out at the company level. But there's the company-level ones, those are released a few days before the teams are going to do theirs. And then, generally, the teams have a fairly good sense, too, just from what's on their roadmap and what they've already been working on. And, since we set them at the department level, very often at the department level, we're doing that, looking up and looking down and trying to merge them together.

And realistically, probably, for one of every five KRs, you catch the ships on the collision course where you're like, “Oh, you want this and you want that.” And “these cannot live together in the same saline of OKRs.” And so, that's what this week and last week has been all about. And it has been a lot of running around and me going, “I'm so sorry. I didn't realize that I was asking you to do those two completely incompatible things at once. This is my fault.” And just having a lot of conversations with the teams.

I think that if we were more disciplined about actually having the exec team get it together, like three weeks before the team was writing theirs, it would be a much smoother, easier kind of merging. And, yeah, it's a work in progress.

Rebecca: When you figure that out, please tell the world because I don't think I've seen anyone actually do that well.

Emily: Again, the directors are like the load-bearing part of this. They are the ones fitting the gears together.

Rebecca: So you've grown quite a bit and you're successful. That involves hiring in the engineering organization. How does Honeycomb approach hiring software engineers? I'm guessing that you don't have very much trouble attracting talent because it's a pretty cool company and a pretty visible company. But what then? What happens after I apply?

Emily: Yeah. I'll say we are very lucky in this regard, especially because we've got some big name folks who have a good social media presence and who do a lot of conferences. Charity Majors, Liz Fong-Jones. We have a great engineering brand, which I can take almost no credit for myself. But it does mean that we get a lot of interest from great candidates all the time, which is really, really an incredible gift. I've been at so many companies where you had to spend so much time even getting software engineers to reply back to your email, and it's not a problem we have. So I feel very lucky and hope we can keep that for as long as we can.

And I think the big things that are probably different for us… One thing I mentioned earlier, but I could say a million times, we do really stress communication and making sure that someone is really good at communicating with adjacent disciplines, not just with other engineers. And maybe one way to talk about that a little more specifically is we do have a technical take-home exercise. I know many people love to hate those, but I think it does actually tell you quite a lot about how someone will take on a task where time is a component. You know, time is not everything, but, at the end of the day, if you work at a startup, time has a value.

So we set fairly loose parameters around it, but we do at least look at how long it took someone to complete that exercise. And then one thing that we do that I really like, that's maybe a little different, is we always make sure we do a synchronous debrief with two people on our team after that technical take-home where the person sits down and they talk us through, “this is how I approached the problem. This is the thing that was weird.” And we ask them to give us feedback and the technical take-home is not that good, so most people will have a lot of feedback. And it's like, can you give feedback to someone you don't know in a nice way, but that is genuinely constructive, that points out the things that could be better? And that's been actually quite informative.

Rebecca: Hey, that seems actually really valuable. What are your expectations of engineers as far as working with people across the business? Do they need to do that that often? I mean, of course, there's engineering and product and UX. There's that triad you talked about, but I'm talking more beyond that. Is there cross-functional interaction happening outside of engineering?

Emily: Yeah, I think we have an above-average amount of that. And I think partly it does kind of come from the top. We've really tried to cultivate that respect for the other disciplines, but I think also we want to trust engineers, and we want them to understand the context of the business. And I think not every company wants to extend that to engineers. There's a little bit of “we don't understand them. They do all that weird technical stuff over in the corner.” Certainly they cannot understand our world. And I think if you're building a startup, engineers are often very smart, talented people, but they're also really expensive. You should go get some value out of them, make sure they're aligned with the rest of the business.

And so our engineers often work very closely with, we've built a beautiful collaboration with our finance team, actually. A lot of people in engineering and finance actually talk every day. And they're always talking back and forth about building models and thinking about different optimizations we could make. And I think that's been, again, I take very little credit for that relationship other than suggesting that it was maybe important. And then, Jess Mink, our director of platform engineering, and Lex Neva and a couple of other people on the team just sort of went and made it happen.

And we also work quite closely with product marketing, with customer success, with our sales folks and our solutions architects. And I think, again, it's a little bit of we help them see what's important in the business. And so it feels natural to go, “Oh, I can't actually be successful at this without kind of reaching across the aisle and making sure I'm bringing that person along with me or genuinely making sure that I'm taking into account their feedback as we're building.” And I don't think we're perfect at it, but I think that at least we have gotten to the baseline of every engineer recognizes that it's important and… I'll take it.

Rebecca: And that's so interesting to see how that changes as you grow because it's really easy for the first ten engineers to be very interested in all these things because it directly affects their lives in a way that maybe engineering 100 doesn't feel that same connection. So it's interesting to think about how you create that in your hiring.

A lot of what you're talking about is culture, and so we can use the word ‘culture fit’ here, but I know that's also a bit of a weasel word and can be used to mean, “I hired a bunch of people who have the exact same background as me.”

So what are you doing to make sure that you're also getting diversity of thought, and diversity of contribution, the benefits that you get by hiring a diverse team. What do you do to drive that and get all these positive cross-function collaboration outcomes?

Emily: Yeah, I could talk about this for four hours and so I will try to do the short version. I think, actually, one of the biggest unexpected accelerants for this was actually going from being based in the San Francisco Bay area to hiring around the U.S. and Canada. We started doing that a little bit before the pandemic, but we really accelerated it during the pandemic, as many companies did. And we really committed to, we're going to be distributor forever, distributor first.

We have a San Francisco office, but there's like six people who go in regularly out of 225. And, I think, just having worked at a lot of San Francisco-based companies, there can just be an assumption people walk in the room and they go, “Everyone believes this. Everyone feels this way. Everyone feels this way about this thing. Everyone is striving for this.” And you just realize you can't do that when you work with people who live in a completely different part of the country than you; live a really different lifestyle; have a giant house on 10 acres, and spend some of their time on their hobby farm after work, you know? And I saw that be such an incredible empathy and curiosity builder in a way that I didn't expect. So that was the answer that, for all the things that I tried to design to support this, that was the one that, in some ways, was the most impactful that I didn't expect.

We are quite a values-driven company. And I think that helps kind of cleave apart the gross part of culture fit from the part that is genuinely useful and good. You know, there's the bad part, that’s like, “this person's like me. I would enjoy getting a beer with them after work.” And that's not great. You don't want to hire for that because you just become more and more similar, and then the culture becomes very unappealing to anyone who could bring a new idea to it.

But then there are coherent values that drive the way we work together. And someone who maybe doesn't embrace those is going to have a hard time being successful with us. And you do want to interview for that. And I think, partly because of Charity Majors and how much she is values-driven herself, the company really has consistently tried to spend time polishing our values together and making sure they really are relevant. And the test we always use for that is: do people repeat them in daily work around the company? And as long as that's true, we think that they're probably still relevant.

But we try to ask questions on each of the values in the interview process. We sprinkle them into the different interviews, so it's not a weird all-at-once values thing. But that actually has been pretty successful. I think we really have found ways to drill out the things that are specific to our culture that are about things we need to have a shared mindset around, rather than just, “Does this person seem like a cool person that I want to talk to for a long time?”

Rebecca: Does anyone show up wanting to interview at Honeycomb without knowing of Charity? Is that like a possible thing?

Emily: You know, in the first couple of years – I guess I've been hiring at Honeycomb since about maybe 2017 or 2018 now – and it took a couple years. But now, actually, especially because we hire in Canada and we don't geo-adjust salaries, we get so many people who find us through LinkedIn, and they're like, “This sounds really cool. And that's quite a competitive salary.” And I actually think that's great. It's so good to have a mix of folks who are people who maybe come more from front-end engineering and just are not following along with– the observability conversation has been very DevOps-centric. So, yeah, I love it whenever anyone's like, “Who's Charity? Who’s she?”

Rebecca: And if you don't know who Charity is, dear listener, look it up: Charity Majors. She has a lot to say about a lot of things and she's pretty smart about all of them. So, take a peek.

Last question that I did not brief you on. So, this is a surprise, but any hot takes about tech these days? These days are very different from five years ago days. ..

Emily: Since I didn't have this question ahead of time, I'm going to give you a real honest answer and you can decide if you want to keep it in the final edit or not.

I feel like there were so many things that we worked on improving in the 2010s that really had such a positive impact on company culture and how we ran teams together. And we could do that because talent was so scarce. There was such a premium on – and I wish I had been more cognizant of this in the moment – understanding how much power we had as employees.

But I remember when I was first getting started in tech, I really had a lot of questions about whether I could be successful at all as a woman, as an openly queer person, as all these things. And now all of that stuff seems sort of a given, not in every company, but, many places, it's like, “Of course,” right? And, at Honeycomb, I hope that many people have that ‘of course’ feeling. But I also feel such a need to really quickly write down what worked and be like, “What did we actually do back then?” Cause I have this feeling that we're going to have to go do a bunch of it all over again.

I think about Women Who Code, which just went from being a local meetup to a national organization. And then, recently, the national organization just talked about because of the challenging climate, they're going to shut down. And I'm like, “We have to capture what we did so that we can know so that we can do it all again.” So I feel a little bit of Groundhog Day, but I also feel we learned real things that we can deploy again.

Rebecca: I love this. There's somebody who I've talked to recently who said maybe psychological safety was a zero interest rate phenomenon.

Emily: Oh, that's so real. It is so real.

Rebecca: But yeah, I think it's really been, for us people who have been, I've only been doing this since like 2006. And there are certainly people doing it longer than that. But yeah, what a journey these last few years especially have been. I didn't realize how much– I knew it was good, but I didn't realize how quickly it could turn.

Emily: Yeah. I really wish I'd swung for the fences a little harder.

Rebecca: A little harder back then. Well, Emily, this has been fantastic. Thank you so much for spending some time with me today. And I hope people know more about Honeycomb now. And just really love talking about the real nuts and bolts of doing the VPE thing. So this has been really great!

Emily: Yeah. Thank you so much for having me. Obviously I could talk about this stuff all day. So it's been great fun.

Rebecca: Great talk.