Last October, Arnaud Porterie, the CEO and co-founder of Echoes, spoke at the CTO Craft Con conference in Berlin. During his talk, he explained why developer productivity is not an effective way to measure the performance of engineering teams and shared the insights he gained from his experience as Deputy CTO at Veepee, which helped him forge his perspective.

Below, you’ll find the video of his talk, and a recap of 5 key points that highlight the philosophy on which Echoes is based.

• The Importance of Alignment in Engineering Organizations
  Effective engineering leadership begins with ensuring that teams are aligned with company priorities. Misalignment can lead to inefficiencies and wasted resources, as seen in cases where numerous initiatives dilute focus and hinder impactful delivery.

• Productivity Metrics Are Not Enough
  Metrics such as deployment frequency or velocity are essential but insufficient to measure engineering success comprehensively. The ultimate goal should be driving business outcomes, not merely increasing productivity for its own sake.

• Feedback Loops Drive Continuous Improvement
  Effective feedback mechanisms, such as concise weekly updates, reviews of key metrics, and tools like JIRA, ensure teams stay on track and allow leadership to identify issues early. However, these feedback systems just be efficient and actionable to avoid burdening teams unnecessarily.

• Team Health and Sustainability Are Crucial
  Long-term success relies on maintaining the well-being of teams. Identifying overburdened contributors and using tools to measure team engagement and health can help prevent burnout and foster a sustainable work environment.

• Adaptability and Rapid Decision-Making
  Engineering environments are ever-changing, requiring leaders to stay attuned to shifts in alignment, team performance, and external priorities. Quick adaptation through robust feedback loops and consistent tracking ensures organizational goals remain achievable and relevant.

You will find below the transcript of the video:

Hi everyone,

I hope you are still alive and well after a whole day and a half of the conference already. Let me start by introducing myself and what I'm going to talk about.

Who am I? My name is Arnaud. I was born in Paris, and I’m currently based in Amsterdam after a couple of years in San Francisco. I have two young kids and one startup, which, I'm not going to lie, on certain days feels like either too many kids or too many startups. But that’s what it is.

In terms of my background, I was one of the early employees at Docker. I was a core maintainer on the open-source project, and I was also one of the first engineering managers at the company, managing the core team. We called that maintaining the maintainers, essentially. After that, I had the opportunity to join Veepee. Veepee is a very large e-commerce player, which you may not know about in Germany, but it’s really, really big in France and other countries. I got the chance to lead an organization of 800 engineers, and that's a lot of what this talk is going to be about: sharing my learnings and my personal experience in this company, trying to measure and communicate the performance of my teams. Today, I am the founder of Echoes. It’s my first company as a founder, and we have a booth here if you’d like to talk about it later.

So, let’s get started. We are in 2017. I just got my job at Veepee. “Congratulations, you're on the exec team.” That’s pretty cool, right? Because when you are in the career track of engineering management, that's pretty much the Holy Grail. It's where you want to be. So, I was super happy I was on the exec team. Then the first executive committee happened. It was an experience. I would say it was different—not necessarily what I expected.

The first thing that happened is that I was sitting at the table, and I saw our head of sales and our CFO share some slides and data—very convincing data about the progress of their team and about everything they were doing. Then we had the CMO, marketing, come up and share slides—pretty slides, of course, because it’s marketing. They had a lot of convincing data about everything they were doing, and it was really awesome. I had absolutely none of this. Even worse than this, I could really feel the weight of the expectations.

We are in a time where there are no business results without tech. That means that when you are sitting in an executive committee, a lot of discussions are going to end up talking about tech. You are at the crossroads of so many discussions, and you have stakeholders looking at you asking, “When do we ship? Can we do X? Can we do Y? When does it come out?” You know, like me, that tech is complex, chaotic, and somehow unpredictable. But the fact is, they don’t care about this. They want to know when things actually come out, when things ship, and when we’re going to deliver business value for the business.

That put me in a situation where I had to figure out what it is that I was going to tell them on a weekly basis, in the same way that the CFO and the CMO do, to communicate our activity in a way that would make sense for them and, of course, would feel comfortable for me in terms of what I wanted to be as a CTO. So, what can I share during a weekly meeting with my peers in the exec team? How do I communicate our activity with data and not just gut feelings? Really, the bulk of the issue here is that however competent I am as a CTO, if all I have are gut feelings, then I’m just a guy with opinions, and that’s not going to cut it. I think we should be doing better.

So, any ideas? What should I share? What should I tell them? Any metrics? Any numbers? Any things that come to mind? Sorry, I heard something. Production rollouts in production? That’s a good idea. I see a lot of CTOs doing that. Anything else? Technical debt? No? Lead time? Perfect. Things that are really about outputs and about how much we’re producing.

I’m going to bundle this up into what I’m going to call productivity metrics. It’s not just productive metrics, but a lot of it. So, DORA metrics, agile metrics, velocity, etc. I don’t think they’re enough to capture the entirety of the success of an engineering organization, and I’m going to explain why. The first point is that productivity is very rarely the bottleneck. I don’t know about you, but personally, I have literally never felt as productive as a developer as we have today. We have access to the cloud, Docker, and AI coding assistants. I mean, come on. I was literally capable of bootstrapping a company on my own, and I’m far from being the best developer in the world. There’s an obsession right now about developer productivity, but I don’t think it’s what we should be talking about the most.

The second point is that productivity metrics send the wrong message. What I mean by this is that in my position as a CTO, I strongly believe that being a good CTO these days is about caring beyond the code. It’s about having an impact on your business. It’s about moving the needle for your business. The problem is that as soon as you start using productivity metrics as proxies for the success of your organization, you’re perpetuating the idea that your job is to put code into production. Which, I would agree, is a fair part of the job, of course. But again, what we should be caring about is moving the needle for the business.

Lastly, nobody cares. Especially not the people in the exec team. They don’t care. If you’re productive, good for you, but when do we ship? When do results happen? Productivity is not the goal. Productivity, of course, is important, but it’s not going to matter to your stakeholders unless it translates into positive business outcomes.

Taking a step back then, what are the components of measuring performance? What should I be measuring and sharing if I want to communicate what I believe is the performance of my engineering organization? The very first part and the key building block, in my mind, is alignment. Are we even working on the right thing?

When I joined Veepee, the focus of the team was the main pain point. I could hear many teams tell me that they had just too much to do. I had people tell me that we were very good at starting things but not so good at finishing them. I’m sure this is something you’ve heard before. I see that in so many companies and with so many companies that we’re working with and talking to today. It’s probably the most common issue I’m seeing in engineering organizations today.

How do I make sure that my team is actually focused? How do I get the data to show this to my stakeholders? We took on an exercise that started, like most things in life start, with a spreadsheet. We created a spreadsheet where we said, “Well, in rows we’re going to put the teams and the departments. In columns, we’re going to have the goals. We’re going to have the projects.” We went through each team—and that was very painful, but we had to do it. We talked to each team and asked, “What is it that you’re doing today?” Not in terms of the roadmap or the plan, just what’s keeping you busy right now—this week or this month. We asked for a rough idea of the big buckets of work. As you can see, I’m not going to go into the details, but it’s a very rough score to say, “Are you spending 10% of your time on this, more than half of your time on that?” and trying to map exactly what the activity of the teams was.

This gave us a heat map of the activity—something that I could use not only to measure the focus but, most importantly, to communicate to my peers. For example, when I joined the company, we had five company strategic pillars. How many columns do you think we ended up with in this spreadsheet? Ten? I love your optimism. No, we ended up with roughly 31, if I remember correctly. Five company pillars, which were supposed to be the only things we were doing, and 26 initiatives important enough that they were worth mentioning in this document.

I cannot overstate how valuable this was for us. What it allowed me to do was come to my CEO and the exec team and show them, “This is the reality of the allocation of the budget today.” The way I presented it was, “Our engineering organization is a cake. This is how we are slicing it right now. We can disagree on the way it is sliced, and we can do different arbitrages. I don’t have a magic wand, and I cannot grow the cake. The only thing we can decide now is whether we should be doing different things, making different arbitrages, or keeping it this way.”

It really put the finger on the spiral we were in. We were in a situation where there were frustrations due to a lack of delivery. Because of that, the stakeholders were piling on more topics in hopes that something—anything—would come out. But by doing so, they made the focus even worse, hurt the delivery even more, rinse, and repeat. This is where we were with 26 initiatives on top of the five company priorities. But it went a long way in building trust because, for the first time, we had this kind of transparency. We could say, “Look, this is not a tech problem. This is a company problem. We are in a situation here where this is not about us not delivering results. This is about the company collectively not agreeing on what truly matters.”

There’s a point I want to stress here about why I believe alignment is so important. There is no amount of developer productivity or delivery performance that is going to fix this. Best case scenario, you’re going to go faster on your 26 initiatives and get to the 27th one. But that’s not going to help in the long run. This is how we approached alignment and how we got better at prioritization.

Then comes the question of delivery performance. Once you have clarity of goals, sure, now we can discuss how we’re going to reach them faster. But that’s not the starting point. You’re all familiar now with DORA metrics and delivery performance metrics. I’m going to talk specifically about two learnings I got from this experience and also mistakes I made with them. The first thing I want to say is that these delivery performance metrics are great at setting a baseline.

I was in a situation where I joined after years of underinvestment in tech. That’s the nice way to say that we had three giant monoliths running the company, and nobody knew how to touch them anymore. The DORA metrics were great because they were a way for us to measure if the teams were progressively regaining autonomy and the ability to deliver. When dealing with a transformation like this, it’s not going to be binary. One day you have the monolith; the next day, the monolith is gone. It doesn’t really matter because what’s important is not that the monolith is gone. What’s important is that the teams have regained their ability to deliver value constantly and frequently for the business. That’s what the DORA metrics really gave us.

However, there is one mistake I made in the beginning and that I see many CTOs also making. Delivery performance metrics do nothing on their own. I hear a lot of talk about people assuming that measuring the DORA metrics and putting them on a dashboard is enough. High five, we’re done. Nothing happens. As an engineering leader, there are at least a couple of things you have to figure out. One is your expectations regarding those metrics. Do you want every team to improve on a monthly basis? Do you want to set a threshold and expect that this is the new norm for the organization? It’s a complex discussion to have with the teams to figure out what’s most important.

In our case, we went with the basics. We set a threshold on deployment frequency because that was the most important at the time. Then, when you have set those expectations, you actually have to help the team get there. You’re going to face teams that have possibly never heard about these metrics, don’t know what to do with them, and even if they understand the number and what it means, they don’t necessarily know the levers to improve. You can’t blame them because it’s not something they should necessarily know. I would say you really need to provide support to help them match the expectation. It’s probably a combination of different things.

The way we approached it at Veepee was to say, “It’s going to be a combination of an internal platform that they can use to go to production more efficiently and safely and coaching using staff engineers.” Staff engineers weren’t a role that existed when I originally joined, but they really helped in terms of evangelization and bringing the team up to speed in terms of what they could do differently to improve delivery performance.

One final thing I want to add to this is that the teams who already know what to do to get better with delivery performance aren’t going to be your worst performers anyway. They already figured it out. They didn’t wait for you to implement modern best practices. That’s why you really need to provide assistance to the teams that need it most.

Once we have talked about alignment and delivery performance, the truth is, none of this matters unless the team is healthy and capable of producing sustainable results. That ties back to the panel we had before about balancing productivity and well-being. What good are business results that you deliver on a regular basis if it comes at the expense of the team? You can’t have talent attraction and retention as an afterthought. This has to be built from the ground up to ensure you’re doing things in a way that is sustainable for the teams.

This is also a part where data can help in two ways. One is using quantitative data to support the team’s health. One example we have with customers is noticing when there are patterns that are unhealthy for a given contributor. For instance, when you see that someone is taking all the load of incidents or working nights and weekends repeatedly, putting their health at risk. The problem with these kinds of people is that they’re not going to be the ones who bring it up during one-on-ones. They’re your most committed and engaged contributors, and they typically think this is their mission. That’s great, except they might put themselves at risk by going too far. Unless you have the data to detect that these things are happening before it’s too late, you might burn out your most engaged collaborators.

The second part is using qualitative data to support the team’s health. Surveys can help understand the team’s pulse. At Veepee, we used Officevibe to conduct surveys and create engagement reports. I used them frequently, even to get the pulse on specific topics, such as asking, “Do you have confidence that this project will ship successfully? Do you have confidence that this is the right thing to do for the company?” Again, it’s about tying back into the mission of having an impact for the business and not just moving bits around and shipping code.

Having said that, back to the executive committee. I’m in the room. Everybody’s looking at me. “When do we ship?” is the only thing that matters to stakeholders. So far, what we’ve covered is that I have visibility into the team’s focus. I can tell and show with data that we are working on the right things. We are measuring delivery performance, so I know what to expect from the teams. I know our typical pace of development. We have confidence that the team is engaged. We have confidence that the team is solid. What is missing?

Any ideas? End users? Business impact? Yes, that’s absolutely an issue. I wish this was my answer. Actually, it’s the one you will have with peers in the executive committee. What we’re missing to answer, “When do we ship?” and whether we’re on track is execution tracking—having feedback loops into the execution to know what’s going well and what’s not going well.

My point is that all the things I discussed before are never a given. Engineering success is never a done deal. This is both bad and good news. It means you can’t just fix things and move forward, but it’s also what makes the job fascinating. Everything is always changing. The alignment you thought you had yesterday might be entirely gone due to a major production incident. Delivery performance might shift due to dependencies between teams that didn’t exist before. Everything changes constantly. The only question becomes: how fast are you aware of these changes, and how fast can the organization react?

This ties back to feedback loops. Feedback loops are one of the most difficult roles for the CTO. It’s about figuring out how to get the information from the field to know that the things you’re doing are the right things and minimizing surprises. I’ll give you examples. One is a quote from a French unicorn CTO, who said, “In an ideal world, I would like to participate in the standup of all my teams. This guy has 40 teams. Of course, he's not going to do it, but you get the message. He would like to be able to hear everything that is happening on the field before it's too late, before waiting for the next one-on-one or the next status update. Another example I heard is from Jean-Baptiste Kempf, who you probably know as the founder of VLC. He is also the CTO for a European cloud provider called Scaleway. The way he approaches this is by having monthly team reviews. On a monthly basis, he reviews the metrics, reviews the goals, and has a discussion with the team. He also has a weekly four-liner that the teams have to send, which is automatically consolidated into a PDF and sent to the whole company. It's just four lines, but it’s four lines that every team sends on a weekly basis.

This is just one example of how people approach staying connected to the execution to ensure everything is still on track. Probably the most common thing I see in terms of feedback loops, and I’m sure many of you are doing it, is using JIRA tickets. JIRA tickets are a very common form of feedback loop. A lot of companies mandate them from the teams because they want traceability and visibility into what’s happening. I think it’s important to understand this as a feedback loop because then you can start questioning whether it is a good feedback loop or not.

What makes a good feedback loop? A good feedback loop is one that is short, meaning that whenever something happens, you are aware sooner rather than later. The second aspect is whether it actually takes time for the team or is costly for the team to do. When we ask for status updates, monthly reviews, four-liners, etc., we are asking for work that primarily benefits us. That’s something important to keep in mind. Ultimately, this is something you are doing mostly for your own needs. Yes, it is a forcing function, and it is interesting for the teams to take a step back and think about how they’re doing and what their status truly is. But still, it’s mostly something you’re doing for you. That means you have to be very considerate in what you’re asking.

Most importantly, is it a useful feedback loop? Are you doing anything with the data that the teams are producing? This is typically where I see issues in the industry. A lot is being asked of the team, but not so much is being used afterward in the data to actually make decisions. Once you develop a habit of measuring things—alignment, delivery performance—there’s a lot you can do to get efficient feedback loops that tell you when things are off track before you actually wait for the next one-on-one or the next team review. You can tell through the alignment metrics when a team has gotten entirely sidetracked and is working on something different. You can tell sometimes with the delivery metrics when a project is just unusually slow compared to the regular pace of the team. These should be signals that you use.

Closing thoughts: Steering an engineering organization is a lot like steering a boat. You need to keep an eye on the basics. You need to know the weather, know the wind, and know if the engine is doing well. You need to care for the crew. You want to avoid mutiny and make sure that people are healthy and safe onboard. You need to stay on course. You have a goal, and you have to get there. But, last but not least, you have to get there in time because if you’re carrying winter coats, nobody’s going to celebrate if you arrive in the spring.

Thank you so much for your attention. I’m very passionate about this topic, so please add me on LinkedIn. Tell me if you agree or disagree. Let’s chat about this. Again, we have a booth here. I’d love to show you a demo of what we’re building. I’d love to continue the conversation, and you have a chance to win a Lego. Thank you very much for your attention.