What does DORA stand for?

When trying to go beyond the classic question of team velocity, or developers' productivity, we land quite easily on the DORA Metrics topic. The DORA framework, which stands for DevOps Research and Assessment, owe their popularity to the influential book Accelerate, which played a crucial role in promoting these four key performance indicators. Since 2018, they have gained significant recognition across the industry, and are now widely utilized in development analytics.

As a reminder, what are DORA metrics?

They are four devops metrics: lead time for changes, deployment frequency, change fail rate, mean time to recover, that are perceived as the key metrics for evaluating the efficiency of an engineering organization. The DORA KPIs play a key role in addressing DevOps challenges like bottlenecks, testing issues, production issues, incident management...

How DORA metrics can help engineering managers to monitor team performance ?

As more and more organizations delve into these concepts, it becomes evident that relying solely on the DORA framework is not a guaranteed path to ensuring the success of an engineering team. A nice DORA metrics dashboard won't be sufficient to build a fulfilling environment for your developer teams, and set up an authentic and engaging DevOps culture.

At Echoes, we believe that DORA metrics could be used as a reference for continuous improvement, more than as targets, or a comparison tool between teams, or a sole measurement of a team performance. We like to use the metaphor of a runner using Strava to measure his performance, and try to improve week after week. We see DORA as the Strava metrics for a software development team. In another words, it is a relevant framework for working on being the best version of ourselves.

Engineering managers should have a routine based on DORA reports. Every week, it makes sense to check trends, analyze increase or decrease of lead time for changes, or MTTR, and communicate about it to the team. Follow-ups seem often more relevant than trying to reach a target defined by a benchmark, context is key here. Custom dashboards should display graphs and trends, more than figures and values, and be a starter for constructive discussions during rituals as dailies, or retro: "How are we doing this month, in comparison to earlier? Are we better, or not? How could we improve? How could we be a better version of ourselves?"

What should engineering managers pay attention to?

DORA metrics are valuable monitoring tools for engineering performance. By utilizing these metrics, you can effectively identify when a team is pulling away from flow efficiency. They allow you to proactively address potential issues and bottlenecks, before they become major obstacles in the development and deployment process. With DORA metrics, you gain the ability to stay ahead of the curve of software delivery, and ensure an optimal engineering workflow, by covering DevOps performance issues.

So, as an engineering manager, what should you pay attention to:

A minor increase in Lead Time For Change

Metric: The lead time for change refers to the duration it takes for a code modification to move from being committed to being deployed in production.

Striving for a minimal lead time for changes is crucial for gauging your organizational performance, even if it's important to note that the definition of "low" can vary between organizations and even among different teams.

What to look for: One important aspect to pay attention to is a potential increase in lead time. This can serve as an indicator of possible inefficiencies or obstacles that might exist within the development workflow. A performing software development team is reactive, and should be able to implement changes quickly. By proactively identifying and addressing issues with lead time at an early stage, teams can effectively mitigate any negative impact they may have on delivery performance. The lead time metric is the most widely recognized DORA metric.

A small decrease in Deployment Frequency

Metric: This KPI measure how often code changes are successfully deployed to production. As lead time, deployment frequency is focused on measuring team velocity.

Tracking the frequency of new deployments in a production environment allows teams to identify patterns and trends, enabling them to evaluate the effectiveness of their deployment pipeline. Targets should be high, teams should deploy everyday, in an ideal environment. Production deployment encompasses areas such as PR size, CICD, test coverage and automated testing, among others.

Engineering managers should prioritize a high deployment frequency, as it serves as an indicator of strong team performance, demonstrating the ability to promptly respond to user feedback, among other factors.

What to look for: When monitoring deployments, it is crucial to remain vigilant for any decline. A decrease may serve as a red flag, signaling potential underlying issues within the workflow. These issues could stem from problems with the release process, code stability, automated testing, or integration challenges.

By promptly identifying and addressing these concerns, teams can anticipate issues that might get worse with time, effectively mitigate further delays and guarantee a more seamless development process. It is essential to prioritize early detection and resolution to maintain a productive and efficient workflow.

A rise of Mean Time to Recover (MTTR)

Metric: MTTR is a key metric in incident management. It measures the average time it takes to restore service after a failure. Analyzing the duration of incident resolution and improving time to restore service helps organizations in pinpointing areas for enhancement, and optimizing their workflows. Obviously, engineering managers are looking for low targets in this area. The smaller is time to restore, the better are performance outcomes.

What to look for: a rise in Mean Time to Repair (MTTR) can be an indication of challenges in incident response or resolution processes. It is crucial to address these issues early on in order to minimize downtime, and enhance the overall reliability of the system, as the impact on business outcomes might be direct.

Mean time to recovery is a relevant indicator of organizational performance and engineering efficiency. Taking proactive measures to enhance incident response and resolution are good devops practices that greatly influence the system's performance and uptime.

A high Change Failure Rate?

Metric: CFR is the percentage of changes that result in a failure in production. Like time to restore, Change fail rate is focused on code quality. Change Failure Rate assists teams in gaining confidence that their deployments to production contribute to system enhancement rather than causing issues or failures.

The optimal failure rate is zero, and impossible to achieve. Nonetheless devops team aim at achieving the lowest rate possible. As per DORA research, elite performers maintain a change failure rate under 15%.

What to look for: A high change failure rate, or an increase in change failure rate, could indicate potential problems with code quality, automated testing procedures, CICD issues, or deployment pipeline issues. Maybe your team should use feature flag in order to make deployment processes easier, and disconnect deployments from releases.

However, it is crucial to recognize and address these issues early on to avoid common problems and outages that may arise as a result, and that will have a direct impact on business performance.

Detect friction before it's becoming an issue

By regularly monitoring DORA metrics, an engineering team can gain valuable insights into its own performance. This monitoring process allows team members to identify areas where they can improve both individually, and as a cohesive unit.

Moreover, it enables them to quickly identify any emerging issues or challenges that may hinder their progress. Early identification of problems is crucial for an engineering team as it allows for prompt corrections and improvements, and optimize any negative impact that may have on business performance. This proactive approach fosters a positive developer experience and promotes a culture of engineering excellence.

However, it is important to note that the specific metrics and their corresponding thresholds may vary depending on the organization, or even the team and its unique goals. Benchmarks are useful but not a rule. Therefore, it is advisable for each team to establish their own targets based on their specific needs and aspirations. After all, as said earlier, DORA metrics can also be seen as a way to track a team improvement vs itself, as a runner track his performance in order to improve.

But DORA, and only DORA?

Since 2018 and the release of Accelerate, DORA metrics have proved many times how relevant they are, and became a standard of the engineering industry. It's a reference in development analytics.

Although these metrics offer very valuable insights, at Echoes, we believe that the Accelerate metrics are just one component of a complex puzzle when it comes to support software performance, and engineering efficiency. DORA metrics can be supplemented with additional data to effectively guide an engineering organization toward success.

Engineering success in 3 concepts: alignment, performance, and an healthy environment

At Echoes, we firmly believe that the success of an engineering organization is built upon three essential pillars. These pillars encompass alignment with business goals, such as OKRs for instance, engineering performance, and the cultivation of a virtuous and respectful environment for engineers. By focusing on these key elements, engineering leaders may have a better visibility on how their teams are progressing, and get better leads on how to drive them to maximize their achievements and deliver impact.

We develop this point of view in this article, you should take a look at it.