This post is from the Collabnet VersionOne blog and has not been updated since the original publish date.
Measuring DevOps Performance Using a Value/Based Approach
Are you measuring the value, risk, and quality flowing through your DevOps pipelines? Here is a value/based approach to measuring DevOps performance that will help your organization better evaluate the effectiveness of its DevOps initiatives. As organizations become increasingly value/stream conscious, new metrics and insights are required to continuously streamline the flow of value through DevOps. Merging data generated during DevOps with the critical business data generated earlier in the software lifecycle can produce new and powerful insights that can help dramatically improve how DevOps teams create, validate, and deliver business value. 3 Value/Based DevOps Performance Measurement Categories 1/ DevOps Flow DevOps flow metrics describe the ease in which potential value, in the form of backlog items, flows through each DevOps “delivery phase.” These DevOps metrics identify bottlenecks in the value stream and highlight the amount of time value spends in non/value adding “wait states” at any point between developers and end users. Leveraging flow metrics can help organizations eliminate waste and accelerate time to value by decreasing the amount of time backlog items spend moving between developers and end users. 2/ Deployment Risk Deployment risk metrics objectively describe the relative risk of any incremental deployment compared to previous deployments. These DevOps metrics analyze dependencies, detect technical anomalies, and identify changes to fragile code as well a whole slew of additional proven risk factors that drive deployment risk. Leveraging risk metrics can help organizations prioritize testing activities and identify costly defects before they are deployed, while improving audit and compliance data to support quicker deployment cycles. 3/ Code Quality Code quality metrics help measure the overall quality of code over time as well as the effectiveness of testing activities — manual or automatic. These DevOps metrics highlight known defects in any potential deployment, document the ratio of new effort vs. repair effort for any given release, and connect defects with the underlying source code to support “cluster reporting,” as well as other metrics that precisely track relationships between defects and source code. Better code quality metrics help businesses maintain high levels of customer satisfaction and help the organization strike a comfortable balance between speed and quality. *Keep an eye out for our upcoming articles that will dive into the specific value/based metrics you can use to measure your DevOps flow, deployment risk, and code quality. Why Your DevOps Metrics Must Be Value/Aware As software development organizations focus on increasing enterprise agility, the importance of software value streams inevitably bubbles to the forefront. Software value streams describe the flow of business value, in the form of new software, as it flows from idea all the way to the end user. In order to measure DevOps performance and enterprise agility, you need metrics that explicitly describe the inner workings of your value streams between developers and end users. For more information about value streams, check out my recent video interview with Dean Leffingwell, co/founder and chief methodologist at Scaled Agile, Inc. [embed]https://www.youtube.com/watch?v=WrPTkJjXrAw[/embed] You can also read the blog post based on my interview with Dean Leffingwell: 6 Crucial Elements of Tracking Business Value from Strategy through Delivery The three categories of DevOps Performance Measurement — DevOps flow, deployment risk, and code quality — provide end/to/end visibility of your entire value stream. It’s important to note that none of the metrics described above can be derived using only data generated from DevOps. To produce these metrics, DevOps must be tightly integrated into the overall value stream, utilizing a single unified and correlated data model. The Problem with Traditional DevOps Metrics Each week I talk to enterprise leaders who share frustration about the lack of actionable DevOps metrics. To be sure, DevOps tools can generate tons of data and hundreds of novel metrics. Yet traditional metrics provide limited strategic insight for one critically important reason: DevOps pipelines are not the start of the software value stream; in most cases they are really just the last mile. However, these pipelines are profoundly unaware of the underlying business value, in the form of new features and fixes, flowing through them. This has resulted in a crippling misalignment between DevOps and the value/centric data generated by the business upstream. Until these two distinct segments of the enterprise value stream are joined together into a single end/to/end flow with correlated data, DevOps metrics will continue to provide very little strategic value to the business. To help illustrate this point, I’ve assembled a list of twenty standard DevOps performance metrics based on direct customer feedback and my own real/world experience. If you are familiar with DevOps, you’ve probably already dealt with many of these metrics at one point or another. Outdated DevOps Metrics
- Commits per day or period
- Commits per developer per day or period
- Number of unique code contributors
- Change volume — ratio changed vs. static code
- Builds per day or period
- Average build duration
- Average time to deploy
- Deployment frequency
- Percentage of failed deployments
- Ratio of develop units vs. test units (time, cost, headcount, etc.)
- Requirements/test coverage ratio
- Total number of automated tests
- Ratio of manual tests vs. automated tests
- New defect/ticket volume or rate
- Defect/ticket resolution rate
- Mean time to discover defects
- Mean time to restore/repair defects
- Broken build rate
- Mean time to restore/repair broken build
- Number of broken build commits by developer
Note: For this post, I have intentionally omitted system performance and feedback metrics such as system availability, usage, and response time. 5 Shortcomings of Traditional DevOps Metrics While the metrics above can be extremely helpful to DevOps teams, especially when adopting a “whole system” perspective, they can also have some major shortcomings:
- When viewed independently, they can easily miscommunicate problems and promote actions that negatively impact the software value stream.
- They do little to promote and extend enterprise agility.
- They provide few actionable insights back to the business.
- They make no attempt to describe the business value that is constantly flowing through DevOps.
- They don’t describe value flow, code quality, or deployment risk.
You can easily collect endless streams of information about changes to source code, the build/CI executions, manual and automated testing activities, building and tearing down virtual infrastructure, and artifacts as they get deployed into environments, and on and on. The problem with all this data is that it is rarely correlated back to the flow of business value. For example, your deployment automation tool can probably tell you that it just successfully deployed some binary code into some environment, but a much more important question is what backlog items did it just deploy and what “net/new” value was injected into the target environment as a result. At the end of the day, the two primary objectives of DevOps within any organization are fairly basic:
- DevOps should streamline the flow of business value, in the form of incremental software enhancements, between developers and end users.
- DevOps should facilitate fast feedback to further support business decision making.
DevOps relies on a combination of people, process, and tools to accomplish these primary objectives. A quick scan of the metrics above and it’s clear that they actually tell the business very little about how well DevOps is “performing” against its high/level mission. Metrics are fed by data and you can’t produce meaningful metrics without meaningful data. As enterprise organizations are increasingly focused on streamlining their software value streams, end/to/end, the disconnect between value creation and value delivery is short/circuiting our ability to provide metrics that really matter to the business. Value/based performance metrics allow organizations to measure DevOps in the context of its ability to build, validate, and deploy potential business value. Conclusion I hope this has encouraged you to take a deeper look at how your organization is tracking the value that is being delivered to customers. These three value/based DevOps performance measures are just the tip of the iceberg. If you want to learn more, check out the other articles in this series:
- How Measuring DevOps Performance Increases Enterprise Agility
- 3 Performance Measures That Can Help Reduce DevOps Wait Time
- 4 Basic Building Blocks Needed to Measure DevOps Flow