Measuring the Impact: How to Use Data to Optimize Software Delivery
We're excited to reveal an updated and vastly expanded edition of our popular The Manager’s Guide to Continuous Delivery, now called The IT Manager’s Guide to DevOps: How to Drive the Business Value of Software Delivery. To celebrate the new version, we're using the XebiaLabs blog this week to highlight some of the new insights and advice coming your way from co-author, Tim Buntel. The following excerpt looks at how DevOps teams can mine the data produced by their delivery pipelines to optimize their software delivery efforts. You can download an advance copy of the complete ebook here.
Note: Those attending the 2018 DevOps Enterprise Summit in Las Vegas will get the print version of the advance copy of the book. You can register for the event here and use the promo code XEBIALABS20.A core principle of DevOps is fast feedback. The sooner a determination can be made that the value created is delivering in the way the business expects for customers, the sooner it can be enhanced—or changed—if necessary. The same goes for the process of developing and delivering that value. An early understanding of what works and what doesn’t work in the development process enables teams to invest more in high-value activities and stop or modify those that fail to deliver. These ideas are not new, but the amount of data available with which to make those measurements is significantly greater today than ever before. As more of the development process is automated, more data is produced. And better instrumentation of applications produces more data about how users interact with a system and get value from it. Business leaders need to mine this valuable data to understand the impact that a DevOps transformation is having on both the customers and the development process itself.
Impact to the Development ProcessDevOps teams are typically focused on goals such as “improve quality” and “increase speed of delivery.” There are some key metrics that provide evidence of attainment of those goals:
- Time to delivery: How long does it take for a single-line code change to get fully deployed? Answering this question helps you understand the efficiency of your process and see where bottlenecks might exist.
- Frequency of releases: How often are releases pushed live? Keeping batch size small helps drive early feedback, so it’s important to release often.
- Change volume: What value is going into production with each release? This may be measured in user stories, for example.
- Success rate: What percentage of deployments failed, resulting in an outage? This highlights the quality of both the code and the release process.
- Mean time to Recovery (MTTR): How long does it take to recover from a failed change? High-performing DevOps teams can react rapidly and respond to unfamiliar problems.
- Defects: How many defects were identified and resolved while still in development versus being released into production and fixed later?
- Team comparisons: How are teams performing against their peers? Is their performance on this list of metrics better than the average?
FREE EBOOKRead more.
Impact to CustomersRecent studies suggest that as little as one third of all changes actually improve outcomes for users. The rest either have no impact on desired outcomes or, shockingly, make outcomes worse! High-performing DevOps teams, therefore, focus less on time to delivery and more on time to value. The tangible and intangible benefits of a DevOps transformation for customers may include increased adoption, higher conversion rates, improved happiness, and reduced need for technical support. Some metrics to consider:
- Application utilization and traffic
- Calls to APIs
- New user sign-ups
- Performance (i.e., page response time)
- Support ticket volume
- Public or social media feedback (e.g., customer tweets)
- Net Promoter Scores or other sentiment analysis techniques
- Customer churn
Impact to the TeamDevOps has productivity benefits, but it has also been shown to impact team satisfaction. The 2017 DevOps Research and Assessment (DORA) State of DevOps Report found that employees in high-performing DevOps organizations are 2.2 times more likely to recommend their organization as a great place to work. DevOps improves the development workflow for all members of a team, which can lead to higher job satisfaction and lower turnover. DevOps teams report spending less time on less enjoyable tasks, such as rework and bureaucracy, leaving more time for more creative new work that delivers more value to customers. With its focus on cross-functional, collaborative teams, employees using DevOps feel more connected to one another. Developers understand the business benefit of a feature because they work directly with the business stakeholders who drive them. Operations professionals feel less antagonistic to developers because they see that everyone on the team takes responsibility for the successful operation of the software (not just the development of it). And a host of other participants, from regulation and compliance, to security, to business managers, are no longer kept on the periphery of a project.
Insight for ComplianceCompliance considerations play a key role in how software is built, especially for highly regulated enterprises, such as publicly traded companies, financial services, insurance organizations, government agencies, and many others. Teams in these situations need real-time answers to questions such as:
- Is a standardized and approved software delivery process being followed?
- What changes were made to the software?
- Who made them and when?
- Were they approved?
- Did the company follow the necessary rules for each activity as dictated by SOX, PCI, and other regulations?
- Was the appropriate testing conducted?
- Were permissions properly controlled?