
This post is from the XebiaLabs blog and has not been updated since the original publish date.
Measuring the Impact: How to Use Data to Optimize Software Delivery
We're excited to reveal an updated and vastly expanded edition of our popular The Manager’s Guide to Continuous Delivery, now called The IT Manager’s Guide to DevOps: How to Drive the Business Value of Software Delivery. To celebrate the new version, we're using the XebiaLabs blog this week to highlight some of the new insights and advice coming your way from co-author, Tim Buntel. The following excerpt looks at how DevOps teams can mine the data produced by their delivery pipelines to optimize their software delivery efforts. You can download an advance copy of the complete ebook here.
Note: Those attending the 2018 DevOps Enterprise Summit in Las Vegas will get the print version of the advance copy of the book. You can register for the event here and use the promo code XEBIALABS20.
Impact to the Development Process
DevOps teams are typically focused on goals such as “improve quality” and “increase speed of delivery.” There are some key metrics that provide evidence of attainment of those goals:- Time to delivery: How long does it take for a single-line code change to get fully deployed? Answering this question helps you understand the efficiency of your process and see where bottlenecks might exist.
- Frequency of releases: How often are releases pushed live? Keeping batch size small helps drive early feedback, so it’s important to release often.
- Change volume: What value is going into production with each release? This may be measured in user stories, for example.
- Success rate: What percentage of deployments failed, resulting in an outage? This highlights the quality of both the code and the release process.
- Mean time to Recovery (MTTR): How long does it take to recover from a failed change? High-performing DevOps teams can react rapidly and respond to unfamiliar problems.
- Defects: How many defects were identified and resolved while still in development versus being released into production and fixed later?
- Team comparisons: How are teams performing against their peers? Is their performance on this list of metrics better than the average?
FREE EBOOK
The IT Manager’s Guide to DevOps: How to Drive the Business Value of Software Delivery
Whether you’re brand new to DevOps or looking to scale, this book helps you create a high-performing organization that drives business value up and time to market down. Read more.
Impact to Customers
Recent studies suggest that as little as one third of all changes actually improve outcomes for users. The rest either have no impact on desired outcomes or, shockingly, make outcomes worse! High-performing DevOps teams, therefore, focus less on time to delivery and more on time to value. The tangible and intangible benefits of a DevOps transformation for customers may include increased adoption, higher conversion rates, improved happiness, and reduced need for technical support. Some metrics to consider:- Application utilization and traffic
- Calls to APIs
- New user sign-ups
- Performance (i.e., page response time)
- Support ticket volume
- Public or social media feedback (e.g., customer tweets)
- Net Promoter Scores or other sentiment analysis techniques
- Customer churn
Impact to the Team
DevOps has productivity benefits, but it has also been shown to impact team satisfaction. The 2017 DevOps Research and Assessment (DORA) State of DevOps Report found that employees in high-performing DevOps organizations are 2.2 times more likely to recommend their organization as a great place to work. DevOps improves the development workflow for all members of a team, which can lead to higher job satisfaction and lower turnover. DevOps teams report spending less time on less enjoyable tasks, such as rework and bureaucracy, leaving more time for more creative new work that delivers more value to customers. With its focus on cross-functional, collaborative teams, employees using DevOps feel more connected to one another. Developers understand the business benefit of a feature because they work directly with the business stakeholders who drive them. Operations professionals feel less antagonistic to developers because they see that everyone on the team takes responsibility for the successful operation of the software (not just the development of it). And a host of other participants, from regulation and compliance, to security, to business managers, are no longer kept on the periphery of a project.Insight for Compliance
Compliance considerations play a key role in how software is built, especially for highly regulated enterprises, such as publicly traded companies, financial services, insurance organizations, government agencies, and many others. Teams in these situations need real-time answers to questions such as:- Is a standardized and approved software delivery process being followed?
- What changes were made to the software?
- Who made them and when?
- Were they approved?
- Did the company follow the necessary rules for each activity as dictated by SOX, PCI, and other regulations?
- Was the appropriate testing conducted?
- Were permissions properly controlled?
Ongoing Customer Conversation
The work of a DevOps team doesn’t end when a feature is released to production. If customers do not react positively to the new functionality—if it fails to deliver value—then it doesn’t matter how rapidly it was delivered. DevOps teams rely on ongoing dialogue with their user base to identify desired improvements and feed them back into the process. This feedback loop is critical for realizing the benefits of DevOps.Learn More
More from the Blog
View more

Customer spotlight: Schneider avoiding bumps in the road with DevOps adoption

How testing automation can build a culture of QA while accelerating continuous delivery
