Measuring the Impact: How to Use Data to Optimize Software Delivery

We're excited to reveal an updated and vastly expanded edition of our popular The Manager’s Guide to Continuous Delivery, now called The IT Manager’s Guide to DevOps: How to Drive the Business Value of Software Delivery. To celebrate the new version, we're using the XebiaLabs blog this week to highlight some of the new insights and advice coming your way from co-author, Tim Buntel. The following excerpt looks at how DevOps teams can mine the data produced by their delivery pipelines to optimize their software delivery efforts. You can download an advance copy of the complete ebook here.

Note: Those attending the 2018 DevOps Enterprise Summit in Las Vegas will get the print version of the advance copy of the book. You can register for the event here and use the promo code XEBIALABS20.

A core principle of DevOps is fast feedback. The sooner a determination can be made that the value created is delivering in the way the business expects for customers, the sooner it can be enhanced—or changed—if necessary. The same goes for the process of developing and delivering that value. An early understanding of what works and what doesn’t work in the development process enables teams to invest more in high-value activities and stop or modify those that fail to deliver. These ideas are not new, but the amount of data available with which to make those measurements is significantly greater today than ever before. As more of the development process is automated, more data is produced. And better instrumentation of applications produces more data about how users interact with a system and get value from it. Business leaders need to mine this valuable data to understand the impact that a DevOps transformation is having on both the customers and the development process itself.

Impact to the Development Process

DevOps teams are typically focused on goals such as “improve quality” and “increase speed of delivery.” There are some key metrics that provide evidence of attainment of those goals:

• Time to delivery: How long does it take for a single-line code change to get fully deployed? Answering this question helps you understand the efficiency of your process and see where bottlenecks might exist.
• Frequency of releases: How often are releases pushed live? Keeping batch size small helps drive early feedback, so it’s important to release often.
• Change volume: What value is going into production with each release? This may be measured in user stories, for example.
• Success rate: What percentage of deployments failed, resulting in an outage? This highlights the quality of both the code and the release process.
• Mean time to Recovery (MTTR): How long does it take to recover from a failed change? High-performing DevOps teams can react rapidly and respond to unfamiliar problems.
• Defects: How many defects were identified and resolved while still in development versus being released into production and fixed later?
• Team comparisons: How are teams performing against their peers? Is their performance on this list of metrics better than the average?

FREE EBOOK

The IT Manager’s Guide to DevOps: How to Drive the Business Value of Software Delivery

Whether you’re brand new to DevOps or looking to scale, this book helps you create a high-performing organization that drives business value up and time to market down. Read more.

Impact to Customers

Recent studies suggest that as little as one third of all changes actually improve outcomes for users. The rest either have no impact on desired outcomes or, shockingly, make outcomes worse! High-performing DevOps teams, therefore, focus less on time to delivery and more on time to value. The tangible and intangible benefits of a DevOps transformation for customers may include increased adoption, higher conversion rates, improved happiness, and reduced need for technical support. Some metrics to consider:

• Application utilization and traffic
• Calls to APIs
• New user sign-ups
• Performance (i.e., page response time)
• Support ticket volume
• Public or social media feedback (e.g., customer tweets)
• Net Promoter Scores or other sentiment analysis techniques
• Customer churn

Impact to the Team

DevOps has productivity benefits, but it has also been shown to impact team satisfaction. The 2017 DevOps Research and Assessment (DORA) State of DevOps Report found that employees in high-performing DevOps organizations are 2.2 times more likely to recommend their organization as a great place to work. DevOps improves the development workflow for all members of a team, which can lead to higher job satisfaction and lower turnover. DevOps teams report spending less time on less enjoyable tasks, such as rework and bureaucracy, leaving more time for more creative new work that delivers more value to customers. With its focus on cross-functional, collaborative teams, employees using DevOps feel more connected to one another. Developers understand the business benefit of a feature because they work directly with the business stakeholders who drive them. Operations professionals feel less antagonistic to developers because they see that everyone on the team takes responsibility for the successful operation of the software (not just the development of it). And a host of other participants, from regulation and compliance, to security, to business managers, are no longer kept on the periphery of a project.

Insight for Compliance

Compliance considerations play a key role in how software is built, especially for highly regulated enterprises, such as publicly traded companies, financial services, insurance organizations, government agencies, and many others. Teams in these situations need real-time answers to questions such as:

• Is a standardized and approved software delivery process being followed?
• What changes were made to the software?
• Who made them and when?
• Were they approved?
• Did the company follow the necessary rules for each activity as dictated by SOX, PCI, and other regulations?
• Was the appropriate testing conducted?
• Were permissions properly controlled?

Automated CD pipelines and other DevOps practices make answering these questions much easier than in traditional development processes. And individuals responsible for compliance are part of a DevOps team, so they no longer need to rely on second-hand information about how a project was delivered. The DevOps platform further supports compliance by applying controls to the CD pipeline itself. Steps can be made mandatory, and access controls can be applied in granular ways if needed.

Ongoing Customer Conversation

The work of a DevOps team doesn’t end when a feature is released to production. If customers do not react positively to the new functionality—if it fails to deliver value—then it doesn’t matter how rapidly it was delivered. DevOps teams rely on ongoing dialogue with their user base to identify desired improvements and feed them back into the process. This feedback loop is critical for realizing the benefits of DevOps.

Learn More

• DevOps 101: What is a DevOps Platform?
• DevOps, Pipelines, Toolchains, and Application Release Orchestration: A Short Primer
• Scale DevOps Across the Enterprise
• The Periodic Table of DevOps Tools