Integrating Metrics into your Software Delivery Process with Dynatrace and XL Release

Most of the discussion around Continuous Delivery and improving your software release process today seems to focus on speed, speed, speed. Accelerating time to market is an important driver for many organizations’ CD initiatives, of course, but there’s not much point in being able to get your code to production in seconds if everything ends up broken more often than not.

In short: we think it’s critical that speed and quality are both treated as first-class citizens in the CD process. And of course it’s great to see teams adding more and more test automation to their software delivery process.
The thing is: the traditional types of tests that tend to be added first – unit, functional, integration etc. – tend not to reveal much about the architecture of your applications. This is a problem because architectural issues are key contributing factors to poor end-user experience: scalability, availability, response time etc.
Luckily, most organizations are already using one type of tool that is well placed to collect metrics relevant to the architectural quality of your applications, such as the number and size of requests made to load a web page, or the number of database queries issued to complete a transaction: APM tools.
Here, we’ll look at how XL Release works with Dynatrace to integrate service, business application and user-level metrics into your software delivery process. This allows you to get a more accurate and, importantly, much earlier picture of the architectural quality of your app.
Once the application is live, this also gives you a much more detailed insight into how your users are actually working with the application, allowing you to much more effectively determine whether the features you’ve just released are successful.

Preparation

• If you are not already a Dynatrace user, download and install Dynatrace.
• If you have not already done so, download and install XL Release. XL Release is a release orchestration tool that allows you to define, automate and improve your release process, enabling you to move towards more fully automated, efficient Continuous Delivery pipelines. See the product page for more information.
• Download the latest release of the XL Release Dynatrace plugin and install it by copying it to XLR_SERVER_HOME/plugins
• Once you have (re-)started your XL Release server, you should see the following new tasks:

1. Service-level metrics: registering a test run, retrieving the results and making automated decisions

As soon as you are running even simple local integration tests, with stubs such as an in-memory DB or a fake web service, you can collect lots of interesting metrics that indicate the “architectural health” of the application: number of DB queries issues, number of calls to the web service, size of the web service response etc.
The absolute values of these metrics are certainly useful, but what’s especially important here is to look for big changes against the baseline, whatever that baseline may be: the “small configuration change” that results in N+1 queries being issued for a collection fetch instead of a single one, for example.
If you’ve instrumented that initial integration testing environment (typically your CI server), Dynatrace’s Test Automation feature will track the above and other key metrics and automatically perform outlier detection.
Using the Register Test Run task, you can register a test run in Dynatrace straight from XL Release before calling your CI server (or your integration test harness directly). You can then retrieve the Dynatrace’s analysis using the Retrieve Test Results task and automatically or manually decide whether to abort the current pipeline run, or proceed:

2. Business application level metrics: starting, stopping and reviewing a recording

Once you’ve deployed your candidate application(s) to a more “realistic” environment where they’re actually talking to “real” versions of each other and external services, you’ll probably already be subjecting them to performance and stress tests to gauge whether they will behave acceptably in Production.
APM tools like Dynatrace are very common choices to monitor and report on application performance characteristics in such a setting, so integrating them into your pipeline at this stage is a natural choice.
From XL Release, you can use the Start Recording task to automatically tell Dynatrace to kick off a data recording session just before you start your performance test run. The description of the recording session links straight back to the XL Release release, so everyone can immediately see what this session is about.
You can use the Stop Recording session to immediately end the recording once the performance test has been completed, ensuring that you don’t need to mentally “trim” the Dynatrace reports and visualizations if the session runs on for a while before someone remembers to stop it.
XL Release also provides direct links to the Dynatrace reports that the team intends to review, to ensure everyone is quickly on the same page and can efficiently make a decision on whether to proceed with the release:

3. User-level metrics: registering a deployment and monitoring user experience

Easily collecting the relevant quality data you need to go live, and making reliable go/no-go decisions based on that data efficiently, is naturally a very important component of your release pipeline.
Ultimately, however, what’s hopefully even more interesting is how your new application versions are actually faring once they’re “out there” in Production. How are your users responding to the new features you’ve just released?
Here’s where you quickly want to see a clear “before/after” view that shows you the user behavior stats you need to evaluate the success of your release – whether that’s the level of Android vs. iOS app usage, the number of successfully completed transactions from a certain region, the average response time in a particular geography, or any other user-level metric.
Dynatrace’s User Experience Management already records and visualizes such metrics. For a “before/after” view, however, you need to determine that moment relative to which you want to make the comparison – usually, the moment when your new application is fully live in Production.
XL Release’s Register Deployment task does this for you automatically, allowing you to quickly jump to the right “before/after” view (you might even have multiple, for different milestones in a phased go-live, for example):Details of the release, and a link back to XL Release, are automatically added to the event record in Dynatrace, so everyone working with Dynatrace can quickly figure out what each deployment was about, and where to find more details if needed.

4. Putting it all together

Here’s how those three components fit in the overall pipeline we’ve been using as an example here:This is only one of many possible release process flows, of course. XL Release leaves you complete freedom to define and improve your releases at your own pace, moving towards metrics-driven Continuous Delivery pipelines in the way that makes most sense for you.
If you’re going to be at the PERFORM 2015 conference, by the way, catch me and Dynatrace ace Andi Grabner talking about getting started with metrics-optimized in much more detail at 1:45pm on Wed, Oct 14th (Continuous Delivery track). Hope to see you there!