Continuous Delivery in the Real World: Moving to CD with a complex portfolio and compliance requirements
In a previous post, I talked about how helping users handle the challenges of adopting Continuous Delivery in the real world is one of our key goals at XebiaLabs. I shared excerpts from one particular thread and would like to continue in the same vein here. This discussion focuses specifically on the challenges of a landscape with complex dependencies, and of how to introduce CD across a portfolio of applications with very different levels of CD maturity. "As a starter, Jez Humble & co have some good thoughts on release management. Their context, though, is a single application, probably a small/simple one relative to our enterprise application complexity level.Our release and change management functions are designed to coordinate multiple concurrent changes in interdependent application suites. The application architectures span many languages and platforms, some CI/CD enabled, some not. For example, three major enterprise apps must deploy changes to Prod on a single weekend, in order to maintain interoperability. The changes have interdependencies, so individual app deployments in CD-style would break the overall system in Prod. How do we manage this situation with CD, short of refactoring the entire app portfolio?"It's not very helpful, indeed, that so much has been written about the single pipeline merrily pushing updates to production for an app with no dependencies whatsoever when that case so seldom matches reality. Complex dependencies between components or services are not just a challenge when dealing with "old-fashioned" applications, too. Ironically, "state-of-the-art" microservice architectures can increase the dependency management headache, certainly when the goal is to update them more and more frequently. At a conceptual level, our experience is that addressing this challenge means looking beyond individual pipelines and modelling the entire "pipeline dependency graph", ensuring the release process (as represented by a pipeline) for each component takes the appropriate dependencies into account. Especially if some kind of manual approval is required, it often makes sense to facilitate this by synchronizing the last stages of the pipelines for the individual components using a "fan-in" design or release trains.In terms of our tools, we already support dependencies between pipelines in XL Release, as well as manual steps for the validation activities we commonly see in current release processes as things get close to Production. Additionally, a focus for both XL Deploy and XL Release moving forward is more support for groups of (simpler) deployments and pipelines. Current architecture and delivery trends are shifting the challenge from complex single deployments and pipelines to flows of simpler, interdependent releases. We're looking to make sure that users at both ends of the spectrum, or (as is likely to be the case for most) somewhere in between, can derive maximum value from our tools."Release and Change Management also help our application teams maintain compliance with various enterprise policies. Those compliance requirements still exist in a CD world for us. How does CD deal with that?"From my experience, that depends quite strongly on which definition of CD you subscribe to. If one views CD as "a single, fully-automated pipeline to Prod or nothing", handling the interaction with Change and Release Management (CM & RM) is challenging. I try to look at Continuous Delivery more as a powerful means of making software more relevant and valuable for customers by enabling quicker and more focused release cycles. From that perspective, including steps in the pipeline that ensure compliance with CM & RM requirements is not "incompatible with CD" in any way, even if those steps are manual. In our tools, we try to allow for steps such as the creation and verification of change tickets to be automated as much as possible, but that's simply an option that's available. Of course, meeting CM & RM requirements does introduce some challenges from a CD perspective. For example, since, as you mention, Change reviews are often scoped to cover all pending changes to an environment, it is usually necessary to "batch" the changes. This can be done by having all pipeline wait on a shared gate or through a fan-in/release train layout where the individual component pipelines merge into one for the last mile to Production. But even if CM procedures can be integrated into your pipeline graph in this way, they can still represent a bottleneck, of course. As with any other manual step, Change reviews will likely limit the frequency at which your pipelines can run. The question here is whether that is really a problem. What counts as acceptable throughput depends on the scenario (a lot of our users would not want their XL Deploy and XL Release servers to be updated every 30min, for example), and increasing the frequency of releases is in any case only one of many possible goals for a Continuous Delivery initiative. I would say that trying to figure out a way in which the release process can be further accelerated while remaining in compliance with legal requirements and company policy - which is certainly possible, given the examples of highly-regulated financial organizations with fully advanced delivery pipelines such as LMAX - only makes sense if the frequency you can achieve with the current process isn't good enough."Finally, we are looking at a few years of transition from where we are today to a continuous delivery approach for the majority of our apps. How do we manage the transition, when some apps are on CD and some are not?"Again, I think it depends a lot on what is considered being "on CD". My view is that, since every app has a release process of some kind, every app is "on CD" to some extent. What differs (widely) is the degree of automation and standardization of the release process/pipeline and, closely related to that, the frequency at which the process/pipeline is and can be run. Since, in a portfolio of interdependent apps as described above, pipelines will seldom be able to run in isolation, improvement will mean identifying the bottlenecks in the overall graph of dependent pipelines and iteratively addressing the most critical ones. Trying to implement CD by "going all-out" with one project at a time is not, in our experience, a highly efficient way of transitioning to CD in a landscape with many dependencies.In other words, the overall pipeline graph in any real-world situation will be a mix of individual pipeline at varying levels of automation and standardization - "CD-ness", if you like. In order to adopt the incremental approach above, your tooling obviously also needs to support such a "hybrid" landscape, allowing for small changes, such as automating a couple of very time-consuming steps, at a time. Unsurprisingly, a key goal for us for XL Release is to support precisely such a scenario and, if possible, to accelerate the transition process by highlighting bottlenecks and providing appropriate metrics to document the improvement not just of individual pipelines, but the overall release process.