The deployment-sized hole between Continuous Integration, Continuous Delivery and Provisioning tools

As a developer and “honorary” (nobody else wanted to do it) build engineer, I was always dissatisfied with the amount of glue I had to apply to stick our Continuous Integration or Continuous Delivery tool, whether Jenkins, Bamboo or Go (XL Release wasn’t around at the time 😉 ) to our provisioning tool (Puppet, Chef or the like), which we also used for the application tier.
There was simply a fundamental impedance mismatch between the two types of tools:

1. We needed an application-oriented model for our services, which consisted of multiple components (“traditional” or micro-service style) spread across various machines. The provisioning tools’ domain models all felt too machine-centric.
2. We were looking for clean and easy integrations with our build, packaging and Continuous Integration/Continuous Delivery tool of choice. Since there often weren’t any, we ended up with a bunch of custom tasks and script steps.
3. Our pipelines were – and still are – based on “directive automation”, i.e. we invoked whatever tool(s) were needed for a step, kept track of the process and status and continued (or not) once it had completed. This did not match the “eventually consistent” style of the server mode of tools like Puppet or Chef at all well.
   That mode was great from an operational perspective (“Make sure all my x-thousand machines have the right configuration – I don’t need to watch it happen, just get it done!”), but not really compatible with our delivery pipeline. We sometimes used a type of polling approach (Urgh! And brittle, too), but often ended using the single-node model (chef-solo or standalone puppet) for our pipeline. That worked, but didn’t seem very much like the way the tools were ideally supposed to be used.
4. For the deployment steps in our pipeline, we needed to orchestrate actions across multiple nodes, respecting dependencies but also parallelizing where possible for efficiency. This resulted in a lot of glue scripting, both to describe which kind of orchestration order was required, as well as to actually carry it out.

This “deployment-sized” hole was one of the many motivating factors for XL Deploy, and indeed we use XL Deploy internally also in setups very similar to the above scenario (more on how we integrate Jenkins, XL Deploy, Puppet and Chef in coming blog posts).Packaging and deploying an application consisting of Puppet manifests and Chef cookbooks (Docker container requests also supported, by the way!). Easy integration with Jenkins; XL Deploy takes care of targeting the manifests/cookbooks at the appropriate hosts and all the orchestration of the invocations. Piece of cake!From the many blog posts and forum threads out there that talk of a Continuous Integration/Continuous Delivery tool and provisioning tools as the default delivery stack, however, you’d be tempted to conclude that this problem has been solved. Based on what we see some of our users doing with XL Deploy, this is evidently not the case.
Given the perhaps unsurprising perception of bias on the part of a deployment automation vendor like ourselves, we’re not especially surprised that acceptance of the deployment and orchestration gap has taken this long. Which makes it all the more interesting to finally see growing recognition, both among the Devops practitioners we speak to at events such as devopsdays, and in discussions such as this one, that there most definitely is a place for a deployment automation tool in the standard delivery stack.
I just wish such tools had been around when I was assembling delivery toolchains!