A DevOps Pipeline Map Shows What Your Organization Has Been Overlooking

Implementing DevOps to streamline software development offers expanded benefits, including the promotion of collaboration and sharing between teams across both development and operations. However, many organizations take their DevOps processes for granted and stop short of fully documenting them, end-to-end. One way to gain control and visibility over the DevOps process is to document it. Creating a formal chart can show the specific steps and tools used to progress through each build. As a result, organizations can have a more efficient end-to-end pipeline to deliver value and better products to their customers.

Without a formal diagram of your DevOps pipeline, an organization can overlook major roadblocks to efficient value delivery. Ignoring a key step in the process — such as virtual machine (VM) or container image creation for continuous integration and continuous deployment (CI/CD) — can cause an organization to “paper over” the role this step plays, as well as the tools that run it. This can create blind spots, and inefficient practices ripe for optimization may linger, potentially impacting key performance metrics like overall change lead time.

By charting the end-to-end DevOps pipeline, IT leaders can identify relationships between tool environments and how it all fits together into a seamless whole. We addressed this when we discussed the need to improve integrations in your DevOps toolchain. Diagramming your pipeline is the key first step in determining the urgency of integration between tools or process stages.

Solutions like Digital.ai Release allow you to map your DevOps pipeline, tool by tool, to get an accurate look at how work and value flow through it. Once you map your DevOps pipeline, you can stop taking bottlenecks and pain points for granted and start looking towards changes that can improve value delivery. Example outcomes include increased automation, continuous integration/deployment (CI/CD), continuous testing, and a successful shift left for security, compliance, and governance objectives.

What does it mean to diagram your DevOps pipeline?

Physical pipelines, such as those in the oil industry, are a low-maintenance solution for work (or value) to flow through from one destination to another. There’s no need for manually intensive processes — like carrying oil in buckets or on trucks — when the pipeline facilitates smooth, low-effort flows. 

Just like physical pipelines, DevOps pipelines serve as infrastructure, and not just processes.

“A DevOps pipeline is the bread-and-butter of the DevOps process, the term is used most commonly to discuss the tools, processes, and automation frameworks used to build software artifacts,” says Bryant Son, Senior Consultant at Red Hat.

When an organization’s DevOps pipeline infrastructure is poorly understood, it can be impossible to measure overall performance and obtain insights on how to improve it. Often, changes made to processes in one area of the pipeline negatively impact other areas. The saving grace is having the relationships fully mapped out and understood. Other times, key stages in the process that get overlooked, such as the VM provisioning stage mentioned above, can contribute to change delays or overall process quality issues until they are properly accounted for.

Mapping your DevOps pipeline as an exercise allows your organization to take “inventory” of all the steps and tools needed to progress through a full development cycle. This process involves going through each step and each tool to track the flow of work. Then, a complete diagram is generated, accounting for all stages of work and all tools needed to complete the process. Taking an honest look into your tool stack to include every single tool needed will bring visibility to these sometimes invisible tools that could impact change performance. Inviting input from all teams ensures that no steps are overlooked. 

As an example, one step that’s easy to overlook is provisioning a network cluster for development environment testing and sandboxing. If you don’t include this step in your pipeline diagram, you can also forget to include the tools needed to perform it. Without this step documented, it’s possible that a project manager may fail to recognize a weakness in their product’s security envelope. The diagram can also bring to light issues redundant quality assurance steps in the toolchain, letting you accomplish them earlier during the build phase, and not repeat them later.

Mapping every tool and step of a product pipeline can sound cumbersome, but organizations that use a solution like Digital.ai Release can smoothsimply pick the tools used and have the system auto-map the entire pipeline. IT leaders can choose from a host of best-of-breed tools available, finding the ones they use using a simple point-and-click interface to add the tools, one-by-one. If they do not see a tool they use, they can build custom libraries using a set of common tool traits. The program automatically configures how your pipeline would look given the tools selected, allowing for adjustments based on your unique DevOps environment.

Once you have a full view of your pipeline, you can begin managing it as a whole, working towards smoothing out bottlenecks, increasing automation, and integrating shift-left priorities.

A DevOps pipeline map shows what your organization has been overlooking

In a typical DevOps process mapping exercise, while some steps may immediately spring to mind, others can be taken for granted or overlooked. An IT leader may also have the tendency to lump several steps together in a big mass that doesn’t accurately represent what really happens, as happens in the case of, “build, test, deploy”. In actuality, there are intermediate stages that tend to get ignored, such as the need for integration and version control into your code repository, functional testing, then actual feature changes.

With value stream mapping, what often comes to light is that the areas of the pipeline that are easy to recall are the focus of the majority of process improvement and innovation. The other steps — the forgotten ones — can languish. But each step is usually somehow integral to the delivery of value to end customers and stakeholders alike.

Organizations can also tend to overlook major pain points in their process. Here’s a case study of an extreme example: a development team at a large commercial bank had to deal with a server that would crash when multiple deployments were made at once. Development leads had to communicate and coordinate with one another to ensure no one else was going to deploy at that moment. 

In addition, one leader noted that “there were problems with low code test coverages, cumbersome manual deployment processes, and no way to track code deployments with a defined task or a user story.”

In a case like this, without taking inventory of every step and mapping it as one continuous pipeline, it is difficult to account for these pain points and the size of their impact on the teams’ ability to deliver value. 

Taking inventory also allows you to look beyond obvious targets for automation. IT Organizational Consultant Manuel Pais says, “In large organizations, the longest wait times are often at the boundaries between teams where artifact and resource hand-offs take place. Fixing them can be hard, possibly requiring changes in workflow and team organization. But the benefits can be 10 or 100 times higher than automating only the build-test-deploy functions.” 

Mapping a pipeline using actual tools takes away convenient abstractions and reflects the reality of daily development work. For example, your pipeline will include the web application server you need to live-test the development build. By testing the development build, you may reveal a bottleneck created by the final build test review (testing as a stage) compared to continuous testing used throughout the build process.

Leverage your DevOps pipeline map to start making positive changes

Returning to the bank case study referenced above, once the pipeline was mapped, teams could start adding improvements. 

• VM and application servers were more readily provisioned
• The teams integrated build environment with APM and testing to improve workflow, visibility, and automation 
• The teams widened code testing coverage, improving QA while reducing the amount of potential escaped defects 

“This side project was hugely successful: we almost fully automated the development pipeline, we achieved nearly 100% uptime on our development server, we could track and improve code testing coverage, and the Git branch could be associated with the deployment and Jira task,” says Bryant Son.

In more general terms, organizational benefits of pipeline mapping can include:

• Improved understanding between teams and stakeholders about the actual pipeline processes
• Improved visibility of the net performance of the pipeline as well as within individual stages
• Commitment to creating streamlined pipeline infrastructure that includes convenient integrations and automations that hasten time-to-value while improving efficiency
• Faster feedback to allow teams to pivot strategy or change processes to fix ongoing issues for customers as well as internal teams

Finally, it is important to note that pipelines are meant to help DevOps teams, but these tools are not replacements for human workers. Gilad David Maayan of the AgileConnection community says: “make sure that your pipeline supports and enhances team practices and workflows. Your tooling and configurations should not impede workflows or require teams to completely change processes.” 

What Maayan means is that people’s intended workflows should define the tools selected ⁠— not the other way around.

Building a stronger pipeline makes work easier and products better. Start by visualizing your pipeline as it exists, account for all of your tools, and then move towards an improved state that can deliver better value to customers and stakeholders alike.

Learn the principles and practices to optimize the application of people, process, and technology when you download our Value Stream Management for the Digital Age eBook.