This post is from the XebiaLabs blog and has not been updated since the original publish date.
Webinar Q&A: From Chaos to Compliance: The New Digital Governance for DevOps
At our webinar From Chaos to Compliance: The New Digital Governance for DevOps (which you can watch here on-demand), we had TONS of questions come in - so many, in fact, that we couldn't answer them all live. Now, we have your answers! Enjoy this Q&A with insight into ALL of your questions from hosts Dan Beauregard, VP, Cloud & DevOps Evangelist at XebiaLabs and Charles Betz, Forrester Principal Analyst.
Q: How do you use a flexible governance model in companies with strict regulatory controls (water- and electric-critical infrastructure)?
Charles: It depends on the nature of the workload. For the most critical operational technology, governance will likely remain conservative. But there should always be a clear, quantified risk discussion. Too often a general culture of risk aversion bleeds over into heavy-handed governance for much less critical workloads. Dig into the roots of any statement such as "compliance says" etc. Do they really?
Q: Please suggest some key compliance practices that we can implement.
Charles: Build a fully compliant "hello world" system and provide it to your developers when they are starting to build something new.
Dan: The best advice I have comes directly from the auditors themselves. Have security/compliance teams be part of the process. Not only should you involve them early in the planning process, but also involve them in the design of your release pipelines and delivery processes. This will go a long way in helping your developers become "compliant by default".
Q: With more and more automation, if Change Approval Board is on the decline, what is replacing it to control the risk associated with any change?
Charles: To be clear, the change process remains. More "standard" or pre-approved changes. Smaller changes which are easier to roll back. Broader automated testing and more responsive monitoring to quickly detect issues. Analytics for "release readiness" and change risk scoring. Virtual CABs - face-to-face no longer required if everyone signs off. Robust automation so that machines perform the risky deployment steps, not error prone humans.
Q: How would you build a business case to implement an ARO tool? (In our case, upper management still don't see the need - they need numbers to prove the investment.)
Charles: It will minimize the cost of delay of your new software features and also reduce risk. You can, and should, quantify the cost of delay for your critical software efforts. What is it costing you to not be responsive to changing markets and customer expectations?
- Leveraging ARO to build consistent and repeatable release pipelines will free up your developers from building and maintaining one-off pipelines allowing them to focus on building great software.
- ARO can also provide deep visibility into your releases, giving all stakeholders insights into your software delivery process helping remove bottlenecks, improve efficiency, enable continuous improvement, and deliver code faster and safer.
Q: How do you reconcile the new governance model with strict regulatory regimes such as PCI, HIPPA, and FedRAMP?
Charles: For the most part, these regulations do not specify "how" - they specify "what." We see that the issue is more in people's interpretation of how to follow the regulations. Certain processes and procedures are implemented, and the organization starts to think that the process IS the regulation. It's not. Start by asking: "What is the intent of the control? What can automation do for me here?"
Q: How do you adopt this new kind of governance when you have to execute some old-school quality control mandated by ISO or customer audits?
Charles: I would start by asking, "Why does the quality control need to interrupt flow"? Chemical plants and refineries are very well-governed examples of continuous flow. The idea that we need to stop work while someone inspects a batch is the problem. Not governance itself.
Dan: Involve your security and compliance teams in the planning and design phase.
Q: What are intricacies of executing this new governance model company-wide? What kind of structure and processes do we need?
Charles: You will need to start with fundamentals. Are your product teams documenting their responsibilities, and living up to their commitments? Put a process in place to ensure that. Risk always needs to be identified, quantified, and controlled. This process doesn't change, but the controls need to move away from stage gates to an assumption of continuous flow.
Dan: We look at it as a 4 step repetitive process…
- Collaborate and simplify your existing process.
- Codify and incorporate controls into your pipeline.
- Automate as much as possible.
- Learn and improve.
Q: How can practices such as IT4IT help in creating end-to-end traceability and audition?
Charles: IT4IT is a great reference architecture for the digital pipeline. From a governance perspective, I would look at where risks and controls are present in the various components. Another Open Group standard to take into account is the Digital Practitioner Body of Knowledge, which I have been involved with.
Q: What are your best examples of good sources describing complex Agile and DevOps in sectors outside of IT/software?
Charles: I would start with the Lean and Theory of Constraints bodies of knowledge. There are books on Agile hiring and contracting, but they are focused on those topics in the context of software.
Dan: CALMS is an acronym for a framework which allows businesses to assess how prepared they are on their journey to DevOps, and where they can improve. CALMS stands for Culture, Automation, Lean, Measurement, and Sharing. These are all important values which enable teams to break down silos, work better together, and provide frequent value to users that can be continuously improved. These values can easily be related to other sectors where two or more teams need to work together to deliver outcomes for the business.
Q: What is your take on setting expectations at scale?
Charles: Big complex systems are unpredictable, and that's the primary expectation you should set. I'm concerned about any big strategic plays that lead to people wanting to set expectations at scale. "Plan, then execute" is some of the most dangerous advice out there. Experiment and course correct.
Dan: Many large organizations struggle while trying to scale their DevOps practices. This is largely due to the inability to share knowledge across teams and environments. You need to look at how to build best practices from the teams that have been successful, so you do not have to reinvent the wheel each time. Scale slowly, learn from your mistakes, and you will get a better sense of how to properly set expectations.
Q: I would like to understand how to keep the governance aspect of IT change management incorporated in the ever-increasing pace of Continuous Delivery/Deployment.
Charles: Addressed above. Automate risk scoring (of both team and proposed change). Automate and log. Keep changes small so they can be easily reversed. Test and measure throughout (but don't interrupt flow to do so).
Dan: Governance does not go away just because you are moving faster. The need to automate and track all changes becomes extremely important. We call this your Software Chain of Custody which proves the end-to-end compliance of your software delivery pipelines.
- Watch the on-demand webinar From Chaos to Compliance: The New Digital Governance for DevOps
- Blog post: Establish an Immutable Chain of Custody for Software Releases
- White Paper: DevOps, Pipelines, Toolchains, and Application Release Orchestration: A Short Primer