This post is from the XebiaLabs blog and has not been updated since the original publish date.
How Data is Driving the Next Wave in DevOps Intelligence
The amount of data that we create around the world every day can barely be fathomed - and it will only grow over time. According to Raconteur, by 2025, the amount of data generated daily will be 463 exabytes. That's the storage capacity of 14,468,750,000 32-gigabyte iPhones.
The data industry is hot with acquisitions. The three biggest, most recent acquisitions in data are:
- Microsoft acquired GitHub for $7.5 billion
- Looker becomes the newest part of Google Cloud for $2.6 billion
- Salesforce buys Tableau Software for more than $15 billion
There have been 14+ acquisitions of data companies in 2019 - that's a lot. If companies are going to spend this much money, data is clearly important! However, as the saying goes, "more data, more struggles."
The Top 10 Data Struggles
- Overwhelming amount of data: With all this data, risk managers and others are getting overwhelmed. Organizations may receive information on every incident and interaction that takes place on a daily basis, leaving analysts with thousands of interlocking data sets.
- Collecting meaningful and real-time data: When employees are overwhelmed, they may not fully analyze data or only focus on the measures that are easiest to collect instead of those that truly add value.
- Data from multiple sources: Trying to analyze data across multiple, disjointed sources where different pieces of data are often housed in different systems, is difficult.
- Inaccessible data: How do all stakeholders get access to the data that is important to them? People, processes, and tools can stand in the way between data and stakeholders who need it.
- Poor quality data: Meaningful insights can only be as good as the quality of data collected.
- Choosing the right tools: With hundreds and hundreds of data collection tools on the market, who knows what's best?
- Lack of collaboration: Different users and groups may have access to some data, but without collaboration, it is not shared.
- Lack of skills: Many employees do not have the knowledge or capability to run in-depth data analysis.
- Visual representation of data: To be understood and impactful, data often needs to be visually presented. This can take a lot of time and needs to be created by the right person.
- Understanding the data: Understanding how the data relates back to the business (the people, process, goals, etc) is critical to actually have the data mean anything at all.
In short: data is a problem, one that featured very prominently at this year's DevOps Enterprise Summit. Here a few keynote speakers who mentioned it:
- Optum: Accessing data is the #1 developer challenge.
- Adidas: "We see data as the baton in a race...it is not always the fastest that wins, but the handoff that makes a difference.“
- John Deere: Data analytics must be made available to the teams that need it.
Right now, there are so many disconnected DevOps tools.
There has to be a way to connect your 30+ tools together into a DevOps toolchain, collect data from all of them, and unify this data in a single place...and there is! The XebiaLabs DevOps Platform connects, orchestrates, collects, and unifies your DevOps data across your end-to-end DevOps toolchain.
Only when you have known, good data can it actually be useful to your organization. Without a connection across all of your tools, the data will be incomplete. It's important to have an end-to-end toolchain with good data and THEN dive into the super exciting world of DataOps, AIOps, and Machine Learning!
DataOps, AI, ML, Oh My!
DataOps is DevOps processes applied to data operations. DataOps involves delivering more agile and automated approaches to data management. It's about enabling self-service access to data to accelerate the development of database-driven applications and data-driven decision-making. And, it's about supporting business agility in response to rapidly changing business requirements.
AIOps is a form of DataOps. AI is the science and engineering of making intelligent machines. AIOps platforms utilize big data, modern machine learning and other advanced analytics technologies to directly and indirectly enhance IT operations (monitoring, automation, and service desk) functions with proactive, personal, and dynamic insight.
AIOps leverages data across the toolchain, enabling end-to-end automation and insights. It is the next logical evolution in using data to understand how the business runs. AIOps relates all data to the “application” as a central unit by:
- Unifying all relevant data
- Connecting application health data back into CD cycle
- Wherever possible, taking automated actions
Gartner came up with the major elements of AIOps. AIOps can amplify DevOps practices through:
- Machine Learning
- Performance baselining
- Anomaly detection
- Automated root cause analysis
- Predictive insights
AI and Machine Learning (ML) are not the same thing. ML is an advanced form of AI.
ML allows computers to learn without the help of humans. In the chart above, you'll notice "Deep Learning" (DL) is within ML. DL is computers behaving through neurologics - basically, computers behaving similarly to a human brain. Deep Learning doesn't really apply to DevOps right now; we are still in the early stages of Machine Learning, but it's still important to note how much potential there is in the industry.
Back to AIOps...
AIOps on DevOps data will enable higher automation, faster delivery, and better insight by:
- Driving improvements and outcomes based on metrics: tracking application delivery data across tools allows you to uncover things like anomalies in that data, long build times, late code check-ins, and slow release rates to identify many of the software development wastes.
- Ensuring application quality: Using ML to analyze output from testing tools can intelligently review QA results and efficiently build a test pattern library based on discovery. Learn more from known, good releases.
- Securing application delivery: Like fingerprints, user behavior patterns can be unique. Applying Machine Learning to Dev and Ops user behaviors helps in identifying anomalies, which can represent dangerous activity.
- Managing production: ML comes into its own by analyzing an application in Production due to larger data volumes, transactions, etc. This occurs more in Production when compared to Dev or Test.
- Troubleshooting and triage analytics: Tools can use ML to detect anomalies in general processing and can analyze the release logs to correlate with new deployments.
- Preventing production failures: ML can be used to predict production failures based on patterns.
- Analyzing business impact: In DevOps, understanding the impact of code release on business goals is critical for achieving success.
Example: Software Chain of Custody
Like a legal case, the Software Chain of Custody tracks everything in your software delivery pipeline. Know the who, what, where, and when of each change made throughout the delivery process for ultimate visibility. The XebiaLabs Software Chain of Custody comes equipped with push-button audit reporting, automatically producing audit reports and therefore removing the pain from your audit process and improving the completeness of your data.
AIOps performance baselining can analyze data release after release, find anomalies like software breaches, and overall protect your production environment through a warning to your team.
Example: Risk Prediction Module (AI/ML based)
AIOps and Machine Learning curb your risk. Risk alerts warn the team when a release is likely to be delayed or fail. Risk forecasts shows predicted delays and failures for every task so you know what's happening. Statistics for similar releases provide historical analysis at a glance, so you're aware of trends now and can figure out what improvements to make in the future. Release forensics enable teams to identify their biggest pain points - and figure out how to solve them.
DataOps, AIOps, and Machine Learning can amplify your DevOps objectives - and they should! Collecting data is one thing, but knowing what data is good data and analyzing it are the only ways you'll actually know what's working in your organization and where to improve.
- Check out another blog post: Establish an Immutable Chain of Custody for Software Releases
- Read the whitepaper: Baking Compliance and Security into Your Software Delivery Process
- Learn more about the Software Chain of Custody