Top 10 Causes of Chaos in Agile Software Development
“Everything is so chaotic.”
“Seems like we are constantly in a state of chaos.”
“Is agile always crazy and chaotic?”
Chaos is a word I hear a lot lately while working with software development teams that are either initially adopting agile software development or possibly undergoing a Lean/agile reshaping to improve their existing agile development approaches. I am often asked if the chaos will subside, and the good news is it will — to a certain extent. But to best understand when things will slow down (or even if they’ll slow down), it’s good to explore some of the causes that make things chaotic with agile software development.
And in my world, there’s no better way to assess than to make a list of Top 10 causes of chaos in agile software development:
1. New Teams Forming.
This one is obvious — as people come together to work with one another there’s a feeling-out period. Usually during this period, teams are learning how to collaborate and work through conflicts. In addition to the people learning to work with one another, there are plenty of established organizational structure and cultural barriers that slow the progress of agile teams forming.
2. Process Learning.
Another obvious one. Although most agile process frameworks (e.g. Scrum or Extreme Programming) are fairly easy and well documented, it takes time and practice to learn the mechanics. Couple this with #1 and, well, there can be some real challenges to getting things done.
3. HEAVY Learning Mode.
This may seem redundant, but it really cannot go under emphasized. In addition to learning about each other and learning the process frameworks, as engineers — we are constantly learning about technologies, learning about our products, learning about our customers, well — just getting smarter. Needless to say, this all adds up.
If you ever have watched the Deadliest Catch on The Discovery Channel, you get to see the struggles and pains of the first-time deckhands – called Greenhorns. They are often in way over their heads, running into challenges around every corner, and are just flat out exhausted. In addition to physically and mentally killing themselves, they are trying to prove themselves. Well, this is true with just about every new team member. Not only are they dealing with Items #1-3 above, the intensity of learning is magnified; until they have some wins and time under their belts, chaos exists.
5. When Quality is NOT Built-in.
It is my opinion that in software development, over the years we’ve invented this approach to quality that is “risky and invites failure.”  Yes, I stole that — in most software development shops, quality is something that is done by different people to ensure “separation of duties” and because of the mentality that ‘developers make bad testers.’ Couple that with the attitude that QA engineers can’t, or don’t necessarily need to know how to code, we have what we do today — a ton of end-of-stream testing, out-of-band automation efforts and, honestly, even the staunches of agile shops struggling with testing and ensuring quality. Without quality weaved into the Design>Build>Test cycle, we tend to see a lot more of these noisy things called Defects. Defects take a ton more time and energy than building it right in the first place.
6. Quality Automation Doesn’t Exist.
Without automation you’re going to find it almost impossible, or at least extremely constraining, to effectively and efficiently deliver software quality in a predictable manner. Similar to the “build quality in” challenges, teams often struggle because their estimation processes call out quality separately (which makes it a target for evil doers), and it often does not account for the work-around automation. Many organizations adopting agile software development don’t have an automation strategy for their legacy code. Therefore, they tend to have bouts of analysis paralysis around the problem space or they simply say, “our product or software is too hard to automate the testing” — so they won’t try. The other challenge around automation is that some see it solely as an end-of-line UI automation thing where a couple engineers work on it. Test automation is a holistic challenge and needs to be treated as such.
7. Lack of Cadence.
When teams are starting out, they don’t realize that the first thing to establish is their cadence — get their schedule in place and timebox everything. The cadence surrounds the process touch points that require formal communication and help us to build habits, thus making the process aspects of agile software development more muscle memory. If you feel like you are always in meetings or your iteration meetings are not occurring at the same Bat-Time and same Bat-Place, it might be time to reset; your cadence is lost.
8. Unpredictable Release Cycles.
This one is an enigma because there are teams I run into that say, “Our product is too large and complex to release in short cycles.” And then there are those that say, “We just release when we have to, it could be twice today or two weeks from now.” Looking at these two statements, they appear to be opposite in cause; however, it all comes down to #7 above and understanding what is the ideal batch size that reduces thrashing allows for tighter alignment among teams; reduces “Integration Hell;” and prevents the amoeba-style releases that never seem to end.
9. Deadline-Rich Environment.
Projects are the problem — or at least the traditional sense and meaning of a project drives the idea of fixed scope and fixed schedule. Let’s look at the PMI’s definition of ‘a project’:
A project is a temporary group activity designed to produce a unique product, service or result.
A project is temporary in that it has a defined beginning and end in time and, therefore, defined scope and resources.
At the end of the day, we drive our business off of the idea that of pushing for a date — we get emails from colleagues asking “when?”, we get emails from tools telling us “now!”, and we have other teams telling us “yesterday!” Ultimately, projects drive expectations that are generally dates; we can’t seem to get away from them until everyone realizes that we should design and define the scope to fit the schedule, not the schedule to fix the scope. This is because the schedule usually flexes in these situations, not the scope.
10. Estimation (and For That Matter, Capacity) is Not Understood.
We often see teams measuring productivity units of time versus being measured as units of value. This is the case even in mature agile shops. Everyone is so focused on trying to come up with a voodoo formula to determine capacity of a team or organization and another voodoo formula to normalize story points across teams in order to build a long-term plan based on the cooked-up numbers. The funny thing is that in many cases, the approach used for estimation doesn’t really change once an organization starts using agile. Everyone continues to plan predictively what we really don’t know. Agile software development is an adaptive approach to estimation and capacity. We work with what we know, we measure value, we assess complexity, and we often simply size efforts based on relative uncertainty. If people would just keep it simple, try to get all stories to the same level or near the same level of certainty, and do the same with the to-do’s (a.k.a. tasks and tests) — then in a couple iterations, teams could just count the stories and count the to-do’s accomplished within a timebox and use that for planning. Oh, if only it could be that simple … it is.
Okay, this was just my brainstorming or brain dump (literally) of 10 things that cause chaos in software development, in particular in the situations where an agile adoption or reshaping is underway. Just keep in mind, change is constant in business — now, more so than ever. Software development is complex; so are people. The great thing about tomorrow is that we don’t know what is going to happen. So, just practice and keep in mind: if today is bad — then there’s tomorrow.