This post is from the CollabNet VersionOne blog and has not been updated since the original publish date.
Pepperoni, Metrics and Software
Guest post from Don McGreal of Improving Enterprises
The disconnect between the delivery organization and the business is prevalent in the software industry. Somewhere along the line, the real vision behind our projects gets lost. We all know it.
Can better agile metrics help?
Let’s talk pizza.
You work at a growing pizza chain. You are part of their delivery organization. You and your colleagues are responsible for getting pizzas out the door and to the hungry customers. You have managers, drivers, phone operators, vendors, etc.
How do you know if you are being successful? What metrics will you measure your progress against?
I would like you to take a minute to think about this. Write the metrics down. I’ll wait…
Did they look something like these?
Good. I thought so.
A perfectly reasonable set of metrics, right? If you care about improving your department, you should definitely track them.
Now, let’s switch roles. You are now part owner/operator of the pizza chain. I would like you to come up with another list of metrics that matter to you and your partners.
Write them down. Again, I’ll wait…
Did they look like these?
Great! Nice work.
Now, are your 2 lists much different? I am sure there are a few overlaps (e.g., Customer Satisfaction, Costs, etc.) But I’m willing to bet that they are mostly very different from each other. Why is that? Is that even a bad thing?
Here are 3 things to consider:
The delivery organization now has a set of shiny metrics they can work toward. The practices and processes they put in place will focus on moving these numbers. The assumption is that by improving these intermediate metrics, the business will benefit:
“Hey look, we spent all this time implementing these route efficiency algorithms so that we can save 30 seconds on each delivery.” You may reasonably assume that this will be good for the business, but it’s no guarantee and may even become more of a distraction. Popular practices are great, but they aren’t the end goal and we can’t lose sight of the business’ true needs. Otherwise you end up with a Cargo Cult mentality.
The more your people know and understand the true vision and goals of the organization/product, the better decisions they will make. Like it or not, assumptions and decisions will be made independently. You can minimize the negative impact of these decisions by educating your people on the true organizational drivers.
Your pizza shop has quarterly bonuses to hand out. What would you base them on? The delivery metrics or the owner metrics? Which ones have a greater potential for perversion? Any metric can be gamed, but you’ll notice that the more intermediate circumstantial metrics in the delivery department have bigger opportunity for abuse. This doesn’t just create unintended behaviors; it also reduces the transparency and, therefore, the usefulness of these metrics.
A few years ago, my wife and I went to dinner at a chain restaurant. Before leaving, we were presented with a customer satisfaction survey and told, “If we give all ‘excellent’ scores,” we would receive a free appetizer for our next visit. This was obviously not the way the business intended to use these surveys.
So what does this mean?
The delivery metrics aren’t without worth. They are very helpful, even essential, for guiding our more tactical practices. But we run into problems when they are used as false representations of value and set as achievement goals. If the business doesn’t establish the goals along with more direct value metrics, then the delivery organization will have no choice but to offer up their own.
You have probably already done this, but can you now correlate these metrics with the ones on your software projects?
Which metrics are used by software delivery departments? Which ones by the business?
So how does your organization measure success? Are they spending too much time on the left-hand side? Are there opportunities to instead apply metrics that reflect true business outcomes? This concept of using direct rather than circumstantial evidence is known as Evidence Based Management (EBM). It is gaining traction in the technology field and has industry leaders like Ken Schwaber advocating for it.
With EBM, our pizza chain could evaluate the valuable outcomes to determine which parts of the delivery process contribute to them.
For example, if the chain had very large orders for 2 important customers come in at the same time, they may be inclined to wait until both orders are complete before sending out the driver. This would certainly improve delivery metrics (pizzas per trip, miles per delivery, fuel used, orders per driver), but may not be the best move for the business overall.
If instead, everyone involved was focused on EBM, they would be more concerned with the valuable outcomes that have a more direct impact on the organization (customer satisfaction, lead time, repeat customers). This may compel them to create a practice of sending 2 drivers if the orders are big enough.
This disconnect between the delivery organization and the business is prevalent in the software industry. Somewhere along the lines, the real vision behind our projects gets lost.
Challenging our managers, product owners and development teams to identify and promote metrics related to real outcomes is key to bridging that gap. This can promote true agility beyond IT departments and give organizations a real competitive advantage.
Don McGreal is VP of Learning Solutions at Improving Enterprises and is co-founder of TastyCupcakes.org, a comprehensive collection of games and exercises for accelerating the adoption of agile principles. Improving Enterprises is a leading software development company that offers advanced technology consulting and training across our 5 locations in Texas, Ohio, and Canada.