Product backlog is DEEP; INVEST wisely and DIVE carefully

A product backlog stores, organizes, and manages all work items that you plan to work on in the future. While providing agile training, consulting, and coaching engagements at Digital.ai, formerly VersionOne, our clients often ask how to logically structure, organize, and manage their product backlog. Clients also want to know how to prioritize or rank work items.

Here is a simple and easy-to-remember phrase that captures the key characteristics of a well-managed product backlog: Product backlog is DEEP; INVEST wisely and DIVE carefully…otherwise, by implication, you may sink (just kidding, but only slightly).

The key characteristics of a well-organized and managed product backlog are summarized in the image below. DEEP, INVEST, and DIVE are meaningful words. They can be used as very useful acronyms to help us remember the key characteristics. In this blog, I will explain how to manage a DEEP product backlog well by INVESTing wisely and DIV[E]ing carefully.

^{Figure 1: Logical structure and key characteristics of a well-managed product backlog}

The granularity or size of work items should be determined based on how far into the future you are planning a product, i.e., the planning horizon. The longer or shorter the planning horizon, the larger or smaller the work items. This makes sense as it takes a lot more effort to develop, specify, and maintain a large number of small-grain work items compared to developing, specifying, and maintaining a small number of large-grain work items. Smaller work items, stories, are typically developed by breaking down larger work items, or epics. Stories are the unit of software design, development, and value delivery.

DEEP product backlog

A product backlog may have several hundred or more work items, hence the acronym DEEP. Work items can be comprised of stories, defects, and test sets. DEEP is also an interesting acronym, capturing the essence of the logical structure of a product backlog.

• Detailed appropriately: Work items in the backlog are specified at an appropriate level of detail as summarized in Figure 1 and explained below.
• Estimated appropriately: Work items in the product backlog are estimated appropriately as explained below.
• Emergent: Product backlog is not frozen or static; it evolves or emerges on an on-going basis in response to product feedback and changes in competitive, market, and business. New backlog items are added, existing items are groomed (revised, refined, elaborated) or deleted or re-prioritized.
• Prioritized as needed: Work items in the backlog are linearly rank-ordered as needed, as explained below.

Sprint planning horizon, work item granularity, estimation, and rank order

If the planning horizon is the next, i.e., upcoming sprint or iteration (typically two to four weeks), each workcitem is small enough to fit in a single sprint, and is 100% ready (“ready-ready”) to be worked on, as indicated in Figure 1 – see the top red-color region. A ready-ready story has already been analyzed with clear definition (user role, functionality, and business value) and associated acceptance criteria. Work items planned for the next sprint are stories, defects, and test sets. The work items in the next sprint have the highest rank order compared to work items in later sprints or later release cycles. I will soon explain how this rank ordering is done. The rank order information is used to decide the order in which the team will undertake work on work items in a sprint backlog, and also decide which incomplete work items to push out to the release or product backlog at the end of a sprint time-box.

Work items in the next sprint collectively satisfy the well-known INVEST criteria; it is a meaningful English word, as well as an interesting acronym coined by Bill Wake (see his blog Invest in Stories and Smart Tasks). Its letters represent important characteristics of work items in the next sprint backlog. I will now elaborate on the letters in INVEST acronym. Stories in the next sprint backlog should be:

• Independent of each other: At the specification level stories are independent; they offer distinctly different functionality and don’t overlap. Moreover, at the implementation level, these stories should also be as independent of each other as possible. However, sometimes certain implementation-level dependencies may be unavoidable.
• Negotiable: Stories in the next sprint are always subject to negotiations and clarifications among product owner (business proxy) and the members of agile development team.
• Valuable: Each story for the next sprint offers clear value or benefit to either external users or customers (outside the development team), or to the team itself, or to a stakeholder. For most products and projects, most stories offer value to external users or customers.
• Estimable: From the specification of the story itself, an agile team should be able to estimate the effort needed to implement the story; this estimate is in relative size terms (story points), and optionally, it can also be in time units (such as ideal staff hours or staff days for the whole team). Thus, stories are estimated in story points, and also often in ideal time units.
• Sized appropriately: A simpler interpretation of this criterion is that each story is Small enough to be completed and delivered in a single sprint. The letter “S” can be taken to mean Sized appropriately; specifically, each story should take no more than N/4 staff-weeks of team effort for an N-week long sprint (see “Scaling Lean and Agile Development” by Larman & Vodde, 2009, page 120.). Thus, for a two-week sprint, each story should take no more than 2/4 staff-week = 0.5 staff-week = 20 staff-hours of effort. A story substantially larger than 20 staff hours of total effort should be treated as an epic and be broken down into smaller stories. For a four-week sprint, each story should take no more than 4/4 staff-week = 1 staff-week = 40 staff-hours of effort. If a sprint backlog has a mix of stories that are small, medium, or large stories (their average far exceeds N/4 staff-weeks), the average cycle time across all stories will increase dramatically reducing the team velocity.
• Testable: Each story specification is very clear to be able to develop all test cases from its acceptance criteria (which is part of the specification).

Stories may be broken down into implementation tasks, such as analysis, design, code development, unit testing, test case development, on-line help, etc. These tasks need to be SMART:

• S: Specific
• M: Measurable
• A: Achievable
• R: Relevant
• T: Time-boxed (typically small enough to complete in a single day)

If a story needs to take no more than N/4 staff week of team effort (ex. 20 staff-hours for two-week sprints), all SMART tasks in a story should add up to no more than N/4 staff week of team effort. If you have five tasks, each task on an average should take four hours of ideal time effort or less. Stories and their SMART tasks for the next sprint are worth INVESTing in, as the return on that INVESTment is high because they are scheduled to be worked on and delivered as working software in the next sprint itself.

Release planning horizon, work item granularity, estimation, and rank order

If the planning horizon is an upcoming release cycle (typically eight to 26 weeks, or two to six months long – consisting of several sprints), work items are “medium-grain” as shown in the middle yellow color region of Figure 1. Typically, many of these work items are epics; however, they should be still small enough to fit in a release cycle and can be completed over two or more sprints in a release cycle. These epics are typically called features or feature-epics. These feature-epics should still be specified with user role, action, value, and acceptance criteria formalism that is often used for specifying stories, but now you are capturing a larger functionality represented by a feature-epic. Feature-epics are divided into stories – small enough to fit in a sprint – before the sprint in which a story will be implemented.

Over the time horizon of an entire release cycle, INVESTing in stories for an entire release cycle has poor returns because it takes a lot of effort to ensure that the INVEST criteria is being satisfied correctly for a large number of stories covering an entire release cycle, and those stories are much more likely to change over the release cycle spanning several sprints; so this kind of INVESTment may not yield expected results as stories will very likely change during an entire release cycle after they have been specified.

Feature-epics in a release cycle can and should be estimated in relative size terms, but without expending the effort needed to break down all feature-epics in a release cycle into individual stories. This epic-level estimation can be done by comparing relative sizes of epics. This method ensures that all epics and stories are estimated in a common currency of “normalized story point” which represents the same scale for an entire organization across all projects, sprints, release cycles, and teams. There is no need to estimate epics in “swags” or “bigness numbers” which are entirely unrelated to story points.

It still makes sense to rank order feature-epics in a release cycle to decide which ones will be scheduled in sprint one, two, three, and so on. However, this assignment may change as each sprint is completed and more information and learning emerge.

Product planning horizon, work item granularity, estimation, and rank order

If the product planning horizon is over multiple release cycles (typically six to 24 months) going beyond the current release cycle, work items are “coarse-grain” as shown in the bottom gray color region of Figure 1. These large epics or super epics require two or more release cycles to complete. These super epics may be described in plain English (bulleted text) or with screen mock-up or video or prototype or with any form of expression suitable to express the intent and value of super epics. These super epics are divided into feature-epics – small enough to fit in a single release cycle – before the release cycle in which that feature-epic will be implemented.

Over the time horizon of multiple release cycles, INVESTing in stories has even poorer returns compared to INVESTing in stories for a single release cycle. This kind of INVESTment will not yield expected results as stories are very likely to change over a much longer duration of multiple release cycles.

Large epics or super epics that need multiple release cycles to be implemented can and should be estimated in relative size terms, but without expending the effort needed to break down large epics into feature-epics, and breaking those, in turn, into stories. This estimation can be done by comparing relative sizes of large epics.

It may not make much sense to rank order large epics over a multiple release cycle product planning horizon, as this assignment very likely will change over a larger time horizon; besides it does not matter if a large epic which is six to 24 months out in the future is rank-ordered 125^th or 126^th. That level of rank order precision is not required.

I use the strategy of INVESTing in stories and SMART tasks only for the next sprint backlog, but not doing so at the release or product backlog levels. INVEST just-in-time in the next sprint as you plan it. INVESTing in stories and tasks over a longer time horizon will yield poor returns.

DIVE the product backlog carefully

There is rarely enough time or resources to do everything. Therefore, agile teams must prioritize (rank-order, to be more precise) which stories to focus on and which lowest rank-order stories could be pushed out of scope when close to the end of a sprint. For agile development projects, you should linearly rank-order the backlog, rather than do coarse-grain prioritization where stories and epics are lumped into a small number of priority buckets, such as low, medium, high, and critical priorities. Linear rank ordering (i.e., 1, 2, 3, 4 ….n) avoids inflation of priority, keeps everyone honest, and forces decisions on what is really important. It discourages the “kid-in-a-candy-shop” behavior when the business side clamors that everything is of high-priority or of equal importance.

Note that epics and stories are conceptually different, and should not be mixed or aggregated while developing a rank order. An epic rank order is separate from a story rank order.

The responsibility of agile rank ordering is shared among all members of a team; however, the rank ordering effort is led by the product owner. Similar to DEEP, INVEST, and SMART, DIVE is a meaningful English word, and also an acronym. Product backlog items should be linearly ordered based on the DIVE criteria, which requires careful consideration of all four factors captured in the DIVE acronym:

• Dependencies: Even after minimizing the dependencies among stories or epics (which is always a good thing to do), there may still be few unavoidable dependencies and they will have an impact on rank ordering. If work item A depends on B, B needs to be rank-ordered higher than A.
• Insure against risks: Business as well as technical risks
• Business Value
• Estimated Effort

_{Table 1: Summary for managing a DEEP product backlog with wise INVESTing and careful DIV[E]Ing}

I hope you find the statement “Product backlog is DEEP; INVEST wisely and DIVE carefully” a useful mnemonic to remember key characteristics of a well-managed product backlog.

Learn more about how agility can help your enterprise succeed with Digital.ai Agility.