This post is from the Collabnet VersionOne blog and has not been updated since the original publish date.
The Trouble with NFRs
The use of Scaled Agile Framework® (SAFe®) as a way of scaling Agile brings the need to identify Non Functional Requirements (NFRs) throughout the value stream. This need to deal with NFRs is also seen in DAD and LeSS, we have always known that NFRs are key to software success. In some cases the NFRs will be implied, you have a legal requirement to deliver software that is fit for purpose. You also need to comply with any relevant legislation such as the UK Data Protection Act. In other cases NFRs are explicit. They will be called out such as requirements for web site colour schemes to assist with visually impaired people. Other NFRs could be identified by the teams and integrated into the definition of done. These NFRs are based on experience and craftsmanship such as the need to ensure reviews, or configuration management. When looking at the SAFe big picture we see that NFRs are effectively everywhere. These exist on each of the backlogs shown, and constrain these backlogs. There are also other NFRs, as mentioned above. We see that SAFe identifies on the Big Picture NFRs that are applied from portfolio to story level. These constrains are general and can apply to all Epics, Features and Stories in any particular backlog. This means that an NFR must be satisfied by a number of stories, unlike acceptance criteria which are defined on a per feature or story basis. The following is a class diagram that shows how the Non/Functional requirements relate to the epics, features and stories. Note how the Epic does have non/functional requirements while all of the functional requirements for it live at the feature level.
Non/functional requirements come in many guises. They can be known up front or they can be discovered as time moves forward. Like anything else agile they can be elaborated at any time outside of a sprint. For the purpose of this discussion we will recognise two main NFR types. Team NFRs and product NFRs.
Team NFRs are norms of behaviour that are agreed within a team and between teams. The VersionOne product feature of communities is an ideal place to establish and elaborate these NFRs. For instance the Definition of Done would be a Topic in a community called Agile Teams. This Topic would include a number of other topics. For example the definition of done could include:
- Use of TDD and the way the tests are maintained
- Coding Standard and how it is enforced by the teams
- Configuration Management
- Continuous Everything usage
- Documentation Standards
This approach could also be extended to the definition of ready and indeed for any other team norms that need to be in place and continuously improved by the teams. These NFRs are then available for all to understand and improve as need be.
Other NFRs are at a more technical level. Examples include the performance of applications, security, and maintainability. In fact many words that end with ility suggest NFRs. These NFRs can be tested, in many cases using test automation. Using the VersionOne features of Tests and Test Sets to record NFRs, we can imagine that these are developed as we progress. Here we would model these test sets with the tests using Given, When, Then where we could and otherwise using whatever language fits best. The problem with this approach is that the V1 tool only allows a test set to belong to one backlog item at a time. Therefore we would have to apply them only to the Portfolio Epic, or whatever the highest level abstraction was. Tests are likewise constrained. However, test sets and tests can belong to more than one regression suite at a time, something we will look into further on. We should consider that part of a definition of done for a story or feature would include the successful demonstration of the NFRs that are specified to constrain the item. An Epic’s NFRs are met when all of the features (or Sub/Epics) in the Epic have their NFRs met and all other Epic level NFRs are satisfied. For example a performance requirement may be for an entire user transaction which may contain a number of stories. The individual story needs to be integrated into the feature level before the performance requirement can be tested. It can then be tested for every story that is part of the feature. This aligns to the way the regression testing works. We see that NFRs apply to the epic and feature level as well as to the story level. There is a nesting in place that lines up to the way that NFRs constrain the epics, features and Stories in a SAFe model.
Defining NFRs, an Example
One way to define NFRs in V1 is to use the regression testing features. In this mechanism we can generate a number of regression tests and test sets into a regression test suite which can then be part of a regression plan, which can be used to bring NFRs into the testing regime. The advantage to this approach is that a regression plan can have tests and test suites assigned to them in a non/exclusive relationship. Let’s take an example, security. Imagine two different security test suits, penetration testing and SQL injection testing. These work at different levels and can be applied to different scope. Imagine further that each of these test sets are made up of a number of tests such as shown in the table below. The Pen tests show different attacks and the injection tests also different types of SQL injection attacks. In reality both test sets would be much bigger, but this is enough to form the discussion. Now a test set cannot be related to more than one epic/story at a time. It can however be assigned to a number of regression suites at a time. It is confusing to note that tests can be directly assigned to a regression suite and also be assigned to a test set that can be assigned to the suite as well. This makes sense in usage however.
|Penetration Tests (can be bought as a package sometimes)||Scan all ports for status|
|url malformed attack|
|Injection tests||Delete information attacks|
|Reveal information attacks|
|Alter information attacks|
Scope and Use of NFRs
Now we have two test sets which can be part of a security regression plan, and which could be configured as two regression suites. These suites would work like this. The penetration suite would be best applied at the system or release level. Penetration tests are usually targeted at a system under test which is a full build. These tests are about accessing the system through unauthorised approaches. Injection tests are applied to individual fields in the completed application. They therefore apply to the story level. In this case an injection test set would apply to nearly every story that contains a UI component, or other exposure of database functionality. And these UI components would need to pass this NFR before they were ”done”. So in this case of injection NFRs it may be best to place a test into each story that is for the test set to be performed. Then we could plot the status of the test set, through the status of the test. Not that the test set can be associated with the test set through the relationships. Downside is that a task for this would have to be created for each and every story that has UI features. Another approach for this would be to create a downstream dependency in the test set to the story. This way it is possible to see all of the stories that are dependent upon the injection test NFR. However this approach does not track the dependencies as they are satisfied. The Pen Test NFR is related to a release. This in fact would be related to all releases. So we would want to associate the pen testing test set for NFRs to the Agile Release Train. To do this we can add the actual tests as regression tests and generate test sets for the project from here. These test sets can then be configuration managed as each agile release train could have a different test set, but would all be part of the same regression test suit. The planning level for the regression test suit should therefore be set to the value stream, while the planning level for the test set should be the agile release train. This is a unique relationship as only one planning level can be defined at a time. In fact if we have a number of programme increments (PIs) the pen test set would be growing through these PIs. In this case hold off generating the test set until you are ready to run it, but add new regression tests to the regression test suit, in preparation for this moment. The tests in the test sets can be configured later as need be.
1: For a new NFR at the portfolio or value stream level, add these to the regression test suite. These go in as regression tests and are used to either generate or be added to a test set. These test sets can be designed such that they are run at the release train level (Pen Tests) or run at the story level (Injection Tests). 2: If they are to be run at the release train level then the test set has the planning level set to the to show this. 3: If they are to be run at the story, definition of story done level, then a test will need to be created in the story and linked to the test set manually, or a downstream dependency placed on the test set from each story that needs to demonstrate that it has met the NFR constraint. 4: As a new standard of work is encountered then it is raised as a topic in the collaboration room and progressed in this manner. The definition of Done or Ready may well be impacted. The above allows us to create and track NFRs at the various levels that they are needed at. Scaled Agile Framework and SAFe are registered trademarks of Scaled Agile, Inc.