Improving Quality through Continuous Integration

Continuous Integration (CI) has been practiced in the industry for the past few years where the code is built automatically and the tests are run against a specific build. CI provides early feedback with test results to the developers in a shorter timeframe so that the developers can fix the coding issue or at least rollback the code changes so that the code is always stable.  Therefore, using CI, one can perform a lightweight test whereby test results are published quickly, say, within 15 minutes.  The CI process is followed even in cases where detailed functional tests have to be executed that run for long hours, for example, two to three hours or even longer, by running several tests in parallel. Organizations start using CI when the development teams switch from traditional waterfall model to lean and agile practices.

However, in agile teams, in spite of setting up continuous integration, various issues pertaining to security vulnerability, browser specifications, etc., have been identified by the customers in the production and by stakeholders during sprint qualification. This is a huge challenge to the agile teams because they are answerable to the stakeholders for the quality of the product that is developed. There are too many issues that the teams have to address but do not know where to start. These challenges become unmanageable resulting in chaos among all scrum teams. The only way for the agile teams to come out of this chaos is to organize their work by analyzing and prioritizing the issues and fixing them accordingly.

In this context, let’s discuss how to define and refine CI strategy, plan on how to implement the strategy and finally measure CI implementation against the maturity map.

Identify Problems to define new CI strategy

At CollabNet the above-mentioned challenge existed and we identified the top three immediate problem areas that could be addressed to improve the quality of the product developed by the organization.

1. Problems faced by the agile teams: The developers were not sure what the unit test coverage would be for the code being developed. They were unable to verify if the code followed best practices. Also, generally, in a product development organization, the product developed works on multiple platforms, databases, and a single or multiple box setup. In such scenarios, the developers did not have the time to test all the permutations of OS with the database.
2. Problems faced by the customers: Customers identified security vulnerability in the infrastructure where the product runs, for example, JDK or OS vulnerabilities. Customers also reported cross-site scripting issues and SQL injection problems in the product. They constantly reported broken links in the static content of the application where help documents are being maintained.

Typically, customers are standardized on using a certain version of a browser in their organization because of which, they faced browser-specific issues. This standardization posed as a challenge to the development team as well because a product may not work on all browser versions.

1. Keeping up with the industry trend: The industry trend encourages the use of a provisioning platform to build your environment for continuous integration. The best practice is to run as many tests as possible, in parallel, to provide quick feedback to developers. For example, the code review tool can be tied with the commit test so that the code review and commit test together would provide a collective response on the quality of code. However, the teams are busy with their feature development and do not find time to catch up with the industry best practices to improve on their agile journey.

Defining the CI Strategy

Once you identify similar problems as mentioned above, the first step is to define a CI strategy to be incorporated into your department roadmap.

Refining the CI Strategy

Once the CI strategy is defined, the next step is to consider a few parameters to refine the strategy. Not all of the identified parameters can be implemented in an organization; also, they have to be prioritized to make a faster impact. The three parameters that are considered for redefining the CI strategy are:

• Type of coverage: Components that need the most test coverage must be identified and addressed first. For example, what are the most widely-used components by the customers? Pick out those first for implementing CI.
• Time for feedback: The process of providing test feedback on critical core components must be made quicker compared to other components. You need to confirm the time factor by getting answers for questions like, Can the team wait for 5 hours for CI to provide feedback or need the feedback in 15 minutes?
• Cost of investment: Finally, the investment available for CI strategy must be analyzed. Can more testers be hired, or can more hardware and tools be bought to implement the strategy? Need to strike a balance between the two.

The above parameters have to be scaled based on the needs of an organization. Not always all three parameters can be fulfilled because of the various constraints within the organization.

Planning the CI strategy

For each one of the problems identified earlier, plan the strategy by looking at how the problem can be solved. Can it be solved with the help of tools? What value does it bring to the product or to the organization?

Here is a table to define the CI strategy and a few problems identified earlier are mapped onto the  value that brings into your product:

Implementing the CI Strategy

The strategy planned above was implemented as part of the application life cycle management (ALM) and with Jenkins. There may be other ways and other tools that can be used to achieve the same result. In the above case, the strategy was implemented using 50 servers that were provisioned boxes in cloud. All the tools in the plan were set up in Jenkins. Jobs were executed automatically with one job per tool and results were provided as feedback to the agile team.

Sometimes, the tools in the market cannot address your problem and hence some in-house tools have to be developed. For example, at CollabNet, the automatic installer was developed to install the product in various servers on different OS and database combination. Including the ALM tool in CI strategy is a value add for maintaining traceability and to establish CI dashboard. The diagram below shows how the servers with Jenkins master and slave were set up in CollabNet for CI implementation.

CI Dashboard

A dashboard is very important because you can get the CI results to all agile teams so that they are well informed to act upon the results. In addition, e-mail notifications are sent to the development team on test failures, security vulnerability, static code analysis, etc. Below is a dashboard that displays the result of all the tools that are run by Jenkins.

Define Metrics

Metrics need to be defined to measure the problem against its resolution. A few common metrics are used to see the trend of customer reported defects after the implementation of the strategy. Another effective way to measure is to identify how long it took to run all the tests in the past after the strategy. The metrics can be defined by the value that brings in to your product. Primarily the metrics focus on factors such as time/investment saving, or early-to-market. Here is an example of one of the measurements on an in-house tool that installs the CollabNet product automatically through CI. The following chart shows the time (in hours) saved after implementing the Installer Automation through CI.

Measure & Mature CI Strategy

The following diagram explains where the build, test, visibility, deployment and quality are mapped to various stages in maturity from basic, intermediate, advance and optimized. The circled part shows where the CollabNet strategy has matured. In the case of Test, it is at an Optimized level but in the case of Deployment, it is Intermediate, which means that more work is required to mature. You can define your strategy and line up with a diagram below to see where you are in the CI maturity map.

Summary

The following are the benefits of CI, which add value to agile teams and products:

• Defines quality, prevention & reduction of defect
• Increases confidence and reduces risk
• Builds value with faster feedback
• Reduces software release time
• Results published, increases visibility to agile teams

The challenges continue as it is complicated to automate data warehouse ETL testing in CI. The strategy uses many tools and keeping up with tools and upgrading them regularly result in maintenance overhead. In large scale refactoring & architecture changes, the strategy has to be revisited and modified often. Overall, CI strategy, in addition to improving quality, guides you to fine-tune, measure and mature in your agile journey.

I would like to hear about your thoughts on this article and also share anything interesting that you implemented with respect to CI.  Please let me know.