You shall not pass – Control your code quality gates with a wizard – Part II

You shall not pass – Control your code quality gates with a wizard – Part II

In the previous blog post you learned how to select, test and deploy predefined quality gates with CollabNet’s code quality gate wizard for Gerrit. Those quality gates will make sure that all conditions regarding code quality and compliance are met before a commit can be merged into your master branch and trigger a pipeline that will eventually promote it into production.

In this blog post we will focus on how you can define quality gates by yourself, using a methodology very close to setting up email filter rules.

Underlying technical concepts of quality gates

Before we jump into the functionality of the wizard, let’s give a short overview on the underlying foundations (I am sure most of the technical folks will appreciate). TeamForge is using Gerrit (current version 2.8, 2.9 coming soon) as its Git backend. If you are new to Gerrit, we recommend to have a look into this brilliant series of free, recorded webinars. The third and final part of his series is the best introduction into Gerrit’s change based workflow I have seen so far.

If you prefer written explanations and pictures over webinars, I can recommend this blog post from my colleague Dharmesh.

When we are talking about commits, we are referring to commits which are controlled by Gerrit’s review workflow, in other words, Gerrit changes. TeamForge comes with multiple review policies for Gerrit repositories which automatically make sure that Gerrit access rights are setup in a way that you cannot directly push to real branches but always have to go through the Gerrit review process (which does not mean you have to involve any manual reviewers or build systems, see below). When we are talking about pushing commits into production, we are referring to Gerrit’s functionality to submit/merge changes to their anticipated target branch which in turn trigger CI systems which will be responsible for all further actions. Quality gates are a collection of submit rules which define under which conditions Gerrit allows to submit a change. In a vanilla Gerrit, the only way to customize those conditions is to write Prolog programs. CollabNet’s Code Quality Gate Wizard for Gerrit shields its users from the complexity of Gerrit’s Prolog based system (more details in the third part of our blog post series).

The following Prezi shows how our idea was presented at the latest Gerrit User Summit.

You shall always pass – Simple example to start with

Enough with grey theory, haven’t we mentioned multiple times that setting up quality gates is as easy as setting up email filters? Let’s stick to our word. Inside the quality gate wizard, we have three tabs. The first one allows you to edit the title and description of your policy, testing it against Gerrit Changes (dry mode) as well as deploying it to Gerrit.

The picture above shows the first tab of the wizard with a very simple policy: Submit is always enabled, in other words “You shall always pass”. This policy makes it possible to push any commit into production, more precisely, any Gerrit change will be submittable immediately. In addition to the title, description, test and deploy elements, you will notice two check boxes: Enable code review and Enable verification. Those checkboxes define whether users looking at a commit will be presented with the ability to give a Code-Review/Verified votes on the corresponding Gerrit change. This is independent of their actual permissions. In other words, if a user does not have permissions to cast a Code-Review or Verified vote, then this check box will not magically enable them to. As the Submit is always enabled policy allows to merge a commits unconditionally (i.e. as long as the user submitting the corresponding Gerrit change has Gerrit Submit permissions), there is no need to present users with Code-Review or Verify options, so those check boxes are not checked.

The second tab of the quality gate wizard shows the actual submit rules. Submit rules are the technical details behind every policy you can define in the quality gate wizard. Like an email filter, submit rules have actions. While an email filter allows you to specify whether to delete or move an email into a subfolder, submit rule actions define whether to allow a commit to be merged into production, whether to block merging or not to do anything (ignore).

The Submit is always enabled policy has only one submit rule whith action set to allow if it is satisfied. As there is no condition associated with this rule, it will always be satisfied, so it will always result in allow and hence will always enable Gerrit’s submit action.

You shall never pass – Designing the opposite policy

Let’s have a look at the opposite policy – Submit is never enabled

In contrast, the satisfied action is set to block. If a submit rule gets evaluated and results into block, Gerrit’s submit action is disabled, IOW the Gerrit change cannot be submitted, so the corresponding commit cannot be merged into its target branch and hence this code cannot be pushed into production.

You shall pass as usual – Gerrit’s Default Submit Policy

Too simple examples? Let’s move on then and examine Gerrit’s factory settings. CollabNet’s Quality Gate Wizard includes a template called Default Gerrit Submit Policy which simulates Gerrit’s standard submit behavior.

This policy consists of three submit rules:

1. If somebody gave Code-Review +2 and there is also at least one Verified +1 vote, submit is allowed

2. Any Code-Review -2 vote blocks submit

3. Any Verified -1 vote blocks submit

The screenshot below shows the conditions of the first rule in detail (you get to the voting condition dialog if you double click on a submit rule).

The algorithm deciding who shall pass

At this point, we have to elaborate a bit what happens if multiple rules are satisfied. First of all, the order of rules does not play any role, it can be completely ignored. The algorithm used to decide whether submit is enabled or not, looks as follows:

a) For every submit rule that can be evaluated, figure out whether its voting conditions are satisfied (if a submit rule does not have a voting condition, it is automatically satisfied)

b) If all voting conditions are satisfied for a submit rule, the rule gets evaluated to the action specified in the actionIfSatisfiedField (ignore if no value set), otherwise the rule gets evaluated to the action specified in actionIfNotSatisfied field

c) If any of the evaluated submit rules got evaluated to block, submit will be disabled and the display name of all blocking rules displayed in Gerrit’s UI as reason for this decision

d) If no evaluated submit rule got evaluated to block but at least one to allow, submit will be enabled

e) If all evaluated rules got evaluated to ignore, submit will be disabled and the display names of all potential submit rule candidates displayed (details below)

Potential candidates explained

Going back to Gerrit’s Default Submit Policy, this means that if rule two (Code-Review-Veto-Blocks-Submit) or three (Verified-Veto-Blocks-Submit) are satisfied, Gerrit will block submit and show their display names as reasons, no matter whether the voting conditions for rule one (Code-Review+2-And-Verified-To-Submit) are satisfied. Only if those two rules are evaluated to ignore (as their actionIfNotSatisfied field is implicitly set to ignore) and rule one is evaluated to allow, submit will be possible.

So what happens if there is no Code-Review or Verifed-Veto but the change has not received both Code-Review +2 and Verified +1 votes yet? In that case we end up with the last step in our algorithm where you probably stumbled over the term potential submit rule candidates. Potential candidates are all submit rules that got evaluated and have at least one of their actionIfSatisfied and actionIfNotSatisfied fields set to allow. What does this mean exactly? Well, let’s think about a user who is looking at a Gerrit change in that very state.

The screenshot below shows a change like that.

There are no hard blockers (rule two and three) but as long as rule one is not satisfied, submit is still not enabled. How would the user looking at the Gerrit change even know what to do in order to enable submit? This is where potential candidates come into play. These are all rules which could potentially evaluate to allow, hence making the change submittable. Their display names are now shown in Gerrit’s UI as a hint for the user what possibilities exist to merge the corresponding commit and push it to production. In our case, that is the display name of rule one (Code-Review+2-And-Verified-To-Submit, see last line of screenshot above).

Turning a Gerrit change into a submit rule

So far, the examples provided did not add any particular value to what is already working out of the box in a standard Gerrit environment. Let’s switch gears and add a more interesting submit rule: If the target branch of a Gerrit change is called refs/heads/experimental, we will not need a Code-Review+2 anymore. Instead, having at least two Code-Review +1’s will be sufficient, or more precisely, the sum of all Code-Review Votes should be at least 2 in order to allow submit. Verified votes are not needed in this case but the blocking behavior of rules two and three will still apply. Like an email filter rule wizard, our quality gate wizard supports turning existing Gerrit changes into submit rules. We will use that very feature now to turn a Gerrit change for the experimental branch which has two Code-Review +1 votes into a submit rule.

If you press on the button Add Change-Based Rule, a wizard similar to the one below appears. We will select the very Gerrit change we were looking at in the screenshot above as it already has the conditions we wanted to capture in a submit rule: Two Code-Review +1 votes, one Verified +1 vote and target branch experimental.

We are only interested in copying the Voting details and Commit and change details. If you like to design a submit rule that copies Commit stats (like lines modified, deleted within the commit) or User and group details (who voted, who owns the change, which groups those users belong to), you have to check additional check boxes in the wizard. Once you click finish, a new dialog will open with the generated submit rule.

The generated submit rule will match the Gerrit change it was based upon as close as possible. In practice, this is probably too narrow, as you like to match similar (same branch) but different (other files changed, different commit message) Gerrit changes too. Consequently, we now have to remove some voting conditions and filters. First though, we will change the display name of the new rule to Experimental-Branch-Adds-Up-Votes. Then, we will remove all but the last (Code-Review) voting condition as those are too narrow for our use case. Now, we double click on the last remaining voting conditions, clear out all filters and only keep the minSum=2 setting:

like in the screenshot above. Once we click Next, we will see some further filters with values populated by our wizard based on the characteristics of our change. If any of those filters is not matched by a Gerrit change, the whole submit rule will not be evaluated, so we have to make sure to only keep the filters which are relevant for our use case. In our example, the only relevant filter is a change detail filter specified in the branch pattern field (see screenshot below). All other filters should be cleared as they are too narrow for our use case.

Finally, click on Finish and deploy the new policy into your repository. The end result should look like this:

You can use the testing facilities shown in the first blog post if you want to simulate the impact of the policy first. If we now look at the Gerrit change again, it will actually be submittable:

Sharing your results

Like your new policy and want to share it with your organization? Good! There are two ways of sharing your results: Saving them directly in the repository as an XML file and saving as an XML file on your hard disk. If you save them within a Git repository, the wizard will automatically create a commit and push it to a special Gerrit reference (refs/meta/config), so your ordinary code and branches inside that repository will not be affected. You may wonder what the difference is between deploying a policy to Gerrit and committing the XML file to refs/meta/config. The policy currently enforced by Gerrit is saved in a file called rules.pl and the Deploy button in our wizard makes sure this file name is used whenever you deploy. If you want to store a policy within the repository but do not want to enforce it there, just enter any other filename, the wizard will automatically append an .xml suffix to it.

Whether you like to save an XML file to your local file system or save the file within the repository, it all starts with the File -> Save action for your editor (you can use CTRL+S shortcut or the floppy icon too).

When opening the quality gate wizard, you can select to open an XML file from your file system, the currently enforced policy (rules.pl)  or any stored but not enforced policy inside the repository you are currently looking at.

One idea how to centrally manage your quality gate policies is to create a dedicated Git repository where you can save all your policies. This is possible because you can load policies from different repositories of the same TeamForge project as where you deploy them. Please tell us if you see the need for an even more centralized solution.

Summary

In this blog post we demonstrated how you can design simple quality gate policies by yourself. A policy consists of one or more submit rules. When submit rules are evaluated, their voting conditions will be checked. Similar to an email filter rule, you can define actions based on the result of this evaluation, namely allow, block or ignore. You also learned about the algorithm used to determine whether a Gerrit change is submittable based on those actions. Basically, block actions will block submit no matter what, if no block is there, allow actions will make changes submittable and if there are only ignore actions, Gerrit will guide the users by displaying the names of all evaluated submit rules which could potentially still result in an allow action.

If you have a Gerrit change with characteristics that should enable or disable commits to be pushed into production, you can turn them into a submit rule. The wizard will inspect the Gerrit change and setup the submit rule filters and voting conditions as precise as possible (similar to an email filter rule wizard you point to an existing email). You will have to clear the filters you are not interested in to make sure that your filter criteria are not too narrow for your use case. Only submit rules which match all filters specified will be evaluated by the algorithm that decides whether to allow or block the change from entering its target branch.

While going through the change based submit rule wizard, you have probably wondered about all the possible filters and conditions you can specify. We encourage you to go through the list of all predefined quality gate policies and learn how those filters are used in detail. The next blog post will also show some further filters in action.

Once you came up with your own policies, you can share them with your organization by either exporting them into an XML file or saving them in a dedicated Git repository.

Coming up next …

There is one tab in the quality gate wizard we have left out so far: The last tab is showing an XML representation of the language used to describe the policy you are currently looking at. In the third blog post we will exploit the full capabilities of that language, designing a real powerful policy, namely

• Author Must Not Approve own changes (4 Eye Principle)

• Legal has to approve changes in copyright file

• Only verified from 2 CI users counts

• Only team leads can submit

The power policy is developed step by step, covering all details of our submit rule evaluation algorithm. Finally, we will also explain how to use this language independently from our graphical wizard for your own tooling. If you would like to become a quality gate wizard yourself, read on!