You Shall Not Pass – Control Your Code Quality Gates With a Wizard – Part III

If you read the previous blog post in this series, you should already have a pretty good understanding on how to design your own quality gates with our wizard. When you finish reading this one, you can call yourself a wizard too. We will design a very powerful policy consisting of quite complex quality gates. All steps are first performed within the graphical quality gate wizard. For those of you who are interested in what is going on under the hood, we will also show the corresponding snippets of the XML document which is generated by the wizard. You can safely ignore those details if you do not intend to develop your own tooling around our quality gate enforcing backend. If you play with this thought though, we will also show you how to deploy quality gates specified in our declarative language without using our graphical wizard.

Your reward – The Power Example

Before we reveal the last secrets of our wizard and the submit rule evaluation algorithm, you probably like to know the reward for joining us. The policy we are going to design consists of the following steps:

1. At least one user has to give Code-Review+2 , authors cannot approve their own commits (their votes will be ignored)

2. Code-Review -2 blocks submit

3. Verified -1 blocks submit

4. At least two CI users (belonging to Gerrit group CI-Role) have to give Verified +1 before a change can be submitted

5. Only team leads (a list of Gerrit users) can submit

6. If a file called COPYRIGHT is changed within a commit, a Gerrit group called Legal has to approve (Code-Review +2) the Gerrit change

The final policy can be downloaded from here. Please note that it will not work out of the box for you as your technical group ids for the Legal and CI groups as well as the concrete user names for team leads will differ. We will guide you step by step how you come up with a result that fits your specific situation.

Starting with something known – Gerrit’s Default Submit Policy

Looking at steps 1, 2 and 3, you probably realized that they are quite similar to Gerrit’s Default Submit policy. Because of that, let’s start by loading the template Default Gerrit Submit Policy. Once you see the first tab of the editor that opens, adjust name and description as shown in the screenshot below.

If you now switch to the Source tab (the third one), you can see the XML the wizard generated for the default policy:

The XML based language you can see here is enforced by our Gerrit Quality Gate backend. We believe that this language is way easier to learn than writing custom Prolog snippets (the default way of customizing Gerrit’s submit behavior). Furthermore, it exposes some features of Gerrit (like user group info) which are not exposed as Prolog facts. Our Quality Gate backend is implemented as a Gerrit plugin that contributes a custom Prolog predicate which in turn parses the XML based language and instructs Gerrit’s Prolog engine accordingly. This amount of detail is probably only relevant to you if you intend to mix your own Prolog snippets with policies generated by our wizard.

The schema describing our language can be found here. Looking at the screenshot above, you can clearly see that the XML top element GerritWorkflow contains all settings of the first tab of our wizard. You have probably spotted the attributes for name, description, enableCodeReview and enableVerification. The latter two store the info whether to present users with the ability to vote on the Code-Review/Verified categories (given appropriate permissions).

The only child elements accepted by the GerritWorkflow element are SubmitRules. You can clearly see the three submit rules of the default policy, we have covered in detail in our second blog post. Let’s examine the first submit rule named Code-Review+2-And-Verified-To-Submit. If all its voting conditions are satisfied, it will be evaluated to allow, making submit possible if no other rule gets evaluated to block. As this rule has not specified any value for its actionIfNotSatisfied attribute, it will evaluate to ignore if not all its voting conditions are satisfied. Talking about voting conditions, you can see two VotingCondition child elements. The first one is satisfied if somebody gave Code-Review +2, the second one if somebody gave Verified +1. The second SubmitRule element maps directly to step 2 of our power example ( Code-Review -2 blocks submit), the third one directly to step 3 (Verified -1 blocks submit).

Ignore author votes by introducing a voting filter

Let’s modify the first submit rule that it matches the first step of our power example policy:

“ At least one user has to give Code-Review+2 , authors cannot approve their own commits (their votes will be ignored)”

For this, we first switch to the second tab of our wizard (Submit Rules) and double click on the first submit rule. Right after, we double click on the first voting condition (Code-Review) and check the Ignore author votes checkbox in the dialog that opens, see screenshot below.

Once we save this change (press Finish in the two dialogs) and switch back to the Source tab, we can see that the XML of the first submit rule has changed:

The first VotingCondition element now has a VoteAuthorFilter child element. This one has its ignoreAuthorVotesattributes set to true, which in turn will make sure that only votes of non authors will be taken under consideration when this voting condition gets evaluated. You also notice the ignoreNonAuthorVotes attribute. With that one, it would be possible to turn the condition around (if set to true) and ignore all but the author’s votes. If both attributes are set to true, all votes will be ignored. Voting conditions always apply to the latest change set of the Gerrit change in question.

Adding a group filter to the verified voting condition

Now that we have realized step 1 of our power example and step 2 and 3 could be just left unmodified from the default policy, let’s focus on step 4:

“At least two CI users (belonging to Gerrit group CI-Role) have to give Verified +1 before a change can be submitted”.

This can be achieved by modifying the second voting condition (Verified) of the first submit rule. This time, we do not ignore Verified votes from authors (we could by just checking the same box again) but by adding a group and a count filter.

Like shown in the screenshot above, enter 2 into the Vote Count Min field and add the Gerrit group of your choice that represents your CI users. The wizard allows you to select TeamForge groups, TeamForge project roles and internal Gerrit groups.

If we finish the dialogs and switch back to the Source tab, we can see that the second voting condition of our first submit rule has changed:

Two filters appeared, one VoteVoterFilter and one VoteCountingFilter. The first one makes sure that only votes casted by the CI_ROLE (we chose TeamForge project role role1086 here) will be recognized when evaluating the surrounding VotingCondition.

The second filter is a counting filter. Counting and summing filters are applied after all other filters within the same VotingCondition have been already applied. In our case, it will be applied after all votes which

a) do not fit into voting category Verified (votingCategory attribute of parent element)

b) do not have verdict +1 (value attribute of parent element)

c) have not been casted by a user which is part of the CI_ROLE (see paragraph above)

have been filtered out.

After that, our VoteCounting filter will only match if at least two (minCount attribute) votes are left. If this is not the case, the surrounding VotingCondition will not be satisfied and as a consequence, its surrounding SubmitRule will not be satisfied either.

Introducing SubmitRule filters

So far, we only talked about voting conditions and its child filter elements. Sometimes, you do not want an entire submit rule to be evaluated if a certain condition is not fulfilled. Our second blog post already used a submit rule filter for a rule that should only be evaluated if a commit was targeted for the experimental branch.

Step 5  of our power policy is another example:  “Only team leads (a list of Gerrit users) can submit”

We will add a filter to our first submit rule that will make sure that it only gets evaluated if a team lead looks at the Gerrit change. As we only have three submit rules so far and the first one is the only one which can potentially be evaluated to allow, it is sufficient to add this filter only to the first one. To do that, we switch back to the Submit Rules tab, double click on the first submit rule and click on the Next button in the dialog that opens. After that, you can see four tabs, grouping all available submit rule filters. You probably remember those tabs from the second blog post where the values for those filters have been automatically populated based on the characteristics of an existing Gerrit change (more precisely, its latest change set).

This time, we will manually enter the filter values we need. Let’s switch to the User tab and select the accounts of your team leads. In the screenshot below you can see that we chose the accounts of eszymanski and dsheta as team leads.

Once you select your team leads instead (our wizard makes it possible to interactively select any TeamForge user or internal Gerrit account), let’s click on Back and finally adjust the display name of our submit rule to its new meaning: Code-Review+2-Verified-From-2-CI-And-Project-Lead-To-Submit

If we finish the dialog and switch back to the Source tab, you can see that our first submit rule has not only changed its displayName but also got a new child element:

The UserFilter element makes sure that the surrounding submit rule will only be evaluated if at least one of its CurrentUser child elements matches the user currently looking at the Gerrit change.

If there are multiple submit rule filters, all of them have to match if their surrounding submit rule should be evaluated. You may ask what happens if no submit rule can be evaluated because none of them has matching filters. In that case, submit will be blocked and a corresponding message displayed in Gerrit’s Web UI. The same will happen if you have not defined any submit rule at all. As always, you can test your submit rules directly in the wizard against existing changes before deploying.

Providing guidance to your users with display only rules

Before we design a submit rule for the final step (6), let’s try to remember the submit rule evaluation algorithm and what will happen if a non team lead looks at a Gerrit change with our current policy. Quoting from blog post two:

a) For every submit rule that can be evaluated, figure out whether its voting conditions are satisfied (if a submit rule does not have a voting condition, it is automatically satisfied)

b) If all voting conditions are satisfied for a submit rule, the rule gets evaluated to the action specified in the actionIfSatisfiedField (ignore if no value set), otherwise the rule gets evaluated to the action specified in actionIfNotSatisfied field

c) If any of the evaluated submit rules got evaluated to block, submit will be disabled and the display name of all blocking rules displayed in Gerrit’s UI as reason for this decision

d) If no evaluated submit rule got evaluated to block but at least one to allow, submit will be enabled

e) If all evaluated rules got evaluated to ignore, submit will be disabled and the display names of all potential submit rule candidates displayed

As our first submit rule (Code-Review+2-Verified-From-2-CI-And-Project-Lead-To-Submit) has a submit rule filter which will not match if you are not a team lead, this rule will not be evaluated. This leaves us with submit rules two (Code-Review-Veto-Blocks-Submit) and three (Verified-Veto-Blocks-Submit). Neither of those submit rules have a submit rule filter so they will always be evaluated. Both rules have one Voting Condition, checking whether there is any Code-Review -2 or Verified -1 vote. If the corresponding voting condition can be satisfied, the surrounding submit rule will be evaluated to block, blocking submit and showing its display name as reason within Gerrit’s Web UI.

Let’s pretend nobody has vetoed our Gerrit change so far. In that case, all evaluated rules will be evaluated to ignore and the final step (e) of our algorithm will kick in. Submit will be disabled and the display names of all potential submit rule candidates, IOW all evaluated submit rules which can potentially be evaluated to allow will be shown. In our case, there are no potential submit rule candidates though as the only submit rule which can potentially evaluate to allow is submit rule one. This submit rule was not evaluated though as its submit rule filter did not match (no team lead was looking at the change). As a result, Gerrit can only show a very generic message why submit is not possible, leaving non team leads confused on what to wait for.

How to give guidance under those circumstances? Should we just modify our algorithm and also display the display names of submit rules that did not get evaluated? Probably not. Imagine you have a secret quality gate for a group called Apophenia who can bypass other quality gates if the commit to the enigma branch if the number of lines added to the commit is 23 (for anybody who does not know what I am talking about, I can really recommend this movie).

The corresponding submit rule would have submit rule filters making sure that the rule only gets evaluated for that particular branch, commit stats and user group. As long as those filters are not matched, the display name of surrounding submit rule must not be revealed under any circumstances. We are sure you can imagine a more business like scenario with similar characteristics.

Fortunately, there is a way how to guide users under those circumstances: Display only rules

Display only rules are submit rules without any voting conditions and no submit rule filters. Consequently, they are always evaluated and will always satisfy. They do not have any value (or ignore for that matter) set for their actionIfSatisfied attribute though. Hence, they will never influence whether submit is enabled or not (that’s why they are called display only after all). Their actionIfNotSatisfied attribute is set to allow. This makes them potential submit rule candidates.  In other words, their display names will always be shown whenever no other submit rules allows or blocks submit, providing perfect guidance.

In our particular example, we will create a display only rule with display name Team-Lead-To-Submit which will give all non team leads guidance why they cannot submit although nobody vetoed the change.

At this point, we like to demonstrate another cool feature of the Source tab. It is bidirectional, so you can also modify the XML and your changes will be reflected in the first and second tab of our wizard. Let’s paste our display only rule as one child element of the GerritWorkflow element:

<cn:SubmitRule actionIfNotSatisfied="allow" displayName="Team-Lead-To-Submit"/>

If you switch back to the Submit Rule tab, it should look like this

You probably recognized that this is the first time we used the Not Satisfied Action field, admittedly for a quite exotic use case, namely display only rules. The final step in our power policy will hopefully demonstrate a more common use case to use this field.

Not Satisfied Action for Exception Driven Rules

Step 6 of our power policy is an example of what we call exception driven rule:

“If a file called COPYRIGHT is changed within a commit, a Gerrit group called Legal has to approve (Code-Review +2) the Gerrit change”

Why exception driven? Well, having somebody from Legal approving a change is not sufficient by itself to enable submit, so having a separate submit rule with actionIfSatisfied set to allow is not the answer. Should we then just add legal approval as voting condition to all submit rules which can potentially enable submit? This is probably not a good idea either. Not every commit has to be approved by legal, only the ones changing the COPYRIGHTfile.

Hence the best idea is to keep the existing submit rules unmodified and add a new submit rule which will

I) if evaluated, checks whether legal has approved the change and if not blocks submit (exception driven)

II) only be evaluated if legal has to actually approve the change (if the COPYRIGHT file changed)

Let’s tackle I) first by creating a new submit rule (push Adding Rule Manually button) with display name Legal-To-Approve-Changes-In-Copyright-File and setting Not Satisfied Action to block.

If we kept our new submit rule like this, it would not block a single change as it does not have any voting condition (and hence would always evaluate to satisfied). So let’s add a voting condition that requires a Gerrit group called Legal to give Code-Review +2. The screenshot below shows how this condition should look like. In our case, Legal is a TeamForge user group (group1008).

In the current state, all changes which do not satisfy our new voting condition would be blocked.

Implementing II) will make sure we only evaluate this submit rule (and its voting condition) if the corresponding commit changed the COPYRIGHT file. To do that, we have to click on Next, and switch to the Commit Detail tab which contains all submit rule filters which match characteristics of the commit associated with the evaluated change. The only field to fill in is the Commit delta file pattern. Its value has to be set to ^COPYRIGHT as shown in the screenshot below.

Why ^COPYRIGHT and not just COPYRIGHT? If a filter name does not end with Pattern, it only matches exact values. If a filter ends with Pattern though, it depends on the field value.

If the field value starts with ^, the field value is treated as a regular expression. ^COPYRIGHT will match any file change list that contains COPYRIGHT somewhere. If the field value does not start with ^, it is treated as an exact value. If we entered just COPYRIGHT, this would have only matched commits where only the COPYRIGHT (and no other file) got changed. Keep this logic in mind whenever you deal with pattern filters. Branch filters and commit message filters are other prominent examples where using a regular expression is probably better than an exact value.

If we finish the dialogs and switch to the Source tab, you can see the XML for our new submit rule:

The actionIfNotSatisfied attribute is set to block, we have one submit rule filter (CommitDetailFilter) and one voting condition with a filter (VoteVoterFilter).

Congratulations, you have successfully designed the power policy and can now test and deploy it!

Learning more about the XML based quality gate language

Although you have seen quite a bit of our XML based language so far, we fully realize that we have not shown you every single feature. We do not believe this is necessary though, as our graphical wizard supports all features of the language. If you are unsure how a certain filter works, just create one example with the wizard, switch to the Source tab and find out how to do it properly. Our schema is another great resource as it is fully documented and will make sure that you do not come up with any invalid XML document. Last but not least, our wizard ships with many predefined templates. We tried to cover every single feature of the language within those templates.

For those of you who are familiar with Gerrit’s Prolog cookbook, we turned all Prolog examples into our declarative language and were able to cover the entire functionality demonstrated. The results can be found here.

As always, if you have any questions regarding the language, feel also free to drop a comment on this blog.

How to deploy quality gates without the graphical wizard

As explained before, our Quality Gate enforcing plugin ties into Gerrit’s Prolog based mechanism to customize its submit behavior. Gerrit expects the current submit rules in a Prolog file called rules.pl in a special ref called refs/meta/config. The deployment process for rules.pl is explained here.

Whenever our wizard generates a rules.pl file, it makes use of a custom Prolog predicate called cn:workflow/2which is provided by our Quality Gate enforcing plugin. This predicate has two arguments. The first one takes the XML content as is, the second one will be bound to the body of Gerrit’s submit_rule/1 predicate. In a nutshell, the generated rules.pl looks like this:

submit_rule(Z):-cn:workflow(‘<XML document describing your quality gate policy>’, Z).

Our wizard does not use any other Prolog predicates. You can use our predicate as part of your own Prolog programs if you decide to come up with your own tooling and generate rules.pl by yourself. While passing the XML content, make sure it does not contain any character which would break Prolog quoting (no ‘ characters no newlines or XML encode then). Our graphical wizard takes care of this step.

Final words and Call for Participation

If you made it through the entire third blog post you can proudly call yourself a wizard too 

Designing quality gates from scratch can be a complex matter. Fortunately, our wizard comes with many predefined templates you can just deploy. In addition, we turned any example from the Prolog cookbook into our format. If you are unsure how to match a certain state of a Gerrit change, just use the built in functionality of our wizard to turn it into a submit rule and adopt it according to your needs. Before you deploy, you can always simulate your quality gates within the wizard. It will follow the submit rule evaluation algorithm step by step and shows the evaluation result for every rule. If you do not like our wizard and do not like Prolog either, feel free to use our XML based language independently. This blog post has demonstrated how to do that.

Talking about the XML based language, its specification is Open Source. We encourage you to build your own wizard or other frontends and will happily assist if you have any questions regarding its functionality. Gerrit’s functionality to customize submit behavior is unmatched in the industry. We hope that with our contributions we made it a little easier to tap into it.

Coming up with the wizard, the language and our backend was a team effort. About half a dozen people worked for two months to get to the current state. We like to know from you whether it is worth investing further in this area. Want to have more examples? Better documentation? A tutorial video? A Web UI based wizard? Performance is not right? Cannot express the rules you like to express? Want to use the feature with vanilla Gerrit?

Please, spread the word about this new feature and give us feedback!