Why Every Test Automation Engineer Needs Jenkins

Continuous Integration (CI) is a game-changer for increasing the adoption of Test Automation for every step of the Software Development Process. CI adoption brings various opportunities into the software development process and moreover the process requires an additional professional role known as a Test Automation Engineer. In this article, we will describe how  CI affects the working day from a Test Automation Engineer’s point of view and what are the achievable benefits of Continuous Integration.

We will explain concepts referencing a tool that has become a sort of emblem/synonym of Continuous Integration: Jenkins-CI (briefly Jenkins).

What is the role of a Test Automation Engineer?

Test Automation Engineers, for example, are a mandatory building block that allows this challenging scenario to succeed. TAEs’ main task is to validate software quality under the continued compression of working time. Every software development cycle contains more features and requires more validation.

Test Automation Engineers come into action during the software life cycle process in specific phases and can affect the execution in an important manner:

• Acquisition of software, product requirements with an orientation towards test automation, it’s pretty clear TAEs do not have a crystal ball;
• Preparation of test environments and all necessary procedures and/or infrastructure to develop and to execute automated tests;
• Developing automated tests using the acquired requirement/specification executed in a prepared environment.

Moreover, one important activity for TAE is the selection/configuration of tools that best fit automation needs. This is where Jenkins comes into action to integrate automated tests into the existing CI process flow.

It’s important to mention two questions related to the time plan of a Test Automation Engineer:

• How is the TAE’s workload composed?
• How long is the TAE involved in the testing process?

To answer the first question consider that a TAE mainly interacts with software tools shaping them with necessary features (e.g. Configure Jenkins plugins) with the goal to define a Test Automation Solution that best fits the project testing scope (e.g. JUnit with Webdriver).

The second answer is that “he can collaborate for a long time but probably in a discontinued way”. To better describe this we must separate the TAE workflow during the test automation process/roadmap into phases:

• Test Automation Solution setup workload can be important and provided continuously during its execution (e.g. Select tools, define the test development process, produce proper documentation, etc);
• Recurrent activities can be less heavy in terms of workload and continuity. An important recurrent activity is developing a new test case for the features in the next delivery. Perhaps if the next delivery introduces fewer testable features less effort is necessary.

What is Continuous Integration?

Continuous Integration is a well-known methodology used in the software development process based on continuous environments updating via software versioning tools.

   

In the above picture is a synthetically summarized basic CI workflow with three parts:

• Software Configuration Management acts as historical memory for software (both the product under test and test automation);
• Build Automation Server acts as a reference for each automatic step related to handling code like compiling, testing, reporting, packaging, etc;
• Delivery Manager has the responsibility of interpreting reports provided by BAS and allows (or not) delivery of current artifacts.

The Test Automation Engineer’s pertinent area is in the middle of the two red arrows visible in the picture above. TAE provides the required software and infrastructure to deliver readable reports from automatic test executions.

How is “Continuous Integration” related to Jenkins?

In a previous chapter, we focused our attention on the “Build Automation Server” role and the involvement of a Test Automation Engineer in its management to achieve the necessary test automation execution chain into CI methodology. Now we want to go deeper using a reference software application into BAS scenario: Jenkins.

Jenkins is an open-source build automation server written in Java with a rich plugin ecosystem built for the purpose of Continuous Integration. Jenkins’s base concept is the definition of the job execution starting from the SCM system to a deliverable artifact. Thanks to its architecture plugins are continuously increasing in terms of variety, features, and the support of third-party tools. The complexity of the build automation pipeline is not important. With Jenkins, it is possible to customize both the granularity of the CI pipeline (e.g. one big fat job or many small chained jobs) and which specific one to use (e.g. approach based on a third-party plugin or write in-house job code).

In this article, we will use “CI pipeline” as a sort of synonym for Jenkins because we will code the entire pipeline logic from a point of view into Jenkins. Typically we compose a generic CI pipeline with many steps and only some of them are important for TAE activities (e.g. source code branching handling, delivery of the reporter). This situation typically shares the responsibility of Jenkins among multiple professional roles. Perhaps it is important to have a competent matrix strategy (e.g. by running a job).

Jenkins provides a friendly user interface to broke in subtask a complex CI pipeline enriched by an extensive plugin ecosystem that simplifies technical integration with other tools/frameworks.

Why Test Automation Engineers need Jenkins

This chapter will describe some practical aspects that any Test Automation Engineer must consider when it’s planned to work with Jenkins.

Jenkins is the de-facto standard to realize a CI workflow and its integration with test automation activities. The only limitation to the previous phrase is when there are constraints related to company policies that limit Jenkins adoption but today these are rare cases.

 

Following the above picture there are three sections:

• Top, here we have categories of tools that require service from Jenkins (e.g. Agile dashboard, test management);
• Middle, build automation server category so we place Jenkins there;
• Lower, The tools used to perform raw activities at a low level (e.g. Testing framework, command-line interfaces, shells, compiler, virtual machine, etc..).

Into the “Top” category more than 90% of tools have for free/paid a plugin that allows integrations with Jenkins (e.g. launch job, show result summary, link to the detailed report, etc).

• Into the “Lower” category there is no practical limit to integrate any tools included here. There are two possibilities for integration with the target tool:
• The existence of a native integration or plugin developed by the community. In this case, TAE’s duty is to install the necessary plugins and study online documentation on how to handle the web interface;
• There is no integration. In this case, shell scripting or Groovy language resolves complex issues (bash or PowerShell).

Jenkins follows an MIT license. Anyone, in general, can access its source code of:

• Extensions and introduce a new feature into the Jenkins platform (typically with plugins);
• Participate in the resolution of a discovered bug in various ways (reproduce configuration, testing, troubleshooting, etc).

The possibility of gathering technical support is another important benefit (e.g. implement a new task, and resolve encountered bugs) in various ways. We base this on the final CI customer format, cost, and background.

Format and cost typically go together in every product under the open-source umbrella so we describe it together:

• Via the community without any external cost assigned, looking on internet forums through shared experience to understand how to solve current requirements/issues;
• With consultancy and the external cost assigned. In this case, CI activity loses the technical profile and becomes a commercial one with parameters like cost and risk.

Handle the related background separately because it carries important advice for readers related to the final CI customer core business:

• If the core business is software development, it’s straightforward that handling the necessary technical skills in house and interacting with the community is the first choice;
• If the core business is not software development, here we need to suggest a commercial partnership to reduce the technical risk for setup and maintenance of the CI process.

As mentioned in the previous section, Jenkins has no licensing cost to obtain/use the application software. Now we will focus on the correlated costs that can affect a hypothetical business case taking into account when speaking of infrastructure (e.g. machine, network, etc.) and workload (e.g. man-hours to set up) correlated to Jenkins activities.

Jenkins can execute a generic task in multiple configurations (e.g. with/without a specific plugin, with/without clustering, etc.). In general, there isn’t a unique “best way” to implement a test process flow on Jenkins because each test project can bring different situations.

This consideration brings two thorny questions typically made to Test Automation Engineers in the early phase of Jenkins setup for test automation. IT members make the first question.

Which kind of resources do you require to implement and use Jenkins?

It’s a simple question but it’s necessary to weigh it with care. It’s reductive to provide only information about how the amount of CPU, RAM, and Disk (taken from an internet search for “Jenkins system requirements”). The suggestion is to consider the points below before commissioning a machine for Jenkins installation.

Will Jenkins:

• Be used for a single project?
• Execute subprocess time/resource consuming (e.g. batch shell script)?
• Require custom firewall policies?
• Need collateral licensed software (e.g. Oracle JDK rules here)?
• Have a 24h available machine?

These questions have the scope to better realize which IT requirements face mandatory implementation. A good practice would be to analyze all of them soon to avoid future wasted time (and maybe with an incoming deadline).

The Project managers pose this second question where Jenkins setup will be billed and it sounds like:

How much effort does using Jenkins require?

As already described in this section the creation of new CI pipeline steps requires a startup effort based on involved technologies and deliverable targets. After a startup effort typically it is suitable to allocate less effort for maintenance or reworking activities that periodically can occur. Generally speaking, the startup effort is not always mandatory because organizations tend to have already used Jenkins as a reference tool and there is already a generic installation ready to be populated with tasks and pipelines when necessary.

Here we describe some general rules to better break down activity and evaluate the corresponding effort. Many of the below questions are influenced by experience and similarities with other projects. Perhaps for the already experienced reader below the list can be unnecessary but we will include all details for completeness.

• Are necessary observation points available to under test services?
• Do all necessary test technologies have the corresponding Jenkins plugin?
• Are there time-consuming activities involved (e.g. long execution batch script)?
• If used as a plugin. Are there are software compliance criteria? (e.g. Web browser & Selenium)
• There are constraints on execution time (e.g. scheduled, timely execution, etc…)?
• Is documentation required for a new collaborator (e.g. new test developer)?
• Does a CI flow already exist or is it necessary to define a new one?
• How to format reports and which channels to share?
• Can CI pipeline steps be parametric? In which manner?

After setup completion, the Test Automation Engineer’s main activities change as already described. Typically activities directly related to Jenkins lessen and only increase as important automated tests are developed and executed. You should plan this activity as a dynamic effort.

How can Jenkins integrate with Experitest?

In this final chapter, we will describe a practical integration example between Jenkins and Experitest using a free plan. This example will come from the point of view of a Test Automation Engineer with the duty to set up this integration.

Typically this kind of activity starts with a telegraphic mail from the project manager with a few requirements and we report an example:

• Test Automation code ready for execution and located here
• The running platform is located in the Experitest SeeTest Cloud
• Feel free to set up a new Jenkins installation

Maybe these requirements are a bit synthetic but they provide the TAE with a starting point to break down them and to realize which activities are really required.

Input test automation code

Luckily it’s a Github repository with standard Maven project that includes Selenium test suite developed in Java 8 for execution on Experitest’s platform. Perhaps the TAE takes a note related to Maven and the required Java version because Maven and Experitest solve other dependencies.

Experitest SeeTest Cloud

Following online documentation as TAE we proceed with two preliminary subtasks:

• Register a free trial account providing a mail address and phone number
• Extract from the online dashboard our “Access Key” to grant remote execution

Setup a new Jenkins installation

Here we have the challenge of this TAE activity. Setting up a new Jenkins installation. As a first step we approach Docker virtualization to create a ready environment and after that place it in “production”. Perhaps we search on Docker Hub choosing the latest Jenkins image and setup new environment in three steps:

1. Setup a Docker machine (e.g. a laptop with good RAM dotation)
2. Create a Docker volume to grant Jenkins data persistence
3. Execute Docker container with latest Jenkins version

The first step is Docker installation and detailed information is here according to the host operating system. This step concludes when we have Docker running on our machine with the command docker.

The second step is pretty simple and it is one line of command:

docker volume create jenkins_home

The third step is simple to perform but it involves a series of consideration and activities but directly on Jenkins:

docker run –rm -p 8080:8080 -v jenkins_home:/var/jenkins_home –name=jenkins Jenkins/Jenkins

With this command, our machine starts to collect resources over the Internet (e.g. fetch Jenkins plugins) and produces many logs (useful for troubleshooting) and when it concludes we have the link

http://localhost:8080

a Jenkins environment is ready to be configured for our scope.

First, we need to unlock the administrator grant providing a random password showed after docker invocation, we provide this password into a user interface like below.

   

After Jenkins proceeds with the installation of plugins for the first time, now we can select the default plugins set and wait for a while.

   

To conclude this phase we choose admin credentials and conclude the setup procedure. Congratulations! We have set up a Jenkins environment in less than 5 minutes (your connection speed will influence the download time).

Perhaps this environment is an empty shell and the duty of the Test Automation Engineer is to fill it with reliable test automation tasks. According to the input requirements extracted by test automation, these are the next steps:

1. Install Maven and Java 8
2. Create a new test automation task
3. Link task to Git repository
4. Link task to Experitest
5. Setup invocation of maven test goal
6. Execution of just created task

Install Maven and Java 8

Logged as admin into Jenkins is possible to handle menu “Global Tool Configuration” as shown below

   

Scrolling the tools page we can find the section related to Maven and configure it as shown below

   

After we need to search the JDK section into the same page and again configuring it as shown below

   

With these configurations, we have given instructions to Jenkins to download/install Maven and JDK locally at first utilization. It is important to adopt name conventions into each Jenkins task to ensure that you use the correct software version (e.g. in this example “maven” and “JDK” strings).

Create a new test automation task

Click on “New Item” button to fill the name field selecting the “Freestyle project” template.

   

After confirmation we land directly on the task configuration where we configure the Maven task invocation into the “Build Environment” section as shown below:

   

We reuse configured Maven installation label as shown below.

   

Link task to Github repository

Here we need to join our just created task with a remote Git repository via the “Source Code Management” section as shown below.

   

Shortly we select “Git” and add URL provided into the input requirement leaving other fields as default.

Link task to Experitest

Provided test automation code is ready for Experitest integration and only requires an “Access key” from working account.

   

We add this key via the “General” section with the steps listed below:

• Select “This project is parameterized”
• Add “Password Parameter”
• As name “SEETEST_IO_ACCESS_KEY”
• As default value our “Access key”

In this manner when the task is executed there is an environment variable passed to JVM with our access key and running test fetches it. Conclude this step save the entire Jenkins task via the user interface.

Execution of just created task

Time to run! Take into account that the first run was special because tools will be fetched for the first time. Then search and click the button below for just saved task

   

Leaving unchanged default value for access key and press “Build”

   

Looking into the left panel it’s possible to see the running task and we open its “Console Output” panel. Into this first run we can see in order:

 

Cloning Git repository

 

Installation of Maven

 

Installation of Java

After we can see the execution of automatic test code with lines

   

Here test execution is performed on the Experitest SeeTest Cloud where will be allocated necessary instances (e.g. browser, mobile devices, etc) and performs our test logic. As a result, when the test is finished we see it directly in the report panel.

   

If all is working well we have concluded the Jenkins environment setup phase and we can promote this environment for production.

But how?

Remember our Docker volume for data persistence? After the docker container stops, we can take what volume contains and move it to another machine (this time tailored for 24h running), and with a few adaptations, we can use it as starting point for Jenkins instance for all test automation engineer tasks.