Skip to main content
Experitest Partners with Infosys to Offer End-to-End Mobile Testing Solutions Worldwide1

This post is from the Experitest blog and has not been updated since the original publish date.

Last Updated Jul 09, 2019 — Continuous Testing Expert

Experitest Partners with Infosys to Offer End-to-End Mobile Testing Solutions Worldwide1

Continuous Testing


What is TestNG Framework


TestNG is an automation testing framework. The NG-suffix abbreviation stands for "Next Generation". The concept of TestNG is similar to the more familiar JUnit which uses the annotations (@).


TestNG Framework provides vast opportunities for test engineers for example:

  • Multiple executions of the same test are without looks are possible by using mechanism (annotation) called 'invocation count.'
  • Failed tests can be re-run only by running the tests that are in TestNG-failed.xml file (test-output folder).
  • TestNG automatically handles uncaught exceptions without terminating the test prematurely. TestNG reports these exceptions as failed steps in the report.
  • Different tests are arranged into groups by placing them into the TestNG.xml file.
  • Cross browser testing allows the execution of multiple test cases on multiple browsers.
  • Reports are generated with respective skip, failed and passed percentages.
  • Annotations used in testing are self-explanatory ex: @BeforeMethod, @AfterMethod, @BeforeTest, @AfterTest (


In other words,

  • TestNG annotations are self-explanatory and easy to understand
  • Test cases can be organized in groups
  • Parallel testing is possible


How to build and set-up a project from scratch


To start, here is a video-snap for project creation from scratch in the IntelliJ IDEA based on java, maven, TestNG:  (video-snap).


Important note: Before creating a project from scratch - please ensure that you have java and maven installed locally. ‘java -version' and ‘mvn --version' commands in the command line should provide exact versions installed.

IntelliJ Maven

Meanwhile, the maven-archetype-quickstart is supposed to be selected when creating a project from scratch.


GroupId and ArtifactId should be given. Please see the format: of the GroupId and ArtifactId.


Finalized Test Project Structure


This section provides a high-level overview of what the structure of test frameworks built over TestNG, Page-Object (using java and maven - as dependency manager for the project) should look like. Please see the video-snap:


How to create tests using Page object-pattern


Pros of using the Page object:


The Page Object Pattern is a decoupling layer. To clarify, a Page object encapsulates web elements descriptions and methods operating on the described element. In contrary to the direct locators and web element references from a test scenario. Most importantly, the page object pattern approach makes a test developer's life much easier especially when the HTML layout is liable to changes.

TestNG framework html test

Pic. 1. Schema of implementation of when a test script refers directly to web-elements locators. An example of bad test design: in case the HTML layout changes the test developer will have to look up changed (affected) web elements, and manually modify locators in different framework places.

TestNG framework html

Pic. 2. Schema of an implemented page object. In Selenium/Appium tests test developer refers to the instances of page objects. Every page object encapsulates elements and methods operating on the elements. In case a page changes the only modification to do is in Page Object, in a single place. Implemented tests remain unchanged.


Getting started with this implementation is actually easy and straightforward:

  1. Basic Selenium project setup (Java, Maven, Git).
  2. Perform analysis of the application under test
  3. Create page objects.
  4. Implement tests
  5. Organize implemented tests into XML test suites
  6. Set-up Selenium grid INFRA
  7. Set-up the CI/launch tests

Some examples of Page object are below including a Selenium test scenario. For demo purposes, we will open the Google landing page and do a simple search (query) there.


For the purposes of describing elements on the page - Google Chrome DevTools > Elements tab.

TestNG framework google

Pic.3. Using this tab and just entering the ‘Ctrl+F' keys combo test gives developers the ability to isolate and find web elements covered and described in the Page Object pattern.


Firstly, let's implement the base Page class that will be a parent for all the page entities. Click here for the full page object entities code source.


An important note here: We will also need this parent class for the purposes of initialization of page objects because page-object will not work unless we initialize the WebElements that we have annotated. So superclass holds this small but important piece of code.


Let's move on to implementing a Page Object representing the Google landing page. Click here for the full code source.


After that, let's consider that we've implemented a page object, and proceed with creating a test scenario.


Simple test scenario(-s)


For simplicity and demo purposes let's consider the following use case (test scenario):


A user opens a Google landing search page, then the user enters 'Experitest' and 'Appium' in search input, and after that analyzes results.


First, we're going to create a parent test class that will contain common stuff for test classes. Click here for the full code source.


And test scenario implemented respectively. Click here for the full code source.


As we can see, the steps implemented in the scenario are pretty self-explanatory. As a result, this implementation helps us keep tests more flexible to adjustments and more flexible in terms of maintenance.


Please also note how ‘Group' annotations are implemented and the way how some specific test is assigned to the test group:

@Test(groups = { "REGRESSION.SUITE" } )


If needed, we may apply multiple ‘Group' annotations, e.g:





In TestNG framework, everything executes via the Suite. Subsequently, Test and Methods and the Listeners give us the ability to add specific action(-s) before and after of every Suite, Test, and Methods.


In technical terms, TestNG Listeners are interfaces that allow you to modify the TestNG framework's behavior. Here are a few listeners:

  • IAnnotationTransformer
  • IAnnotationTransformer2
  • IHookable
  • IInvokedMethodListener
  • IMethodInterceptor
  • IReporter
  • ISuiteListener
  • ITestListener

For more details please see the TestNG spec


How to rerun a failed test. Retry mechanism (implemented as Listener)


In the case of a test failure, test developers are often eager to ensure that the reason for the failure is a functional bug but not connection lag, non-stable environment or any kind of temporary abruption.


For these purposes, there is a great mechanism called the re-try mechanism. Simply speaking it is a mechanism that reruns the same test in case of the failure X-times and only after X-number of failures is the test considered as failed.


First, we need to implement the IRetryAnalyzer interface. Click here for the full code source.


At this point, we're ready to implement the listener's logic itself. The full source is here..


How to invoke Listener


There are two simple options we may invoke listeners in: either in TestNG.xml or in test classes. Let us compare both options.


Listener in TestNG.xml


Click here for the full code source.


Here is the full source for a listener in Test class (in java code).


Given that listeners are added in test class there is no need to add them in the TestNG.xml test suite now.


Multi-thread (parallel) vs single-thread execution (different options in TestNG XML test suites). Examples of TestNG XML suites.

TestNG.xml suites contain a huge amount of parameters. Let's consider the parallel parameter and the values it may take. In terms of TestNG and the annotations we have in TestNG.xml - we may do the parallel execution for tests, classes, and methods. The parallel attribute on the tag can take one of the following values:

<suite name="Demo test suite" parallel="methods" thread-count="10"> <suite name="Demo test suite" parallel="tests" thread-count="10"> <suite name="Demo test suite" parallel="classes" thread-count="10"> <suite name="Demo test suite" parallel="instances" thread-count="10">  

Please have a look at the full code example here.


Here we have 2 tests, where 1 test represents sub-set of all test methods annotated with "SMOKE.SUITE". Considering suite config and parameters are <suite name="Parallel testNG tests suite for demo purposes" parallel="tests" thread-count="2" verbose="8"> that means that we have created 2 browser instances simultaneously. Use one browser instance for the first test (where the sub-set of all test methods annotated with "SMOKE.SUITE" is to be executed) and another browser instance - for the second test (where sub-set of all test methods annotated with "REGRESSION.SUITE" is to be executed).


To Summarize, before you start implementing the framework architecture, please understand whether you need to have tests executed in a parallel way and what kind of parallel testing you actually need. The TestNG framework gives us a solution for the parallel test implementation.


Sometimes it is crucial to have everything executed in a consecutive way (i.e in 1 thread). That may be accomplished by an easy flag value change to parallel="false" .


<!DOCTYPE suite SYSTEM "">

<suite name="Parallel testNG tests suite for demo purposes" parallel="false" verbose="8">



Appropriate Selenium GRID config for parallel execution support

TestNG framework selenium

Selenium GRID representation: Selenium HUB and Selenium nodes communicate with each other. Considering we have 2 parallel threads for execution in this demo project, we will need a hub and 2 nodes. Also, 1 thread will be executed in a Chrome browser instance and another thread - in a Firefox browser instance.



Click here to see the source code for JSON Selenium GRID config: Once this has all been prepared, the start of the grid is pretty straightforward. As we are working on a Windows machine I will provide Windows-specific commands for launching the grid:


  1. java -jar Selenium-server-standalone-3.11.0.jar -role hub -hubConfig hub.json
  2. java -jar -Dwebdriver.gecko.driver=geckodriver.exe Selenium-server-standalone-3.11.0.jar -role node -nodeConfig node1.json
  3. java -jar -Dwebdriver.gecko.driver=geckodriver.exe Selenium-server-standalone-3.11.0.jar -role node -nodeConfig node2.json


To ensure that everything has started correctly - please open: http://localhost:4444/grid/console:

selenium grid console

Maven configuration for test execution


A suite might execute via the following command: mvn clean test


Please note: this should be called from the project root. Alternatively, we may create a Maven run configuration in the IDEA:

TestNG framework rundebug

Take a look at the video-snap of execution and tests results analysis (of Allure HTML report):


Experitest. Digital assurance lab for test execution and its benefits


Experitest is the world's leading provider of quality assurance tools for mobile DevOps, including manual testing, performance testing, load testing and monitoring for mobile applications. Experitest tools support all mobile OS, iOS, Android, Windows Phone and Blackberry.


I got acquainted with Experitest products and one of my favorites is the Digital assurance lab. It allows access to hundreds of browsers & mobile devices (physical/emulated) Hosted in Experitest data centers, from anywhere at any time. There is a large pool of browsers, physical mobile devices & tablets, simulators and emulators available for selection. The lab includes a variety of manufacturers, models & OS versions, including newly released device models and OS beta versions.


I've managed to integrate it into my QA test automation environment with almost no effort (just following the manual provided). Another great benefit worth mentioning here is the possibility to run tests in parallel mode. Given that most projects demand a high percentage of functionality coverage and when test developers have hundreds and thousands of tests to execute - parallel execution is indeed very helpful and less time-consuming.


And also I like the result analytics dashboard - a cool feature of Digital assurance lab, providing test results and prettified HTML representation of the executed tests in a very detailed way (tests suites, groups, timelines, percentages, tests steps, etc).


Most importantly, as test developer that makes test debugging effortless.


Please see more details on the parallel browser and mobile execution: and




For reporting purposes in the demo project, we will use Allure reports. To use Allure - we need to add a couple of dependencies in pom.xml (


As soon as we added Allure dependencies we are eligible to use Allure annotations in the test classes.


Please consider that the Allure reporter also has a large number of settings, that are specified in the public spec:


Once all of the tests are executed, all of the results including the .xml file and screenshots (if any) are usually placed in the following folder: TestNGexample\target\Allure-results. So to generate a clear and well laid out HTML report based on the resulting XML, test developers should open the TestNGexample/target folder and execute the Allure serve command via the command-line Allure tool ( :


Demo Code source


The Complete test projects used for demo purposes are placed in my GitHub repository: All are verified and working.




In short, this article provides how-to instructions for some of the cool features of the TestNG framework. These include organizing tests into separate groups, organizing tests and setting them up for parallel execution, simple steps how to create a test project from scratch based on Java and Maven as dependency manager and also introduces the Page Object pattern - one of best practices that are widely used in QA world nowadays.


Eugene Polschikov - QA automation expert


More from the Blog

View more
Jul 29, 2022

The automation testing advice that management most frequently ignores

Continuous Testing
No one likes to be told “I told you so” but sometimes the people doing ...
Read More
ios 16
Jun 07, 2022 Continuous Testing, first in market to support iOS 16 (Beta)

Continuous Testing is proud to announce that our Continuous Testing solution i ...
Read More
May 31, 2022

How automated/continuous testing solutions help create more secure mobile apps

Continuous Testing
Mobile app security risks climb as remote work increases Security o ...
Read More
May 13, 2022

What tips would you give testers on how to ask better questions?

Continuous Testing
The nature of testing is not about confirmation. It is about questioni ...
Read More
Contact Us