Skip to main content
How to Select the Right Test Automation Framework for you

This post is from the Experitest blog and has not been updated since the original publish date.

Last Updated Feb 27, 2019 — Continuous Testing Expert

How to Select the Right Test Automation Framework for you

Continuous Testing

What is a test automation framework?

 

The more software systems increase in complexity as new development practices emerge, the more the quality assurance and software testing requirements rise. Now, among a number of testing methods, automated testing is the most common candidate to enhance your quality assurance process. At first glance, automated testing may look like a very difficult and resource consuming choice. But, diving into the matter deeper, you may find out that there are a number of established practices, rules, tools and libraries that significantly simplify the process of designing your test cases - test automation frameworks. Basically, test automation frameworks are tools for empowering and simplifying automated testing.

 

Test automation frameworks provide a number of benefits: lower expenses, lower entry threshold for testers, more readable and reusable code.

 

However, there are a lot of test automation frameworks available. In this article I'll cover the most popular ones, that have rightfully become test automation industry standards:

  

Each of these testing frameworks provides its own benefits and applies its own limitation to the test automation process. The differences include supported programming languages, test case organization, a level of applicability. I will describe the benefits and limitations of each framework in details below. The goal of the article is to make it clearer which tests automation framework will meet your needs.

 

JUnit

junit - test automation framework

JUnit is a test automation framework for the Java programming language. As one of the oldest, yet constantly evolving frameworks it holds a leading position in the world of automated testing. As it utilizes Java, it has one of the biggest benefits - cross-platform support.

 

In JUnit, test cases are represented by Java objects, where a simple annotation system manages the execution of test methods. Like in this code.

  [java] import java.net.MalformedURLException; import java.net.URL; import io.appium.java_client.android.AndroidDriver; import io.appium.java_client.remote.MobileBrowserType; import org.junit.*; import org.openqa.selenium.By; import org.openqa.selenium.remote.DesiredCapabilities; public class AndroidWebTest { private static final String ACCESS_KEY = System.getenv("SEETEST_IO_ACCESS_KEY"); private static final String CLOUD_URL = "https://cloud.seetest.io:443/wd/hub"; private static final String TITLE = "Testing Website on Android Chrome with Java"; private AndroidDriver driver = null; @Before public void setUp() throws MalformedURLException { DesiredCapabilities dc = new DesiredCapabilities(); dc.setCapability("testName", TITLE); dc.setCapability("accessKey", ACCESS_KEY); dc.setBrowserName(MobileBrowserType.CHROME); driver = new AndroidDriver(new URL(CLOUD_URL), dc); } @Test public void testAppiumOnChrome() { driver.get("https://amazon.com"); System.out.println(driver.getTitle()); if (driver.getCapabilities().getCapability("device.category").equals("TABLET")) { driver.findElement(By.xpath("//*[@name='field-keywords']")).sendKeys("iPhone"); driver.findElement(By.xpath("//*[@text='Go']")).click(); } else { driver.findElement(By.xpath("//*[@name='k']")).sendKeys("iPhone"); driver.findElement(By.xpath("//*[@value='Go']")).click(); } } @After public void tearDown() { if (driver != null) { driver.quit(); } } } [/java]  

Here, the methods are executed in the following order:

 

@BeforeClass methods will be executed once before all the methods, annotated as @Test

@Before methods will be executed before each of the methods, annotated as @Test

@Test method

@After methods will be executed after each of the methods, annotated as @Test

@AfterClass methods will be executed once after all the methods, annotated as @Test

 

Aside from annotations, that manage the order of test methods executions, there are annotations to parametrize tests and set up custom rules for additional flexibility.

 

Basically, JUnit will suit you in the following situations:

 

If you rely on a cross-platform solution.

 

If your testers have experience in Java.

 

NUnit

nunit- test automation framework

NUnit is a port of JUnit to C# language. So, if you want a .NET solution, then you should consider it as one of the most viable choices. Basically, as a port, it allows you to write JUnit-style automated tests in C#, using similar techniques, such as annotations.

 

Click here for the example code.

  [java] using System; using NUnit.Framework; using OpenQA.Selenium.Appium; using OpenQA.Selenium.Appium.Android; using OpenQA.Selenium; using OpenQA.Selenium.Remote; namespace Test_Selenium { [TestFixture] class AndroidDemoTest { protected static readonly String ACCESS_KEY = Environment.GetEnvironmentVariable("SEETEST_IO_ACCESS_KEY"); protected static readonly String CLOUD_URL = "https://cloud.seetest.io:443/wd/hub"; protected static readonly String TITLE = "Selenium/Appium Android test"; protected DesiredCapabilities dc = new DesiredCapabilities(); private AppiumDriver driver; protected String testName = "Selenium/Appium Android test"; [SetUp] public void Initialize() { dc.SetCapability("platformName", "Android"); dc.SetCapability("app", "com.example.tester.myapplication"); dc.SetCapability("appActivity", ".MainActivity"); dc.SetCapability("appVersion", "1.0"); dc.SetCapability("accessKey", ACCESS_KEY); driver = new AndroidDriver(new Uri(CLOUD_URL), dc); } [Test] public void AndroidNativeAppTestExample() { driver.FindElement(By.Id("fab")).Click(); String text = driver.FindElement(By.XPath("//*[@text='Hello World!']")).GetAttribute("text"); Assert.AreEqual(text, "Hello World!"); } [TearDown] public void TearDown() { if (driver != null) { driver.Quit(); } } } } [/java]  

As you can see, NUnit is very similar to JUnit, it follows the same test design philosophy, utilizes a similar annotation mechanism. So, if you need a testing framework to organize automated testing in C#, then NUnit is the best choice.

 

TestNG

testng- test automation framework

TestNG is, again, a new approach to JUnit. Like JUnit, TestNG utilizes Java programming language, inheriting all the benefits of Java cross-platform support.

 

What is the differences between TestNG and JUnit? Well… TestNG is striving to be a more configurable framework, allowing some additional grouping features as well as more hook annotations. For example, in TestNG, you can group your test classes to Groups and then group Groups to Suites. Besides, it supports annotations such as @BeforeGroup/@AfterGroup, @BeforeSuite/@AfterSuite, and others.

 

Choosing between JUnit and TestNG is not an easy task, so, usually, it is a matter of personal preference.

 

Cucumber

cucumber - test automation framework

Compared to all the previous examples, Cucumber is a different beast. Following aspects of the behavior-driven development paradigm, Cucumber describes high-level test scenarios in the Gherkin notation, very similar to a human language. For an example click here.

 

However, when you have Gherkin scenarios, you need to implement the steps in a more low-level programming language. Cucumber supports steps definition in a lot of languages like Java, JavaScript, Ruby, Kotlin.

 

a href="https://github.com/jonnysteiner/How-to-Select-the-Right-Test-Automation-Framework-for-you/blob/master/step%20definition%20java" target="_blank" rel="noopener noreferrer">An example of step definition in Java:

 

Besides that, you can use Cucumber in conjunction with JUnit, TestNG or NUnit framework to enhance your test cases with additional features.

 

So, you may consider Cucumber if:

 

You are more interested in high-level test scenarios.

 

You have a team with different programming experience, so you can distribute tasks of writing Gherkin test cases and implementation.

 

However, you should be aware that Cucumber is not designed for more low-level tests such as unit tests and API tests.

 

Experitest/SeeTest integration

 

All of the test automation framework from the examples above integrate with the SeeTest Platform in a simple manner. No matter what testing framework you have chosen, you can use Selenium and/or Appium drivers and specify access key and URL capabilities. Here is an example for Java programming language.

 

In order to make this code work, you should add a new system environment variable "SEETEST_IO_ACCESS_KEY" and set its value to your SeeTest access key. Then you can refer to the driver variable to locate your device. This approach works with all the frameworks above, only programming languages may differ. When you run your tests, the driver will connect to the SeeTest cloud and execute your test.

 

Conclusion

 

So, to sum the things up:

 

If your preference is a Java programming language, then JUnit is a great choice for you. The main advantage of JUnit is its popularity.

 

If you are looking for a solution to work with the .NET framework and C# programming language, then Nunit is an obvious choice.

 

TestNG, being very similar to JUnit provides some extra features to organize your tests. It is slightly less widespread than JUnit, though.

 

Cucumber supports a great number of programming languages and offers a kind of unique ability to have test cases be described in Gherkin language and steps defined on the programming language of your choice. However, it seems to be not very convenient to use it for testing anything on the lower level than business/functional

 

Currently, there are enough test automation frameworks available for use. Some of them are very different from the others, some are similar. The main benefit of adopting one or another framework is that it will allow you to keep your tests uniformed, standardized and accessible.scenarios.

 

Konstantin Tonkov - Performance Engineer ISSArt QA team

 

More from the Blog

View more
Apr 12, 2021

These blockers most commonly stop people from learning test automation

Continuous Testing
We are often asked if it is hard to become a test automation engineer. ...
Read More
Apr 05, 2021

Build better apps with a low-code development approach

Continuous Testing
From being an option, DevOps is quickly turning into a necessity for m ...
Read More
Mar 29, 2021

Every web and mobile app tester needs these 7 surprising software testing skills

Continuous Testing
If you look at a job listing for QA or test automation professionals, ...
Read More
Mar 24, 2021

Paving the way for better mobile and web application testing orchestration

Continuous Testing
Keeping applications up to date and ensuring they work with all of the ...
Read More
Contact Us