How to set up and run a mobile test automation project in 45 minutes

Agile software development is profoundly impacting mobile test automation. Above all, apps are released on average every 2 weeks and require speeding up the entire development process. CI/CD practices play an important role here, but, project teams often find that their test environments do not allow complete automation. In addition, the growing mobile device matrix compounds the complexity. Follow along as we set up a complete continuous testing pipeline within 45 minutes!

How did we implement mobile test automation in 45 minutes?

We will begin by describing the tools used to manage mobile app test scripts for a range of platforms and OS. We also utilize a CI environment to automatically trigger tests on multiple devices in parallel. In this demonstration, we will use the following technology stacks and tools:

• Development language and driver: Java, Appium;
• IDE: Eclipse, IntelliJ;
• Testing framework: TestNG;
• Source code management: Git;
• Build tool: Gradle;
• Continuous integration: Jenkins;
• Device cloud: SeeTestCloud;
• Analytics: SeeTestReporter.

We will now set up a mobile test automation project in 45 minutes, as follows:

1. Create a Git repository

Our first step is to set up a Git repository for our project.

This involves the following steps:

• Assign a name to the repository.
• Enter an optional description of the repository.
• Initialize the repository.
• Share the details with peers by sending an invitation. This allows the peers to contribute to the repository.
• Peers can accept the invitation and access the repository.

This step needs 51 seconds.

2. Clone a project using Eclipse IDE and Appium Studio for Eclipse plugin

Our next step in this mobile test automation demonstration is to clone an available sample project in Ecplise IDE. We will use the Appium Studio for Eclipse plugin here. We use Gradle as the build tool, and TestNG as the test framework for continuous testing. In order to access a large number of Android and iOS mobile devices, we will use the SeeTest device cloud.

This step involves the following activities:

• Use Eclipse IDE, with the Appium Studio for Eclipse plug-in.
• Clone the project created recently, copy the location of the clone.
• Create a new Gradle project using Eclipse.
• Use the location information of the cloned project while creating the Gradle project.
• Enter a name for the Gradle project.
• The project will use mobile devices in the SeeTest data center.
• Finalize the Git project configuration, which will create a Java project of Git nature.
• Delete the builds automatically provided since we will use our own build here.
• Right-click on the project, go to “Configuration”, select the “TestNG nature” option.
• This loads the various test builds, e.g., Android, iOS, etc.

This step requires 3 minutes and 28 seconds.

3. Select a device from the SeeTest device cloud and install the app on the device

We are using the full capabilities of the SeeTest device cloud in this demonstration. To start with, we will select an Android device from the device cloud, and integrate it into our mobile test automation project. We then install the app from the sample project on this Android device and launch it.

The following detailed activities are part of this step:

• Select the devices to work with from the SeeTest cloud device lab.
• Open the device.
• Once the device reflects on the screen, select the demo Android application for this project.
• Install the app on the selected device.
• Launch the app on the device.

We took 1 minute and 20 seconds for this step.

4. Prepare for writing an Android login test

In this step, we take the login functionality of our app and prepare to write a test for that in Eclipse. We see the relevant UI page and the elements there. The properties of the UI elements are available for us to see. At this point, we can start writing a test for the Android login functionality.

This step includes the following tasks:

• Duplicate the Android login test, and assign a name to the new test.
• Open the new Android login test.
• Drag and drop the application into the test to configure it, which automatically adds all the needed capabilities.
• Take a dump of the elements on the login screen, and assign a name to it.
• Select the elements in the screen, which shows their properties.
• Drag and drop the elements like the user-id field, password field, login button, etc. into the repository. Assign names to them.

It takes 2 minutes and 44 seconds to complete this step.

5. Write a test and execute it for the Android login. As a next step, we write a test for the Android login, as follows:

• Use the “Send Text” widget to send values to the user id and password fields.
• Incorporate the action of clicking the login button.
• Provide the values for user id and password.
• Use the “Override capability” option to execute the test only on the specific device selected, and not just on any Android device in the cloud.
• Execute the test.
• Verify the successful execution of the test.

This step takes 1 minute and 37 seconds.

6. Utilize Appium Studio for Eclipse features to easily write more tests

Testers might not always be experts in coding, however, they still need to write tests for each mobile app features. We now demonstrate how to do this, as follows:

• Take a screen dump using the Appium Studio for Eclipse plugin, and view each element of the UI.
• Use the drag-and-drop feature, and write tests for each element.
• Utilize specific views the plugin provides, e.g., the “Object view”. These show exactly what will happen when the tests for each element run. Click on each test and see it in the object view.
• Write more tests and execute them.
• Check out the test reports by clicking on the “reports” branch on the left pane.

The time required for this step is 2 minutes and 23 seconds.

7. Use Jenkins and IDE to run tests on other machines

Now that we have written and executed test cases, our next objective is to achieve “hands-free” mobile test automation. Such automation will obviously run from a CI environment, therefore, we configure our Jenkins project at this point. Since this testing automation is independent of any particular IDE, we now run these tests using IntelliJ.

This step includes the following tasks:

• Use Jenkins, already installed on the local machine.
• Create a Jenkins project. This will pull the source code from the shared repository and execute a Gradle test.
• Clone the shared repository into the local environment.
• Import the project in the IntelliJ IDE.
• Refresh the project, which will download all the required dependencies.
• Execute the test from the terminal prompt using the appropriate Jenkins commands.
• At this point, an expected build failure occurs. View the information regarding this failure, which will help in resolving the error in the subsequent step.

This step takes 3 minutes.

8. Configuration of SeeTest “cloud properties”

As things stand now, one tester running these tests from his/her workstation will be successful in executing them, however, another tester will not. This is because the first tester used the SeeTest “cloud properties” file, and logged into the cloud. This file contains key information such as URL, project name, user id, and password. The second tester did not use this file.

Resolving this involves the following tasks:

• Peer testers obtain the same properties file used by the first tester.
• Configure the cloud properties file as the first tester did earlier.
• Tests now run successfully without the above build error.
• Open a new browser tab and see the test cases as they run.
• View the test environment with distributed devices.
• Click on a device and see the test execution.
• Toggle back to the Jenkins environment, and run a build. Check whether it fetches all dependencies from the Git repository and installs in the local machine.
• The expected failure occurs since the cloud properties file is missing from the Jenkins environment.
• Set up the cloud properties file in Jenkins.

The required time to complete this step is 3 minutes and 25 seconds.

9. Test the “Invalid login” functionality

After the necessary configurations with regard to the “cloud properties” file, we now test the “invalid login” feature, as follows:

• Launch the app.
• Take a screen dump.
• Drag-and-drop the UI elements, as earlier.
• Introduce an incorrect user-id/password combination in the test.
• Run a TestNG test.
• Check whether the error messages and associated elements are working as expected.

This step needs 3 minutes and 9 seconds.

10. Review the test execution results

We now review the test execution results by navigating to the relevant sections in our IDE screen, as follows:

• Refresh the Eclipse screen, and view the “Package Explorer” tree on the left pane.
• Go to reports.
• Open the reports with a system editor.
• View the test results for each test.

The time required for this step is 1 minute and 19 seconds.

11. Install Jenkins plugins to facilitate CI

We had earlier alluded to the “cloud properties” file and the set-up the testers need to do. The good news is that using Jenkins plugins, we can inject environment variables like those in the “cloud properties” file directly into the build process.

We can do this as follows:

• Toggle back to the Jenkins environment.
• Go to “Manage Jenkins”, and then to “Manage plugins” menu.
• Install a Jenkins plugin called “Environment objective”, from the Jenkins “Manage plugins” menu, to inject environment variables to the build process.
• Enter the above-mentioned cloud properties file in the “Properties file path” field. This takes care of the cloud properties related error encountered earlier.
• Run tests and open up a view of a device.

This step needs 2 minutes and 44 seconds.

12. Configure TestNG “TestListener”

At this point, we can run tests from local machines and also from the Jenkins CI environment. We can see the tests are running on various Android and iOS machines, therefore, we are utilizing the SeeTest device cloud.

Successful testing projects require reporting and analytics, and that’s where we focus now, as follows:

• A robust test reporting and analytics set-up needs “TestListeners”. This project uses TestNG, therefore, we now set up the TestNG “TestListener” in our IDE.
• In the case of TestNG, it’s called “ManagerTestListener”. This makes the test reporting system aware of when the test starts, and when it ends.
• Set up “ManagerTestListener” in Eclipse.

This step needs 2 minutes and 37 seconds.

13. Set up SeeTestReporter

Having configured the TestListener, the next logical step is to configure the SeeTestReporter. This provides robust test reports and analytics, and these help the project team to make sense of the testing project.

The set-up requires the following tasks:

• SeeTestReporter uses “Streams”, which correspond to different testing projects.
• Streams consolidate test reports for every test executed for the corresponding projects.
• Assign names to streams, and include them in the TestListener set-up.
• Publish the build so that reporting works even when running from Jenkins.
• Configure “Streams” so that it can capture every single test we execute. For e.g., use tags of your choice that you would like to see in the reports.
• Specify the test metrics that these “Streams” should report.

This step takes 3 minutes and 21 seconds.

14. Configure a test set-up using many devices

Mobile app testing gets increasingly complex due to the ever-growing device matrix. There is a multitude of platforms, devices, and OS versions. The good news is that we at SeeTest help you in setting up a test environment that uses a large number of devices from our SeeTest device cloud.

This involves the following activities:

• In the IDE, each test has a parameter that specifies the device it will run on.
• Duplicate tests specific to one device, and create tests for other devices by modifying the properties.
• Use various arguments in the tests, such as “lower than” a particular version. This helps in setting up tests on different versions of one OS.
• Assign a name to each of these tests.
• Use the SeeTest parallel execution feature to test simultaneously on many devices.

This step requires 2 minutes and 37 seconds.

15. Add new test cases, and rerun tests

Having set up the CI environment, SeeTest device cloud, and test reporting, we can now add more test cases, and rerun our tests, as follows:

• SeeTestReporter streams will reflect the new test cases, whether they are added in the local environment or the Jenkins environment.
• Install the “Jenkins Version Number” plugin to distinguish between the local environment and Jenkins environment test cases.
• Use an environment variable named “Build”, whose value is the version number in Jenkins.
• Add new test cases, and push the test cases in.
• Rerun tests.
• View their execution in the parallel test environment.
• Review the SeeTestReporter dashboard and check the additional tags. Note the different build numbers for test cases executed from the local and Jenkins environments.

This step takes 4 minutes and 53 seconds.

16. View test metrics in SeeTestReporter

Review the test metrics in SeeTestReporter. The streams report is comprehensive and covers the relevant test metrics. The quality dashboard provides a detailed test status, moreover, it helps in root cause analysis.

This step requires 1 minute and 19 seconds.

SeeTest: Making mobile testing faster

In short, a device matrix growing in complexity, open-source tools requiring integration support, and CI/CD have made mobile test automation more complex. We at SeeTest are simplifying mobile testing for you with our device cloud, test reporter, and plugins.

Jonny Steiner – Content Manager