Why Functional QA Engineers need Mobile Performance

In today’s highly competitive business world, organizations proactively monitor trends in customer behavior and quickly adapt to changing customer needs. The advent of smartphones has changed the way businesses think and operate. With the increase in smartphones, organizations must focus on mobile-first solutions. As such, mobile performance becomes a key requirement while developing applications.

UI design and UI performance are key aspects that make or break a product. The time for rendering content on the screen and the time for customers to interact with the content are the two important metrics to measure the UI performance. However, developers focus more on code logic instead of improving UI experience. While code is important, UI performance is equally important. To improve UI experience, you should focus on network performance, API performance, and app performance.

Here are a few key items every developer and QA engineer should ponder when attempting to deliver increased mobile performance.

1) Better Understand Networking

To better understand networking, it is important to look at the Open Systems Interconnection (OSI) networking model first. An OSI model is a standard for facilitating communications between different computing devices.

1. Physical Layer: It involves all the physical devices of a network such as cables, switches, routers, Ethernet, etc.
2. Data Link Layer: It defines the format of data.
3. Network Layer: It defines the physical path for the data in the network.
4. Transport Layer: This is where data transmits using TCP/IP communication protocols.
5. Session Layer: In this layer is where you create and manage sessions and ports.
6. Presentation Layer: Here is where data is presented in a usable format.
7. Application Layer: This is a user interface layer wherein users and the network interact with applications.

In a real-time scenario, browsers and mobile apps operate in the application layer, presentation layer, and session layer using HTTP. The TCP protocol operates in the transport layer while the IP protocol operates in the network layer. The Wi-Fi runs in the physical and data link layer.

Network Aspects

Here are 3 important elements to consider when checking data transmission in a network.

1. Latency: It is the time required to transmit a data packet from one node to another and receive a response on the network.
2. Throughput: It is the amount of data transmitted from a source in a specified time, measured in bits per second or data packets per second.
3. Packet Loss: The number of packets lost when the host sends 10 data packets.

Window Sizing in TCP Model

Window sizing is used in the TCP model to better control network congestion and improve data transmission efficiency. It controls the amount of unacknowledged data the sender can send before receiving an acknowledgment.

When there is a packet loss, the protocol will reduce the window size to half and starts slowly to reach the original window size.

So, packet loss significantly impacts the throughput.

Connection Breakdown

Here are the tasks that run when a connection is established:

1. DNS Lookup: When the user types a website name in the browser address bar, the computer will send a DNS request to the Internet Service Provider and receive the IP address from the ISP.
2. SSL Handshake: To protect data privacy and integrity, the client validates the authenticity of the server after which they establish a secure connection for data communication between the client and the server. We call this the SSL Handshake.
3. Send Request: When establishing a secure connection, the client sends a request to the server.
4. Wait for response: After sending the request, the client waits for the information to be presented by the server.
5. Content for download: When the server presents the requested information, the user can download the required content.

To analyze the mobile performance, calculating the time for each task is important.

 

1. DNS Time: when the user tries to open a CNN website, the user types the URL in the browser address bar. The DNS time is the time taken for the client machine to contact the DNS server and receive the IP address.
2. Connect Time: The client uses SYN to initiate a connection with the CNN webserver. The web server responds with SYN/ACK and the client acknowledges with ACK. The total time for these tasks is the connect time.
3. SSL Time: It is the time taken for SSL Handshake between the client machine and the webserver
4. Wait time: It is the time the client machine waits to receive information from the webserver.
5. Receive time: It is the time taken by the webserver to serve the required information to the client.

In a real scenario, a proxy server comes into the picture. As such, the entire process gets repeated. The client contacts the web server via the proxy server. So, you should consider this extra time as well.

Waterfall view with HAR

HTTP Archive Format (HAR) is a useful option that lets you log and track the browser’s interactions with websites. Using HAR files, you can analyze the performance issues as well as page rendering issues.

• To generate a HAR file in Chrome, open the Chrome browser, and open the Experitest website.
• Go to Chrome Settings -> More tools
• Choose Dev tools – > Network
• Select the preserve log checkbox and click on the recording button
• After the recording is done, right-click and save it as HAR with content

A HAR file looks like this:

It clearly visualizes the loading process of the Experitest website. So, you can check how much time it takes for different assets of the website to load onto the screen.

2) Performance – User perspective

From a user perspective, mobile performance depends on 3 important aspects.

• Duration: If a website takes 3 or more seconds to load, it will drive away 50% of customers.
• Resource Hogging: Customers don’t like apps that consume more resources such as CPU, Memory, battery, and data. So, customers more likely remove such apps.
• App Store: Customers look at the app store ratings. When it comes to performance 5 stars is what drives more customers to the app.

Focusing on Transactions

A transaction is a UI event that sends a request to the server and gets a response or leads to another event. So, from a user perspective, the time taken for a transaction is important.

Here are key metrics for a Transaction.

• Duration: The time taken for a transaction should be as short as possible.
• CPU and Battery: Keeping CPU and battery consumption as low as possible increases customer experience.
• API Call Count: It is important to keep API call counts minimal and running in parallel.
• Download Data: Minimal downloading of images and videos via REST APIs improves transaction performance.
• Server response time: The time taken for the server to respond to client requests is important. The measuring unit should be the time taken to receive the first bit.

Speed Index

When you open a website, the webserver presents the site with all the site elements. The process in which the web server presents these elements significantly impacts user experience. In other words, page load time affects the mobile performance.

For instance, website A takes 5 seconds to load but presents all elements at once. Another website B takes 5 secs to load but loads 50% of page elements in the first 2 secs and then progressively loads other elements. From a user perspective, Website B provides a good experience.

When you look at the page load time, both sites stand at the same level after 1 sec but site 2 jumps to 90% in the next sec.

3)How to improve mobile performance?

Now that you understand the networking concepts, you should focus on improving performance testing. Generally, organizations build a mobile app, build server, deploy the server, upload the mobile app, and run tests. The performance testing is done at the end of the development pipeline. Sometimes, it is the user who identifies performance issues. The best way is to move performance testing into the CI/CD pipeline as early as possible.

The best way to improve user experience is to incorporate performance testing into your functional testing segment. For doing so, gather transaction data as a part of functional testing.

For instance, create a Transactions Information database that will store all transaction information. When you run manual and automated tests, the transaction data gets stored in this database. Then, run Transaction analytics on this data to analyze and debug issues.

 

Create a baseline for a transaction time by checking the current or earlier versions. For instance, a login transaction normally takes 4 seconds. While you cannot tell it is good or bad, you can surely say it is not good when this time increases to 6-8 seconds. When this happens, you can analyze the data and see which transaction is causing the delay. You can use a HAR visualizer to analyze how assets are loading and identify the regression issue.

Here is a demo that shows how to identify performance issues.

i) Open the mobile cloud testing platform. eg; Experitest cloud

Here, you can see a list of mobile devices from across the globe. So, you can run mobile testing on various devices. Now open a device for the test.

Here, you can see a transaction recorder that will record the transaction done on the remote mobile device. when you press the red button, the system starts recording the event.

ii) Now, open a webpage on the mobile device and run the recorder. eg: Netflix. When it comes to connection type, it is recommended to choose 3G network instead of Wi-Fi as your customers will be using 3G and 4G. So, running your tests simulating customer network conditions would be a good idea.

It is important to note that the transaction event starts when you press the button. First, type the website name and then press the record button and click go.

It is interesting to note the speed index of Netflix. Most of the page elements load before the site is fully opened. So, it gives a good user experience.

iii) Now, open the recording.

Here, you can run the recording to see the entire transaction process. You can check the details of the site on the top left such as the website name, the platform, device, and the browser. On the right side, you can see the metrics such as duration, CPU, memory battery usage, etc.

When you open the HAR view, you can see the waterfall view that displays how each asset has loaded. You can also click on each transaction and analyze it.

iv) Now, let us check an example of a Login transaction. Open another device from the cloud testing platform and install an application first.

v) Now, open the application and sign in to the application. Note that the event starts when you hit the login button and not when you type the username and password. So,type the username and password first and then press the recorder button and then click the login button so that the recording of the transaction begins.

vi) When the page is fully rendered, stop the transaction and save the recording.

When you open the recording, you can see transaction details.

 

However, you cannot analyze app performance with a single login instance. You have run tens and hundreds of logging transactions and compare them and analyze the changes in time durations to identify regression issues. The best way to do that is to use the existing functional test.

Here is an example of functional testing using Appium and IntelliJ IDE.

• Install the new version of the application from the IntelliJ IDE.
• Log in to the application and set the username and password.

• Start the transaction.
• Click the login button and wait for the progress of the transaction.

• Once the transaction progress disappears, end the login transaction. Now, you can run the tests multiple times and on multiple devices.
• Now, open Jenkins CI/CD tool

For sophisticated visualization, use Blue Ocean, a plugin for Jenkins.

In the Jenkins CI/CD pipeline, you can see the build, deploy, and run test segments.

• Now, start the test execution.

It will also download resources, compile them, and run the tests.

From the dashboard, you can see that all the tests are running in parallel.

Choose analytics to compare the data of multiple transactions. you can choose the transaction type, device type, app version, etc. Here, we selected the login transaction, Android device, and the latest 3 versions.

Here is the result:

If you see the results, 100s of transactions of app version 1 took an average of 2 secs while the transactions of version 1.1 took 4 seconds and that of version 1.2 took 6-7 seconds. When you consider the test time, the change in time is not a big thing but when you consider the average of multiple transactions, it is certain that there is a regression. So, you can consider the old version of the app as a baseline and check the change in time. You can also check the memory, CPU, and another resource usage as well. Once you identify regression issues, you can open that transaction in a HAR view and identify how each element is processed.

By just adding two lines in the code, you can run these performance tests in the existing functional tests and get this data for analysis.

• startPerformanceTransaction()
• endPerformanceTransaction()

This command will start a performance transaction to collect data on CPU, Battery, Duration, Network Traffic, etc. You have to specify the network virtualization profile as a parameter. When no parameter is specified, the 3G-average profile becomes a default option.

As a best practice, perform all measurements in bad network conditions. So, choose the 3G average network as a default network condition.

Summary

UI performance has become a game-changer for mobile apps in recent times. As such, businesses should focus more on UI Performance Testing to improve user experience and thereby retain existing customers while acquiring new sales. UI performance depends on the existing functional testing environment. Better understanding the network performance from a user perspective is important. The key to mobile performance is to move mobile performance testing into the CI/CD pipeline. So, add performance testing to your existing CI/CD pipeline to identify issues at the very beginning and deliver a rich user experience to your customers.

Guy Arieli – CTO