Skip to main content
Continuous Testing Logo
Last Updated Mar 18, 2021 —

The real way to be successful when testing in production

Testing in production was often viewed as a testing method that created a lot of mess. However, we have learned that not testing in production is a problem as well. Read on for more. 

Continuous Testing

“Close the blast doors!” the Stormtroopers yelled as they chased Han and Chewie through the hallways of the Death Star. The doors began to close — but Han and Chewie slipped through before it was sealed. “Open the blast doors! Open the blast doors!” the Stormtroopers yelled as they realized they were now stuck.   
 
This is what testing in production used to look like. Developers would run their releases through the closing blast doors as the QA team chased them. Metaphorically speaking, I’m sure, though I’d love to see that scenario play out in real life. The bad news for QA in that situation was that the escape of Han and Chewie led to the destruction of the Death Star. So, too, in the world of continuous testing wherein the hopes of keeping their mobile apps up-to-date the developers shirk QA and cause more of a mess. In that case, the suffering passes on to the users — and guess where that extends to? Yup — the business itself.  

It is what made testing in production verboten in most businesses for many years. Here is the flip side: not testing in production is problematic as well.  

Since production environments are built out to a larger degree than test environments and often not up to date, you will miss the real-life scale of testing that you desire. The result of that is less effective testing overall.   

The Death Star of testing  

When we speak of testing in production, we are referring to testing in a live environment; the value is in how the environment mimics the real world. Non-production environment testing is too controlled and lacks authentic traffic and data. I’m not trying to say that testing before production is an issue; of course not — it is invaluable in testing-specific conditions and features of an app. The issue is that such an environment has limits.

I hear you out there and yes, if you could perfectly replicate your production environment in a non-production setting it would remove some of the challenges. However, the requirements are astronomical. You must use the same databases, dependencies, and simulate the same user traffic. If there was only a way to test in that style of environment without the cost..

There is!

Testing in production — and it’s what we are here to talk about today.   

The risk is there  

Of course, there are risks to testing in production. If there weren’t, you would not have people telling you not to do it. They are trying to avoid issues with corrupted data, overloaded systems, and unintended consequences of how the testing impacts other production systems.  

Here are some more of the myriad ways testing in production can go wrong:   

Broken production — As you run tests, they either pass or fail. Shocking in its simplicity, I know. The challenge here is that a failed test in a production environment can take down a production environment and cause app downtime.   

Security — Sometimes, security measures need to be disabled to execute testing in production. I am sure you can see the issues with that.   

Mock data — While more than acceptable in a dedicated testing environment, mock data in production can gum up the works by harming your data integrity. That makes testing in production a potential business risk. Removing junk data is also a process fraught with potential danger as you can lose data in the process.  

System performance — If you are running load testing in production, it can impact performance negatively. The damage from that will impact an entire organization.   

Narrow test windows — If you run soak testing, for example, testing in production might disrupt that leading to flaky tests and false fails.  

Fewer tools — Results are harder to obtain when, for security and performance reasons, you cannot use your favorite tools when testing in production.

So yes, it is risky, but I’m not trying to tell you that testing in production is a bad thing. It works to help expose your mobile apps to real-world scenarios. In that type of system, finding bugs is easier than finding them in your regular testing environment. Even if you execute performance testing in a lab environment, it does not mean that you will see the same results in production.  

What do we do? Why, we keep reading, of course!   

Testing in production the right way

The easiest way to move forward in this method is by developing processes and methods that ensure safe testing with minimal user impact. Let’s look at some examples.  

A layered approach  

Let’s start with a multiple-choice question. How do you test in production?  

  1. With test servers within your production data center.  
  2. Separately running applications atop your production platform.  
  3. Running live tests against your full production deployed code.  
  4. All of the above  

Obviously, the answer should be D. Layering your production testing allows you to test your production environment in different ways. Next, you can minimize your testing impact by matching up test cases. The outcome will be lower test environment maintenance and a minimal effect on your users.   

Light traffic  

When executing performance testing, your entire user base might be impacted. Performance testing has a habit of making servers run slowly and that is something no one has time for, especially in a production environment. In preparation for performance testing, crunch the numbers and see when you have the least number of users. If you want to look deeper, you can look also look at times when the most resource-intensive processes run in the environment.   

Collecting data  

When testing in production, use real traffic data that you can collect like user flows, behavior, and resources. Then use that data to drive your test caseload. With that in hand, you can execute tests in production with the confidence that the behavior being simulated is realistic.   

Monitoring is key  

Always watch the metrics. That is what we say. While watching the numbers as you run a production test, you will know if and how your tests affect the end-user experience. If it does, then you need to shut the tests down.   

Opting in  

One last way to test your app with real users in production is with an “opt-in” feature to preview new releases. This method gives you real users to monitor and collect data. The best part is that if there are any issues, your concern for the end-user experience need not overwhelm you as they have agreed to test the app ahead of time. Once you have the data you can adjust your testing accordingly.  

Testing in production is risky but valuable  

The pressure is on. In 2021 more than ever, a company’s ability to push code to production quickly and deliver error-free experiences is essential. That said, you need to test that functionality to deliver the best possible app.

Sure, you may catch a lot of bugs when testing in a non-production environment, but there are always a few that slip through.   

The best part of testing in production is that it gives you the best view of how your app works under end-user interaction.   

The point is that there are risks for sure but not that we should demonize the concept of testing in production. It pays to understand the risks and build systems that mitigate them. It is all about releasing the best possible app, in the shortest amount of time.   

Learn more about Digital.ai Continuous Testing from this webinar video about why testing in the cloud matters more than ever.

More from the Blog

View more
Sep 30, 2014

Don't Risk Your Company's Reputation: Why Mobile App Testing is a Must

App Management
Mobile apps are powerful tools. In the hands of consumers they can boo ...
Read More
Jul 12, 2017

How Feedback Loops Make It Safe to Deploy to Production

DevOps
Almost every organization is being pressured to respond more quickly t ...
Read More
Mar 04, 2021

Getting key stakeholder buy-in for changes perceived as risky

DevOps
Organizational leaders must recognize that change is vital for the sur ...
Read More
Contact Us