4 Mobile Software Testing Use Cases Where Device Clouds Don’t Work

A device cloud, which provides on-demand access to a variety of different types of mobile devices, may seem like the most efficient and effective way to run mobile tests – and it is, in many scenarios.

But depending on which types of mobile tests you need to run or how you need to run them, device clouds aren’t always the best solution. If you blindly turn to device clouds by default to meet your test infrastructure needs for all use cases, you could end up shooting yourself in the foot.

To prove the point, I’d like to walk through four mobile testing use cases where you either fundamentally can’t use a device cloud, or where the disadvantages of cloud-based mobile testing outweigh any benefits of the strategy.

See also: MLOps Execution Engines: Jenkins vs Tekton

What is a device cloud?

Before diving into a discussion of mobile testing scenarios where device clouds don’t make sense, let me make clear what I mean when by “device cloud.”

I’m referring to a cloud-based mobile testing environment where developers and QA engineers can run software tests on remote devices. In other words, a device cloud is a “cloud” of mobile devices that you can access on demand for executing tests. Device clouds serve as an alternative to running tests on mobile devices that you have to set up locally and manage yourself.

Device clouds have become a popular solution for mobile testing because they offer several benefits compared ton testing with local devices:

• Testers can easily test applications across hundreds or thousands of different mobile devices without having to purchase and set up each device themselves.
• You typically pay only for the time during which your tests run, making device clouds a more cost-effective solution in many cases than testing on devices that you have to acquire locally.
• Because most device cloud providers add new types of mobile devices to their clouds on a regular basis, testing teams always have access to up-to-date mobile hardware.

A variety of vendors (including Kobiton, where I work) offer device clouds that deliver these benefits.

Why device clouds don’t always work for mobile software testing

Device clouds are great for use cases where teams need to test mobile apps across many mobile devices and don’t face special requirements. However, depending on your needs and priorities, device clouds may not make sense for your testing routine.

Here are four scenarios where you can’t or shouldn’t use a device cloud for mobile testing.

1. Testing for infrastructure disruptions

Devices hosted in a device cloud operate under best-case conditions. They are fully charged, professionally managed devices that are connected to very stable, high-speed networks.

That’s good in the respect that it means the devices can run tests reliably. But it’s bad for teams who want to understand how their mobile software performs under less than ideal circumstances. If you want to know how Bluetooth interference or a spotty cellular connection could impact mobile app performance, for example, you won’t be able to simulate those conditions using a device cloud.

You can simulate these conditions by deploying devices locally in what QA professionals call an isolation chamber, where testers can introduce “chaos” like weak network connections to evaluate how software responds.

2. Testing mobile apps with strict security requirements

Testing software on a device cloud requires you to upload your applications, along with any data that you need to feed into them for testing purposes, onto mobile devices owned by a third party.

In most cases, this isn’t a big deal. Device cloud providers secure their infrastructure and network connections to protect sensitive data.

But if you’re dealing with exceptionally rigid security requirements – as you might in highly regulated industries, or in situations where you’re testing apps that manage very sensitive data – a device cloud might create too many security risks. You’re better off in that case testing on-prem, where you have full control over data and software.

3. Minimizing test execution time

Mobile software tests that take place on remote devices hosted in cloud almost always take longer to complete than tests that run locally. The main reason why is that cloud-based tests require data to move over the Internet, which is slower than local network connections.

To be clear, we’re talking here about delays of perhaps only 200 or 300 milliseconds. In most cases, that’s not a big deal. But if you require ultra-fast test execution, which is beneficial in scenarios where you need to run a large number of tests and you don’t want them to delay your software delivery pipeline, cloud-based testing might slow you down too much.

4. Testing on obscure devices

Major device clouds cover most of today’s most popular types of mobile devices, which makes them a great solution if you need to test on devices that are widely used by consumers.

But if you need to ensure that your apps run on more obscure devices – such as specialized IoT devices that you might use in your company’s facilities – you’re less likely to find them in a device cloud. You’ll need to acquire the specialized hardware and run tests on it locally.

Conclusion

Device clouds are a great way to meet garden-variety mobile testing requirements. But not every mobile testing use case is simple enough to support using a device load. Sometimes, tests require a level of security, performance and control that only local mobile test environments can deliver.