IOT Testing Challenges and Approach

The Internet of Things or IoT has been around for more than two decades, but its foray into the mainstream business has gained prominence in the last decade. IoT refers to a network of interconnected physical devices collecting and sharing data through a network. Organisations around the world are using IoT to operate efficiently, make informed decisions with real-time data, and provide enhanced customer experience. As per Statista, iPriopertyManagement.com, in 2021, the number of IoT devices is projected to surpass 75 billion by the end of 2025. Testing these devices will pose many challenges for IoT testers in the coming years.

The main challenge which is faced in the testing of IoT is due to its vastness, integrating multiple devices, machines and sensors, like pulse-oximeter, electrocardiogram, a person with a heart monitor implant, a farm animal with a biochip transponder, an automobile that has built-in sensors to alert the driver when tire pressure is low, etc. These sensors and machines communicate in multiple protocols with an edge computing/gateway device which acts as a pipeline to collect the data from different sources. They send the collected information to the cloud, ensuring there is no loss of data, and the data is read in multiple ways by the user through phone/web interfaces etc.

In this article, we are going to discuss some of the common challenges in IoT that testers often encounter. We will also throw light on how to address them by offering an IoT testing approach before guiding you on how to select the right tools for IoT testing.

What are different IoT testing Challenges?

1.Devices/ Sensors: The data generated from a device/sensor doesn’t have intelligence on its own. The true challenge is to test data from ‘n’ number of devices that are sent at once. It can be wired/wireless, with a protocol that might differ as per business requirements. Hence, the verification of the data validation needs to be done in all these scenarios needs to be done to ensure only valid data flows in from the source. In the event of invalid data, there needs to be clarification of errors related to the same.

2. Edge computing/Gateway devices: Though this is a black box, the gateway devices are the heart of the IoT system. This has a complex configuration of the system, the schedulers, data processing and the business application firmware. The Gateway device needs to be verified to ensure that the device never goes offline. Even in case it goes offline, the reasons for failure need to be clear, and the data should not be lost but stored and then sent to the cloud as and when the device is On. The Gateway load needs to be verified to ensure that huge data transmission is not going to kill the memory of the device, but there is always a smooth transmission of data.

3.Data processing/Cloud: There are several cloud data processors available in the market. The verification of the cloud starts by ensuring that the configuration between the gateway device and the cloud is the  same and secured. Voluminous data can have the same or multiple protocol communication methods. The verification of valid data points is key so that no data loss becomes crucial in this layer. Failure scenarios are well possible in case of data loss, and so the cloud also needs to be tested.

4. User Interface (UI): There are again many user interfaces that can vary from smart devices (watches, health monitors etc.), mobile applications, web interfaces, desktop interfaces, Kiosk, TV to anything where a human can read the information. With businesses trying to provide a seamless omni-channel experience, the burden on testers is huge as there are endless software and hardware devices. Testers need to ensure that the application works seamlessly on the required software. For those that are not supported, proper information needs to be given.

5. Security: Data security is pivotal. With numerous devices connected through various networks, testing and fixing vulnerabilities that can pose a security risk for business is crucial. Let’s discuss the security aspect by considering the following example.

You might be interested in : IOT Testing Approach on devices

In a mall, a fire sensor was replaced as it was not working. The sensor which was replaced is of the same brand but a different version type.  In such a situation, testers need to ensure that:

• Ensure the device cannot be configured during replacement.
• While removing the previous device, there should be a proper uninstallation process, or the past data should be recorded.
• While configuring the new device, appropriate information stating that the device does not support should be provided in the respective UI.
• Ensure the brand device of the same version is replaced and there is proper data transmission.
• From a UI perspective, only installers who have access can do the replacement
• Alarm messages are sent only to the configured users.

IoT Testing approaches to follow for perfectly addressing the challenges

From the above challenges, you must have understood that test scenarios are huge for functionality testing. But testers don’t always have time to test all the scenarios in an exploratory model, which may lead to delivery delays. So, it is important for testers to intelligently plan the test approach according to the product/project and minimize the time of test, and finish testing on time.

The following table gives an idea of what testing types will be applicable for the IoT layers.

The best practice to plan testing is to have a checklist or an acceptance criterion clearly defined for the whole product. This helps to set a clear boundary for the testing team and to avoid testing other scenarios. 

The following are a few samples questionaries that must be considered here:

1. 1. IoT testing services involves myriad end-users, and they are important. So, we need to address questions surrounding end users, business users, and actors (includes support/maintenance/installers). These questions include:
   1. a. Who are the end-users (includes specification of age group and family type)? It is good to start testing with a clear persona detail in hand.
   1. b. Who are the Actors (includes trained installers and field operation men)?
   1. c. Who are the business users (includes financial team, CEO, and OEMs)?

2. For Software and Hardware, it is good to start testing once the requirements are clear.  Points that need to be considered here are:

• If it is a mobile application, ensure to get clarity on the mobile OS types, resolution, and supported software.b. For the hardware, ensure that the manufacturer’s detailed version is known.

c. As for the cloud, ensure that there will always be upgrades, and those upgrades are captured in the system. The lower version support also needs to be known.

• 3. For Industrial Internet of Things (IIOT), it is the field which matters, and the final product should work in the field. So, it is always mandatory to understand all the environmental facts with respect to climatic conditions and the specifications and behavior of end devices, too, with respect to the environment. It is advisable to have compliance certification as per the geographical location.

How to choose the appropriate tools for IoT Testing?

Several open as well as paid tools are available in the market. Choosing the right tool depends on the design strategy of the IoT product, Saas, Paas or IaaS.

It is also difficult to have a single tool for a complete product and to design an automation framework for a product as a whole. So, it is good to split up testing according to the layers and use separate testing methods and automation tools.

However, among all the integration, testing will play a major role. While there are CI/CD pipeline models, integration testing brings in lots of possibilities and metadata and meaningful testing to the product. The Northbound of IoT (Cloud-> communication network ->UI) can be chosen to be automated separately. Whereas testing gateway and devices require apt tools according to the product/domain and architecture. As the investment in the hardware is quite high, choosing virtual simulators and emulators can prove helpful in generating the data points.

Another important aspect of IoT testing is pilot testing. In this, the product runs directly on the field post the verification runs are executed in the sophisticated lab environment with virtual simulations. Once the product is installed in the field, testing with the real data is very important. Only when this is done the reliability of the product is assured. In order to understand and collect the data, pilot testing ranging from 1 month to 6 months is advisable.

Conclusion

Even for a single IoT product, several components have to be tested. Hence it is always good to initiate testing in the design phase itself. Though exploratory testing takes a longer time compared to other testing methods, it is important in IoT as there are several users, and the same data is read by various users.

Remember that testing with requirements is never sufficient for the IoT, but exploring data and the reports and insights from the data source actually brings a great useful product in the end.