Load testing deployments

It’s super important to do as much automated testing as possible. To be more precise as much should be done that guarantees error and regression freeness of the system. Doing unit tests, integration test and system integration tests are essential.

Since Thriot may have to process and survive high load it’s essential to do load testing of the system. The load testing in case of Thriot includes the following scenarios:

  1. Automatically create many-many devices that will be used in the later test cases
  2. Using the occasionally connected client (aka REST services) record telemetry data from many devices in the same time
  3. Using the occasionally connected client send messages to other devices
  4. Using the occasionally connected client receive these messages using QoS 0 level of reliability (messages are received at most once – receive and forget)
  5. Using the occasionally connected client receive these messages using QoS 1 level of reliability (messages are received at least once – receive and commit)
  6. Do these test cases (except for the first one) using the persistently connected client (websockets)

Preparing for load testing

Along with the .NET Client there is a Thriot.Loadtester projects. Compile this project and navigate to the bin\Debug folder to be able to execute the test cases.

Here you will see the following files:

image1

Explaining the roles of the files above:

  • env.config : the settings in this environment configuration file will be used at test run
  • env.*.config : prepared configuration files for different environment
  • exec.ps1: Powershell script for executing test cases
  • generate.ps1: Powershell script for generating devices. This will also register a new user so that it is only usable when there is no need for email activation
  • generateforuser.ps1: Powershell script with similar functionality to generate.ps1 however it expects the email and password of an already existing user.
  • Thriot.Client.Dotnet.*: Thriot .NET Client
  • Thriot.Loadtester.*: Load testing executable

(generate*.txt files always contain devicesids and network keys that are used for testing generated by the generate*.ps1 scripts)

The tests below will run against the production azure environment so that the env.config file’s content equals to the env.prodazure.config file. It’s important to verify the content of env.config before running test. In case of discrepancy in most cases one of the prepared env.*.*configs has to be copied to env.config.

Generating devices

Execute the following command to generate 1000 devices for the specified user authenticated by email and password.

.\generateforuser.ps1 -cnt 1000 -email ****** -password ******

image2

As a result devices.txt will contain the parameters of the devices generated.

The generated devices can be verified on the website (central.thriot.io):

image3

Recording Telemetry data

The following Powershell script is to be executed to do load testing:

.\exec.ps1 -devices devices.txt -operation ocrecord -from 0 -to 1000 -batch 30 -sleep 2000

Explanation of parameters:

  • -devices devices.txt: use the devices found in devices.txt for load testing
  • -operation ocrecord: record operation with the occasionally connected client
  • -from 0 –to 1000: using all devices from 0 inclusive to 1000 exclusive
  • -batch 30: testing is done in multiple processes. One process will simulate 30 devices
  • -sleep 2000: sleep 2 seconds between each recording / device

The following screenshot will show the processes simulating 30 devices each:

image4

The following screenshot shows the CPU/Memory/Network utilization of the Azure VM (small instance – 1 CPU Core, 1.7 GB memory):

image5

Between real world circumstances sensors won’t record telemetry data in every 2 seconds which means that a single Small Instance VM running on Azure is able to handle many thousand devices.

 

To record telemetry data using the persistent connections (websockets) issue the following command:

.\exec.ps1 -devices devices.txt -operation precord -from 0 -to 1000 -batch 30 -sleep 2000

The only different is that the operation is precord instead of ocrecord.

QoS 0-level messaging test

We will use the occasionally connected client (REST API) for this test.

To execute receive and forget level (aka QoS 0 –level) tests from the receiver’s perspective execute the following scripts:

.\exec.ps1 -devices devices.txt -operation ocsendto -from 900 -to 1000 -batch 20 -sleep 500 -extra 900

.\exec.ps1 -devices devices.txt -operation ocrecvforget -from 0 -to 900 -batch 30 -sleep 1000

The first command sends data is using the occasionally connected client (ocsendto, REST service) from devices 900 (incl) to 1000 (excl). Each process is simulating 20 devices that are sleeping 500 milliseconds between each sending. The –extra parameter means that devices ranging from 0 to 900 are targeted with messages.

The second command uses the receive and forget method using the occasionally connected client for devices 0 to 900 in 30- batches. Each device is sleeping 1 second between fetching the next message (if there is).

The following screenshot shows the process of the simulation:

image6

If we would have used psendto and preceiveforget command the persistently connected client would have been used for testing.

QoS 1-level messaging test

We will use the persistently connected client (websockets) for this test.

To execute receive and commit level (aka QoS 1 –level) from the receiver’s perspective execute the following scripts:

.\exec.ps1 -devices devices.txt -operation psendto -from 900 -to 1000 -batch 20 -sleep 500 -extra 900

.\exec.ps1 -devices devices.txt -operation precvcommit -from 0 -to 900 -batch 30 -sleep 1000

The first command sends data using the persistently connected client’s websocket service (psendto) from devices 900 (incl) to 1000 (excl). Each process is simulating 20 devices that are sleeping 500 milliseconds between each sending. The –extra parameter means that devices ranging from 0 to 900 are targeted with messages.

The second command uses the receive and commit method using the persistently connected client for devices 0 to 900 in 30-batches. Each device is sleeping 1 second between fetching the next message (if there is).

The following screenshot shows the process of the simulation:

image7

If we would have used ocsendto and ocrecvcommit command the occasionally connected client would have been used for testing.

 

Advertisements