Telemetry data queue

This page refers to the latest development code on GitHub. https://github.com/kpocza/thriot/

To provide more scalable, fast and reliable architecture Thriot supports queueing of telemetry data. Queueing is still an optional feature since it is needed in environments where scalability is really important, eg. cloud services, high load IoT applications. For smart homes or any environment where at most just tens of telemetry data entries are recorded in a second, queueing may seem overkill.

Queueing is supported by the telemetry plugins. Currently the following queueing solutions are supported:

  • Azure Queues
  • Microsoft SQL based queue implemented by a table and SPs (this is considered as antipattern)
  • PostgreSQL based queue implemented by a table and SPs (this is considered as antipattern)

Future plans are:

  • Azure Servicebus
  • Azure EventHub
  • RabbitMQ
  • Apache Kafka
  • etc.

To enable queueing the following steps are to be followed:

  1. Execute build with queueing enabled
  2. Setup the appropriate queue configuration in the telemetryqueue.json config files

The telemetryqueue.json file is put by the build script  into the conventional config folder of the following componenets:

  • Platform API
  • Platform Webservice component (long running process – Windows Service)
  • Telemetry Queue service (long running process – Windows Service)

This configuration file contains the plugin class to be loaded and the queue configuration settings.

The telemetryqueue.json is to be prepared by the build.ps1 file or by the buildconfigure.ps1 to setup connection string and parameters automatically.

For further details please refer to http://portal.thriot.io/package-autoconfiguration/

Installing Telemetry Queue

This section described how to install the currently supported queues to the system.

In case of Windows Azure Queue Storage

Development storage

  1. Open the Visual Studio in Server Explorer
  2. Navigate to Azure/Storage/Development/Queues (ensure that the emulator is running)
  3. Create a new queue named telemetry

This is the default settings that should work with the default configuration.

Hosted Azure solution

  1. Do the same steps as for the development storage but with your live azure subscription OR
  2. Create the queue using the Azure Portal

Grab the connection string of the storage account and the queue’s name just created and configure the system described above.

In case of Microsoft SQL-based queue

Development/Test environment

  1. Create a new database called ThriotTelemetryQueue on the same SQL server instance as you used for the Management, Telemetry and Messaging dbs
  2. Execute the following script on the ThriotTelemetryQueue db from the original sourcecode: Service\Platform\Scripts\Sql\CreateTelemetryQueue.sql
  3. The telemetry queue service should have permission on this db to execute SPs
  4. The queue connections strings must be setup according to the description above.

For further details executing steps like these for SQL please refer to the base system installation procedure description pages.

Production environment

  1. Assuming that you’ve created a built with SQL-based queueing enabled
  2. Create a new database called ThriotTelemetryQueue on you SQL server instance
  3. Execute the following script on the ThriotTelemetryQueue db from the build-script created package: install\storage\queue\CreateTelemetryQueue.sql
  4. Setting up permissions
    1. If you are using Windows Integrated authentication ensure that the running user of Platfom WebAPI, the WebsocketService and the TelemetryQueueService have permission to execute SPs
    2. If you are using SQL authentication ensure that the thriottelemetryqueue user (by default) has permission to execute SPs
  5. The queue connections strings must be setup according to the description above.

For further details executing steps like these for SQL please refer to the base system installation procedure description pages.

In case of PostgreSQL-based queue

Development/Test environment

  1. Create a new database called ThriotTelemetryQueue on the same PostgreSQL server instance as you used for the Management, Telemetry and Messaging dbs
  2. Create a new user called thriottelemetryqueue with password thriottelemetryqueue (this is the default)
  3. Create a new database called ThriotTelemetryQueue and set the thriottelemetryqueue user as owner of this database
  4. Login with the thriottelemetryqueue user and select the ThriotTelemetryQueue database and execute the following script on the ThriotTelemetryQueue db from the original sourcecode: Service\Platform\Scripts\PgSql\CreateTelemetryQueue.sql
  5. The queue connections strings must be setup according to the description above.

For further details executing steps like these for PostgreSQL please refer to the base system installation procedure description pages.

Production environment

  1. Create a new database called ThriotTelemetryQueue on your PostgreSQL server instance
  2. Create a new user called thriottelemetryqueue with your chosen strong password
  3. Create a new database called ThriotTelemetryQueue and set the thriottelemetryqueue user as owner of this database
  4. Login with the thriottelemetryqueue user and select the ThriotTelemetryQueue database and execute the following script on the ThriotTelemetryQueue db from the build-script created package: install\storage\queue\CreateTelemetryQueue.sql
  5. The queue connections strings must be setup according to the description above.

For further details executing steps like these for PostgreSQL please refer to the base system installation procedure description pages.

Installing Telemetry queue service

There is one extra component called Telemetry queue service that is responsible for polling/being notified by new queue messages and recording the telemetry messages into the telemetry database.

Development/Test environment

As the Thriot Websocketservice the Telemetry queue service can be also started as a standalone application. You don’t have to do any other than starting Thriot.Platform.TelemetryQueueService.exe from the command line. It should open a new console window and print Started.

Please ensure that the Telemetry Queue Service has read permission for the management database and write permission to the telemetry queue database and has access to the telemetry queue.

Production environment

In production environment the Telemetry queue service should be installed as a windows service (or stated by mono-service on Linux).

Execute the following command as administrator from the service folder:

C:\Windows\Microsoft.NET\Framework\v4.0.30319\InstallUtil.exe Thriot.Platform.TelemetryQueueService.exe

The expected result is the following:

installtqs

Please ensure that the Telemetry Queue Service (NT AUTHORITY\NETWORK SERVICE user by defeault) has read permission for the management database and write permission to the telemetry queue database and has access to the telemetry queue.

The service starts automatically, but you can use the following command to start the service for the first time:

net start ThriotTelemetryQueueService

The system must indicate the following: The Thriot Telemetry Queue Service service was started successfully.

Advertisements