Loadtest for FIWARE Components

January 16, 2023 · View on GitHub

FIWARE Core

Gatling load tests to run against FIWARE Components that implement the NGSI-LD Api. The repository also includes test-reports for various setups of different FIWARE Components.

Reports
How to run
- Maven
- Helm
- Testresults
Scenarios
- Broker without security
- Notification latency
License

Reports

Reports of executed tests and the used config can be found in the testReports-folder. You can find all the information about how to setup the tested instances and how to rerun those tests there.

How to run

Maven

configure your test run inside the src/test/resources/test.conf
run all scenarios using

mvn install gatling:test

or single scenarios via

mvn install gatling:test -Dgatling.simulationClass=<CLASSNAME>

(e.g. CLASSNAME=simulations.nosec.v2.BatchUpdateSimulation )

In scenarios with a large number of entities, multiple machines should be used. For such cases, the helm chart can be used. The chart will run the tests as kubernetes jobs an distribute them through the cluster according to your configuration. Every job will upload its result to a central ftp server. The uploaded results will be aggregated by the ftp's sidecar container. You can view them in a browser via:

kubectl proxy --port <YOUR_PREFERD_LOCAL_PORT>
http://localhost:<YOUR_PREFERD_LOCAL_PORT>/api/v1/namespaces/{{ .Release.Namespace }}/services/{{ template "orion-loadtest.fullname" . }}:8080/proxy/

See the chart docu for all configuration options.

Testresults

After running your tests locally, the results are published in the standard Gatling Report format and can be found under target/gatling/results

If you are looking for reports of already executed tests see testReports.

Scenarios

Broker without security

The following scenarios will test the broker without any security components installed(e.g. no api-umbrella, no keyrock etc.). Therefore the involved components are:

Orion-LD
MongoDB
optionally: an Ingress Controller or load balancer in front of the broker.

Single value updates for entities

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numUpdates	How many updates should be executed for each attribute.	100
updateDelay	Delay between attribute updates in seconds.	1

Batch updates for entities

A number of entites(see store-entity for structure of the entities) will be created in batches, then update 2 attributes (temprature and humidity) also in batches. The number of parallel calls can be calculated by dividing the number of entities to be simulated through the batchsize.

The creation is done via POST /entityOperations/create, the updates via POST /entityOperations/update.

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numUpdates	How many updates should be executed for each attribute.	100
updateDelay	Delay between attribute updates in seconds.	1
batchSize	Number of entities to be updated in each batch	10

Single value updates for entities with active subscriptions.

A number of entities(see store-entity for structure of the entities) will be created, then update 2 attributes(temprature and humidity) via POST /entities/<ID>/attrs with a delay between each of the updates. Each entity will have an active individual subscription(see subscription), that notifies a configurable http endpoint. To run this test, you need to provide an endpoint, that responds with 2xx to the notifications. We can recommend the labstack echo-server that is also used in the test helm-charts.

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numUpdates	How many updates should be executed for each attribute.	100
updateDelay	Delay between attribute updates in seconds.	1
notificationServerUrl	URL to be notified by the broker	http://echo-server

Single entity get.

A number of entities(see store-entity for structure of the entities) will be created, then they will be retrieved via GET /entities/ in parallel.

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numGets	How many gets should be executed.	100

Query entities by an attribute.

A number of entities will be created(see query entities for there structure), then a subset of them will be retrieved via a query matching to a specific attribute(producer). The queries are done via GET /entities?q=("producer"=="<PRODUCER_NAME>")

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numParallelQueries	Number of queries to be executed in parallel.	100
numQueries	How often should the queries be repeated.	100

Query entities by type.

A number of entities will be created(see query entities for there structure), then a subset of them will be retrieved via a query matching to a specific type(owner). The queries are done via GET /entities?type=https://uri.fiware.org/ns/data-models%23owner)

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numParallelQueries	Number of queries to be executed in parallel.	100
numQueries	How often should the queries be repeated.	100

Query entities by type and attribute.

A number of entities will be created(see query entities for there structure), then a subset of them will be retrieved via a query matching to some attributes and types. The query is: GET /entities?type="store"&q=(("open"=="<RANDOM_BOOLEAN>")|("owner"=="<RANDOM_OWNER"))&options=keyValues"

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numParallelQueries	Number of queries to be executed in parallel.	100
numQueries	How often should the queries be repeated.	100

Notification latency

In order to measure the latency of a notification, a dedicated set of tests is available. We define the latency of a notification as following:

tCreated :  timestamp the value was sent to NGSI-API by the client
tReceived : timestamp the notification was received by the subscriber

latency = tReceived - tCreated

In order to measure that latency with as few dependencies to the subscriber, we are deploying a setup that overcommits resources on the subscriber. For real world scenarios, the subscriber(f.e. QuantumLeap) needs to be setup in a way that it is able to handle the amount of notifications it should receive.

Test setup

To get creation time of the update, we use an entity attribute to hold the timestamp. The entity for testing the latency looks as following:

{
    "type":"timed-entity", "id":"urn:ngsi-ld:timed-entity:<ID>",
    "senttime": {
        "type": "Property",
        "value":  System.currentTimeMillis()
    },
    "humidity": {
        "type": "Property",
        "value": <RANDOM_FLOAT>
    },
    "@context": "https://fiware.github.io/data-models/context.jsonld"
 }

As a subscriber Telegraf is deployed to the setup. Telegraf is built to handle high amounts of metrics and has the ability to transform and persist incoming metrics to InfluxDB. In addition it provides a plugin to set timestamps on received messages. With that setup, all data required for calculating the latency is present inside InfluxDB and can easily be queried, visualized and analyzed with Grafana. A dashboard for analyzing the results is provided by the chart. By using Telegraf we also introduce a decoupling-layer between the database(InfluxDB) and the broker to avoid the need for scaling InfluxDB in the same order as we scale the broker.

The following diagram depicts the setup:

test-setup

Notifications for everything

A subscription to receive notifications for all changes on entities is created. That subscription is similar to those are used by timeseries backends like QuantumLeap or Cygnus . A number of entities will be created, then 2 attributes(timestamp and humidity) will be updated via POST /entities/<ID>/attrs with a delay between each of the updates.

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numUpdates	How many updates should be executed for each attribute.	100
updateDelay	Delay between attribute updates in seconds.	1
notificationServerUrl	URL to be notified by the broker	http://telegraf:8080/telegraf

Notifications for entities of a certain type(only LD)

A subscription to receive notifications for all changes on entities of a certain type is created. In NGSI-LD, this is similar to those are use by That subscription is similar to those are used by timeseries backends like QuantumLeap or Cygnus. A number of such entities will be created, then 2 attributes(timestamp and humidity) will be updated via POST /entities/<ID>/attrs with a delay between each of the updates.

Config

Parameter	Description	Example
baseUrl	Url of the broker	http://localhost
numEntities	Number of entities to be simulated.	100
numUpdates	How many updates should be executed for each attribute.	100
updateDelay	Delay between attribute updates in seconds.	1
notificationServerUrl	URL to be notified by the broker	http://telegraf:8080/telegraf