Benchmarking

March 27, 2020 · View on GitHub

This repository contains the performance results and comparisons obtained by eProsima using different testing tools. Our aim is to gather performance results for different DDS implementations and versions so their performance can be easily compared.

Fast-RTPS Automated Benchmark
DDS Vendors Comparison
Fast-RTPS Cross-Version Comparison
iRobot Performance Tests

Fast-RTPS Automated Benchmark

Fast-RTPS Automated Benchmark provides a framework to automate the execution of performance tests for Fast-RTPS that is easy to deploy and integrate with Continuos Integration (CI) pipelines. The main goal is to ease the deployment and execution of performance test runs, generating reports that can be shared and used to evaluate Fast-RTPS' performance from various angles.

The following is an example of one of the many plots generated for the case of Latency tests. It shows the evolution of Fast-RTPS' latency median performance when sending messages of 512 Bytes between a publisher and a subscriber in the same process when the reliability is set to BEST_EFFORT.

latency_history_payload

DDS Vendors Comparison

We execute and compare performance test for different DDS implementations (Fast-RPTS, CycloneDDS, and OpenSplice), using the performance testing tools provided by each vendor. We look at performance in terms of:

With the purpose of repeatability, each test case contains a description of the executed experiments, detailing the testing environment and configuration, as well as the software versions and any other information required to replicate the experiment.

Results

An article on the latency performance of Fast-RTPS, CycloneDDS, and OpenSplice can be found here. The following constitute an example of such reports: one for latency and another one for throughput, both in the localhost test case (publisher and subscriber share machine but not process).

Latency Comparison Localhost

Latency comparison between Fast-RTPS (010ac53), CycloneDDS (801c4b1) and OpenSplice (v6.9)

Throughput Comparison Localhost

Throughput comparison between Fast-RTPS (0bcafbd), CycloneDDS (aa5236d) and OpenSplice (v6.9)

Fast-RTPS Cross-Version Comparison

To keep track of our own library's progress, we execute cross-version performance comparisons in terms of Latency.

v1.9.2-v1.9.3 Latency Cross-Comparison Localhost

v1.9.2-v1.9.3 Latency Cross-Comparison Same Process

iRobot Performance Tests

iRobot has created a benchmark application to test the performance of ROS 2 in terms of latency, memory consumption, and CPU usages. This repository, not only contains tool for performance testing, but also a detail guide on how to cross-compile ROS 2 for a RaspberryPi 3b+ with Raspbian. We use that tool to characterize the different vendors' performance on a RaspberryPi 3b+. Here, you can find the experiments log, with a detail description of testing environment and configuration, as well as software versions tested. To ease the configuration of the experiments for different test-cases, a set of utilities for the iRobot performance tests has been created.