Benchmarks
September 6, 2023 · View on GitHub
In this section, you'll find comprehensive benchmark results that compare Pipelines in different configuration comparing to other libraries.
Table of Contents
1. Benchmarking Pipelines: Emulating Real-World Application Scenarios
In order to rigorously evaluate the performance and efficiency of the Pipelines library, we have designed a comprehensive benchmarking scenario that closely mirrors the complexity of real-world applications. This benchmark is intended to provide users with valuable insights into how Pipelines can handle diverse and demanding workloads.
1.1 Configuration
- Dataset Generation: prepared a dataset consisting of 500 distinct input types.
- Handler Diversity: to simulate the diverse nature of real-world applications, we have implemented 500 unique handlers. Each handler is designed to process specific types of inputs, ensuring that Pipelines can effectively manage the complexities associated with handling a multitude of data sources and operations.
- Decorator Complexity: each handler in our benchmark scenario is adorned with 5 distinct decorators.
- Scoped dependency injection: each scenario create DI scope similar when handling request
- Benchmark framework: BenchmarkDotNet is used to run benchmarks. Each benchmark sends 25 requests and gets results from each handler.
1.2 Benchmark methods
- WrapperDispatcherGenerator: dispatcher generated by Pipelines.WrapperDispatcherGenerator
- MediatR: popular library for handling mediator pattern
- MediatRWithBehaviours: MediatR with configured 5 behaviours/decorators
- PipelinesReflection: dispatcher handled by reflection mechanism
- WrapperDispatcherGeneratorWithDecorators: dispatcher generated by dispatcher generated by Pipelines.WrapperDispatcherGenerator with configured 5 decorators
- PipelinesReflectionWithDecorators: dispatcher handled by reflection mechanism with configured 5 decorators
1.3 Results
BenchmarkDotNet v0.13.6, macOS Ventura 13.4.1 (c) (22F770820d) [Darwin 22.5.0]
Apple M1 Pro, 1 CPU, 8 logical and 8 physical cores
.NET SDK 7.0.200
[Host] : .NET 7.0.3 (7.0.323.6910), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 7.0.3 (7.0.323.6910), Arm64 RyuJIT AdvSIMD
| Method | Mean | Error | StdDev | Ratio | RatioSD | Rank | Gen0 | Gen1 | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|---|---|
| WrapperDispatcherGenerator | 7.498 μs | 0.0689 μs | 0.0644 μs | 1.00 | 0.00 | 1 | 2.9068 | 0.0305 | 17.84 KB | 1.00 |
| MediatR | 10.091 μs | 0.1624 μs | 0.1356 μs | 1.34 | 0.02 | 2 | 2.3956 | 0.0153 | 14.72 KB | 0.82 |
| MediatRWithBehaviours | 30.773 μs | 0.1503 μs | 0.1255 μs | 4.09 | 0.02 | 3 | 8.7280 | 2.1362 | 53.6 KB | 3.00 |
| PipelinesReflection | 39.550 μs | 0.2030 μs | 0.1799 μs | 5.27 | 0.04 | 4 | 6.3477 | 0.0610 | 39 KB | 2.19 |
| WrapperDispatcherGeneratorWithDecorators | 129.368 μs | 2.5578 μs | 4.7411 μs | 17.10 | 1.07 | 5 | 9.7656 | 4.8828 | 60.08 KB | 3.37 |
| PipelinesReflectionWithDecorators | 160.350 μs | 1.0823 μs | 0.9595 μs | 21.36 | 0.19 | 6 | 13.4277 | 6.5918 | 82.3 KB | 4.61 |
Based on the benchmark results, it is evident that the WrapperDispatcherGenerator method stands out as the fastest among the tested options.
However, an interesting observation arises when we examine the performance of the WrapperDispatcherGeneratorWithDecorators method, which is noticeably slower than MediatRWithBehaviours, despite the better performance without decorators. This performance gap can be attributed to the use of ActivatorUtilities.CreateInstance and the registration of decorators through implementationFactory in dependency injection, which introduces overhead compared to the more streamlined approach employed by MediatR.
In our roadmap, we have plans to address this performance discrepancy and optimize the WrapperDispatcherGeneratorWithDecorators method. Our goal is to enhance the efficiency of Pipelines, ensuring that it remains competitive with other solutions like MediatRWithBehaviours while still offering the flexibility and extensibility that decorators provide.