Benchmarks
July 20, 2026 · View on GitHub
BenchmarkDotNet-based benchmarks of FastBertTokenizer. Two suites:
TokenizeSpeedmeasures the different usage patterns of FastBertTokenizer itself (single threaded, memory-reusing, batched/multi threaded, batch enumerators). It runs every benchmark for the local build as well as for the released NuGet baseline version, each on all supported (non-EOL) runtimes. This answers "did my change make it faster?" and "is the next release faster than the current one?".OtherLibscompares FastBertTokenizer against other tokenizer libraries usable from .NET: Microsoft.ML.Tokenizers, Tokenizers.DotNet (bindings for Hugging Face's Rust tokenizers) and BlingFire (Microsoft's C++ tokenizer, unmaintained since ~2021 but long the go-to fast BERT tokenizer for .NET; it doesn't read vocab.txt — its precompiled bert-base-uncased model ships indata/blingfire/).
Tokenizers that are not natively usable from .NET are benchmarked from their own ecosystems instead, so interop overhead doesn't skew their numbers:
../HuggingfaceTokenizer/BenchPythonmeasures Hugging Face tokenizers (Rust core), flash-tokenizer (C++ core) and tokie (Rust core) through their Python APIs — the way virtually all of their users consume them. It also contains an id-level parity check against Hugging Face tokenizers (verify.py).../HuggingfaceTokenizer/BenchRustmeasures Hugging Face tokenizers natively via criterion.rs, without any FFI or Python overhead.
All benchmarks tokenize the same corpus - 15,000 articles from simple english wikipedia - with the same vocabulary (baai-bge-small-en, which uses bert-base-uncased's vocab) and truncate to 512 tokens per input.
Running
Prerequisites: the .NET SDK version pinned in global.json plus the
.NET 8 runtime. The benchmark corpus (data/wiki-simple.json.br) ships in the repo and is
available after a normal clone.
cd src/Benchmarks
# FastBertTokenizer usage patterns (local build + NuGet baseline, net8.0 + net10.0):
dotnet run -c Release -f net10.0 -- --filter '*TokenizeSpeed*'
# comparison against other libraries:
dotnet run -c Release -f net10.0 -- --filter '*OtherLibs*'
# everything, but with quick short-running jobs (what CI does on PRs):
dotnet run -c Release -f net10.0 -- --smoke --filter '*'
# list all benchmarks:
dotnet run -c Release -f net10.0 -- --list flat
--smoke is this project's own flag and selects BenchmarkDotNet's short-run jobs (and skips
the NuGet baseline); everything else is passed through to BenchmarkDotNet, see
its docs for all arguments.
Results are written to BenchmarkDotNet.Artifacts/.
The released baseline version measured by TokenizeSpeed is defined in
BenchmarkConfigs.cs (BenchmarkDefaults.BaselineNuGetVersion) and
must be kept in sync with the FastBertTokenizer PackageReference in
Benchmarks.csproj and the centrally pinned version in
Directory.Packages.props.
For the Python harness see ../HuggingfaceTokenizer/BenchPython:
pip install -r requirements.txt && python verify.py && python bench.py.
CI
benchmark.yml runs a quick smoke pass on every
push/PR (except markdown-only changes; it verifies the benchmarks and the measured API
surface still work) and the full suite on demand (workflow_dispatch) and monthly. Every run
uploads the complete BenchmarkDotNet.Artifacts results as workflow artifacts.
Results
Numbers below are from this full CI run on a shared GitHub Actions runner. They are reproducible by anyone, but noisier than numbers from dedicated hardware — treat small differences as noise, and note that multi-threaded results depend on the runner's (few) cores.
BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)
AMD EPYC 9V74 2.60GHz, 1 CPU, 4 logical and 2 physical cores (GitHub Actions shared runner)
.NET SDK 10.0.302
FastBertTokenizer usage patterns: .NET 8 vs. .NET 10
- Workload: Encode up to 512 tokens from each of the 15,000 articles (3,657,145 tokens produced).
- ~12.5m tokens/s single threaded, ~31.6m tokens/s multi threaded on the runner's 4 vCPUs.
localjobs measure the working tree built from source,nugetjobs the released baseline package.
| Method | Job | Runtime | Mean | Error | StdDev | Ratio | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|---|---|
| SinglethreadedAllocating | local-net10.0 | .NET 10.0 | 293.4 ms | 3.31 ms | 3.09 ms | 1.00 | 2039.61 KB | 1.00 |
| SingleThreadedMemReuse | local-net10.0 | .NET 10.0 | 289.0 ms | 2.98 ms | 2.64 ms | 0.99 | 996.98 KB | 0.49 |
| MultithreadedMemReuseBatched | local-net10.0 | .NET 10.0 | 115.6 ms | 2.31 ms | 4.27 ms | 0.39 | 13006.03 KB | 6.38 |
| MultithreadedMemReuseAtOnce | local-net10.0 | .NET 10.0 | 116.8 ms | 1.74 ms | 1.54 ms | 0.40 | 180987.01 KB | 88.74 |
| ParallelBatchEnumerator | local-net10.0 | .NET 10.0 | 234.4 ms | 2.64 ms | 2.47 ms | 0.80 | 4795.53 KB | 2.35 |
| BatchEnumerator | local-net10.0 | .NET 10.0 | 458.1 ms | 6.92 ms | 6.13 ms | 1.56 | 2308.48 KB | 1.13 |
| SinglethreadedAllocating | local-net8.0 | .NET 8.0 | 296.6 ms | 2.84 ms | 2.66 ms | 1.00 | 2039.61 KB | 1.00 |
| SingleThreadedMemReuse | local-net8.0 | .NET 8.0 | 302.0 ms | 3.97 ms | 3.71 ms | 1.02 | 996.98 KB | 0.49 |
| MultithreadedMemReuseBatched | local-net8.0 | .NET 8.0 | 130.7 ms | 2.34 ms | 2.88 ms | 0.44 | 13006.34 KB | 6.38 |
| MultithreadedMemReuseAtOnce | local-net8.0 | .NET 8.0 | 130.0 ms | 1.43 ms | 1.34 ms | 0.44 | 180986.88 KB | 88.74 |
| ParallelBatchEnumerator | local-net8.0 | .NET 8.0 | 252.7 ms | 4.03 ms | 3.77 ms | 0.85 | 4798.02 KB | 2.35 |
| BatchEnumerator | local-net8.0 | .NET 8.0 | 479.5 ms | 4.21 ms | 3.94 ms | 1.62 | 2308.48 KB | 1.13 |
| SinglethreadedAllocating | nuget-1.0.28-net10.0 | .NET 10.0 | 282.0 ms | 1.22 ms | 1.08 ms | 1.00 | 2039.61 KB | 1.00 |
| SingleThreadedMemReuse | nuget-1.0.28-net10.0 | .NET 10.0 | 287.7 ms | 2.55 ms | 2.39 ms | 1.02 | 996.98 KB | 0.49 |
| MultithreadedMemReuseBatched | nuget-1.0.28-net10.0 | .NET 10.0 | 112.1 ms | 2.20 ms | 2.06 ms | 0.40 | 13009.12 KB | 6.38 |
| MultithreadedMemReuseAtOnce | nuget-1.0.28-net10.0 | .NET 10.0 | 114.7 ms | 1.92 ms | 1.80 ms | 0.41 | 180987.01 KB | 88.74 |
| ParallelBatchEnumerator | nuget-1.0.28-net10.0 | .NET 10.0 | 222.3 ms | 2.96 ms | 2.77 ms | 0.79 | 4796.37 KB | 2.35 |
| BatchEnumerator | nuget-1.0.28-net10.0 | .NET 10.0 | 452.6 ms | 7.66 ms | 6.79 ms | 1.61 | 2308.48 KB | 1.13 |
| SinglethreadedAllocating | nuget-1.0.28-net8.0 | .NET 8.0 | 307.4 ms | 2.49 ms | 2.33 ms | 1.00 | 2039.61 KB | 1.00 |
| SingleThreadedMemReuse | nuget-1.0.28-net8.0 | .NET 8.0 | 304.1 ms | 2.08 ms | 1.84 ms | 0.99 | 996.98 KB | 0.49 |
| MultithreadedMemReuseBatched | nuget-1.0.28-net8.0 | .NET 8.0 | 152.5 ms | 3.04 ms | 3.62 ms | 0.50 | 13006.34 KB | 6.38 |
| MultithreadedMemReuseAtOnce | nuget-1.0.28-net8.0 | .NET 8.0 | 161.2 ms | 2.30 ms | 2.15 ms | 0.52 | 180986.88 KB | 88.74 |
| ParallelBatchEnumerator | nuget-1.0.28-net8.0 | .NET 8.0 | 274.2 ms | 3.46 ms | 3.23 ms | 0.89 | 4808.74 KB | 2.36 |
| BatchEnumerator | nuget-1.0.28-net8.0 | .NET 8.0 | 496.6 ms | 4.43 ms | 4.15 ms | 1.62 | 2308.48 KB | 1.13 |
vs. other tokenizer libraries for .NET
- Microsoft.ML.Tokenizers'
BertTokenizer(v2.0.0) - Tokenizers.DotNet (v1.4.1), .NET bindings for Hugging Face's Rust tokenizers
- BlingFire (v0.1.8), Microsoft's C++ tokenizer with an official .NET package
All single threaded on .NET 10, same environment and run as above:
| Method | Mean | Error | StdDev | Ratio | Allocated | Alloc Ratio |
|---|---|---|---|---|---|---|
| FastBertTokenizer | 291.9 ms | 3.40 ms | 3.02 ms | 1.00 | 2039.61 KB | 1.000 |
| MicrosoftMLTokenizers | 813.5 ms | 6.50 ms | 6.08 ms | 2.79 | 106349.56 KB | 52.142 |
| BlingFire | 1,260.7 ms | 2.42 ms | 2.14 ms | 4.32 | 0.02 KB | 0.000 |
| TokenizersDotNet | 5,311.9 ms | 45.17 ms | 42.25 ms | 18.20 | 14778.24 KB | 7.246 |
Fairness notes: the libraries don't do exactly the same work — FastBertTokenizer emits
input_ids and attention_mask, Microsoft.ML.Tokenizers and Tokenizers.DotNet emit just
input_ids, and Hugging Face tokenizers (behind Tokenizers.DotNet) computes offsets and
more. Tokenizers.DotNet's number includes its per-call .NET↔Rust interop cost, which is
inherent to using it from .NET; for Hugging Face tokenizers numbers without .NET interop
see the cross-language results below (Python-driven — a fully native measurement is
possible via BenchRust, whose results are not included in these tables). BlingFire does
the least work of all: ids only, without [CLS]/[SEP], and its precompiled model agrees
with Hugging Face on ~99.9% of tokens rather than exactly.
Also note that the Allocated column tracks managed GC allocations only: whatever BlingFire
(C++) and Tokenizers.DotNet's Rust side allocate natively is invisible to BenchmarkDotNet's
MemoryDiagnoser, so the column is only meaningful for the pure-managed libraries.
Correctness also differs: FastBertTokenizer's output is
continuously tested to match Hugging Face transformers'
AutoTokenizer.
Cross-language: Hugging Face tokenizers (Rust), flash-tokenizer (C++) and tokie (Rust)
From the same CI run (Python 3.12, tokenizers 0.23.1, flash-tokenizer 1.2.0, tokie 0.0.10), measured from Python — the way virtually all users of these libraries consume them — tokenizing the full corpus once:
| Benchmark | Mean |
|---|---|
| hf_tokenizers_singlethreaded | 9.13 s ± 0.05 |
| flash_tokenizer_singlethreaded | 1.02 s ± 0.00 |
| tokie_sequential_calls | 817 ms ± 11 |
| hf_tokenizers_batch (parallel) | 4.10 s ± 0.03 |
| flash_tokenizer_batch (parallel) | 731 ms ± 20 |
| flash_tokenizer_batch_ids_only | 489 ms ± 2 |
| tokie_batch (parallel) | 634 ms ± 5 |
The single-threaded Hugging Face number includes per-call Python overhead; the batch mode
amortizes that and additionally parallelizes across documents, so these numbers don't
isolate the Rust core's raw speed (BenchRust can measure that natively, but its results
are not included in these tables). flash-tokenizer's batch number includes its pure-Python
attention_mask/token_type_ids construction; flash_tokenizer_batch_ids_only measures it
without that. tokie may parallelize internally even for single calls, so read its
tokie_sequential_calls number as "sequential API calls", not necessarily "one core".
An id-level parity check (verify.py) shows flash-tokenizer produces ids identical to
Hugging Face tokenizers for 99.6% of the corpus documents, while tokie matches exactly.
For scale: FastBertTokenizer tokenizes the same corpus single threaded in ~0.3 s in the
same CI run (tables above).