Kostya JSON Benchmark

November 19, 2025 · View on GitHub

This repository has been forked from the original to add the Lite³ format to the benchmarks.

Language / LibraryExecution TimeMemory Usage
C++/g++ (DAW JSON Link)0.094 s113 MB
C++/g++ (RapidJSON)0.1866 s238 MB
C++/g++ (gason)0.1462 s209 MB
C++/g++ (simdjson DOM)0.1515 s285 MB
C++/g++ (simdjson On-Demand)0.0759 s173 MB
C/gcc (lite3)0.027 s203 MB
C/gcc (lite3_context_api)0.027 s203 MB
Go (Sonic)0.2246 s121 MB
Rust (Serde Custom)0.113 s111 MB
Zig0.2493 s147 MB

Benchmark data:

To replicate this benchmark, run:

git clone https://github.com/fastserial/kostya-benchmark.git
cd kostya-benchmark/
docker build docker/ -t benchmarks

Warning: this will build a 24 GB docker image.

A single benchmark run can now be performed like so:

docker run -it --rm -v $(pwd):/src benchmarks json

However to produce more consistent results, CPU frequency scaling should first be disabled to minimize variance:

apt update
apt install linux-cpupower
cpupower frequency-set -g performance
cpupower frequency-info

You should see:

    The governor "performance" may decide which speed to use

The OS can also introduce variance by inconsistent scheduling of threads across NUMA-domains. To prevent this, the process and memory should be pinned. Also, not one but multiple runs will increase the consistency of the results.

This command will perform 10 benchmark runs and write the results to output.txt:

lscpu >> output.txt && \
numactl -H >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt && \
docker run -it --rm --cpuset-cpus="0" --cpuset-mems="0" -v $(pwd):/src benchmarks json >> output.txt

Original README:

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.

  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU package during the benchmark: PP0 (cores) + PP1 (uncores like GPU) + DRAM. Currently, only Intel CPU are supported via the powercap interface.

All values are presented as: median±median absolute deviation.

UPDATE: 2025-08-17

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

LanguageTime, sMemory, MiBEnergy, J
Scala (Staged)0.409±0.011229.62±03.27 + 7.00±01.4426.31±00.71
Racket (Staged)0.885±0.000101.37±01.65 + 0.06±00.0634.39±00.17
V/gcc0.985±0.0012.12±00.00 + 0.00±00.0041.65±00.28
Rust1.008±0.0001.88±00.00 + 0.00±00.0042.93±00.33
D/ldc21.078±0.0013.12±00.00 + 0.00±00.0045.78±00.24
C++/g++1.097±0.0023.38±00.00 + 0.00±00.0046.21±00.59
C/gcc1.112±0.0011.38±00.12 + 0.00±00.0047.10±00.09
D/gdc1.116±0.0016.75±00.00 + 0.00±00.0048.89±00.28
C/clang1.136±0.0001.44±00.06 + 0.00±00.0048.52±00.33
C++/clang++1.167±0.0022.88±00.00 + 0.00±00.0048.87±00.76
Nim/gcc1.171±0.0012.00±00.00 + 0.00±00.0048.80±00.70
Java1.191±0.00041.74±00.09 + 1.00±00.0049.32±00.60
Vala/gcc1.202±0.0015.25±00.12 + 0.00±00.0050.70±00.44
Kotlin/JVM1.224±0.00244.48±00.07 + 1.31±00.2551.60±00.64
Vala/clang1.235±0.0005.19±00.06 + 0.00±00.0052.40±00.39
Go1.256±0.0003.75±00.00 + 0.00±00.0052.69±00.80
C#/.NET Core1.283±0.00134.56±00.07 + 0.12±00.1255.31±00.24
Go/gccgo1.287±0.00023.50±00.38 + 0.00±00.0054.23±00.16
Zig1.327±0.0001.38±00.12 + 0.00±00.0055.13±00.40
F#/.NET Core1.507±0.00339.33±00.10 + 0.19±00.0665.29±00.40
Crystal1.545±0.0013.25±00.00 + 0.00±00.0065.69±00.40
Julia1.564±0.003244.00±00.12 + 0.38±00.0067.80±00.43
Nim/clang1.579±0.0002.25±00.00 + 0.00±00.0065.59±00.57
OCaml1.607±0.0023.25±00.00 + 2.62±00.0075.45±00.78
MLton1.737±0.0061.62±00.00 + 0.38±00.0074.87±00.28
Chez Scheme1.758±0.00525.88±00.29 + 3.25±00.0074.43±00.21
Racket1.761±0.013112.86±00.03 + 0.50±00.3872.49±02.28
V/clang1.846±0.0092.06±00.06 + 0.00±00.0080.00±00.53
Scala2.379±0.00366.03±00.14 + 196.44±00.12108.88±00.92
Node.js2.954±0.06049.57±00.06 + 5.78±00.12126.35±03.17
D/dmd3.010±0.0013.50±00.00 + 0.00±00.00113.43±01.11
Haskell (MArray)3.497±0.0014.00±00.00 + 4.00±00.00144.74±00.32
Haskell (FP)3.819±0.0084.12±00.06 + 4.06±00.06163.47±01.56
Ruby/truffleruby (JVM)4.645±0.070354.81±05.59 + 478.37±54.20221.41±05.66
Ruby/truffleruby5.169±0.461197.19±00.69 + 693.00±117.12252.03±23.51
Swift5.497±0.00117.00±00.12 + 0.00±00.00205.09±04.14
Lua/luajit5.686±0.0242.50±00.00 + 0.00±00.00235.32±01.30
Python/pypy9.398±0.03860.19±00.19 + 30.10±00.05419.65±03.72
Idris15.088±0.03221.55±00.26 + 8.00±00.00677.49±01.79
Elixir20.456±0.02590.88±00.30 + 0.00±00.00814.83±07.69
Ruby (--jit)30.807±0.03321.88±00.13 + 4.62±00.001275.90±04.28
PHP34.055±0.04117.61±00.06 + 0.00±00.001447.00±15.80
Lua37.051±0.1682.62±00.00 + 0.00±00.001528.23±05.85
Ruby/jruby38.543±0.745255.48±00.77 + 70.31±00.251732.28±34.61
Python61.009±0.87011.38±00.00 + 0.00±00.002601.10±34.54
Ruby74.107±0.33212.12±00.06 + 0.00±00.003209.85±28.74
Tcl (FP)189.615±0.4654.62±00.00 + 0.00±00.008409.77±69.18
Perl225.517±1.8737.06±00.06 + 0.12±00.009777.03±135.29
Tcl (OOP)373.095±1.3414.62±00.00 + 0.00±00.0016604.13±190.89

mandel.b

Mandel in Brainfuck

LanguageTime, sMemory, MiBEnergy, J
Scala (Staged)7.118±0.079228.10±04.74 + 71.23±06.06356.09±07.48
C++/g++10.143±0.0513.50±00.00 + 0.62±00.00434.41±04.03
C/gcc12.544±0.0191.62±00.00 + 0.25±00.00509.37±03.30
Vala/gcc12.718±0.0105.25±00.06 + 0.50±00.00511.94±00.90
C#/.NET Core12.742±0.02834.69±00.06 + 1.25±00.06494.75±01.32
C++/clang++12.803±0.0223.00±00.00 + 0.62±00.00545.25±01.09
Zig13.030±0.0171.50±00.00 + 1.00±00.00556.10±03.71
Java13.041±0.02041.77±00.15 + 1.94±00.06537.41±01.88
F#/.NET Core13.104±0.04939.32±00.12 + 2.00±00.00527.94±02.27
C/clang13.120±0.0211.50±00.06 + 0.25±00.00579.20±04.21
Kotlin/JVM13.463±0.05244.46±00.06 + 2.56±00.06571.73±03.08
D/ldc214.340±0.0123.06±00.06 + 0.75±00.00591.01±03.53
Rust14.460±0.0211.94±00.06 + 0.25±00.00577.37±02.52
Go14.540±0.0073.75±00.00 + 0.00±00.00584.48±01.83
Vala/clang14.802±0.0125.12±00.00 + 0.50±00.00595.99±01.53
Crystal14.891±0.0543.38±00.06 + 0.38±00.00631.17±02.60
D/gdc15.174±0.0106.62±00.00 + 0.88±00.00640.85±03.66
Racket (Staged)15.722±0.08599.73±00.12 + 71.69±01.06613.58±05.37
Nim/gcc15.766±0.0162.00±00.06 + 1.50±00.00635.49±04.84
Scala15.785±0.06765.94±00.09 + 145.00±00.19692.62±02.98
Nim/clang17.648±0.0352.25±00.06 + 1.50±00.00726.43±04.78
V/gcc17.865±0.0162.19±00.06 + 1.00±00.00675.21±02.51
Swift18.083±0.08516.94±00.25 + 0.12±00.00742.88±01.58
V/clang18.691±0.1262.25±00.00 + 1.00±00.00813.69±05.16
Go/gccgo19.206±0.17923.38±00.12 + 0.00±00.00796.93±09.15
OCaml24.557±0.0064.38±00.00 + 3.14±00.001195.26±09.14
Julia30.315±0.060244.06±00.07 + 0.25±00.001274.74±02.15
Chez Scheme30.884±0.15126.14±00.17 + 3.00±00.001367.28±07.09
Lua/luajit31.804±0.0262.50±00.00 + 0.25±00.001312.84±01.18
Node.js34.692±0.09749.51±00.00 + 8.19±00.061413.09±11.38
Racket35.373±0.400112.90±00.09 + 2.75±00.441573.97±10.34
MLton35.680±0.0511.62±00.00 + 4.28±00.001623.52±10.03
Haskell (MArray)36.789±0.0094.12±00.06 + 5.00±00.001522.12±01.36
D/dmd38.105±0.0053.50±00.00 + 0.75±00.001391.62±02.04
Python/pypy41.153±0.22960.12±00.19 + 31.00±00.091825.02±12.90
Ruby/truffleruby48.381±0.372196.62±01.12 + 582.69±01.752346.69±17.27
Ruby/truffleruby (JVM)50.429±0.473355.72±06.70 + 431.93±35.082253.46±91.61
Idris64.504±0.23721.73±00.14 + 9.88±00.002857.66±12.12
Haskell (FP)79.505±0.1924.00±00.00 + 74.81±00.063368.24±08.74

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

LanguageTime, sMemory, MiBEnergy, J
PHP0.065±0.00118.04±00.18 + 0.00±00.003.27±00.03
C/clang (aklomp)0.095±0.0002.12±00.00 + 0.00±00.004.57±00.10
C/gcc (aklomp)0.097±0.0002.12±00.00 + 0.00±00.004.65±00.04
Go (base64x)0.283±0.0095.00±00.00 + 0.00±00.0014.12±00.33
Node.js0.635±0.00547.63±00.06 + 81.13±00.9529.16±00.53
Zig0.703±0.0001.81±00.06 + 0.12±00.0026.42±00.09
Rust0.842±0.0002.31±00.06 + 0.12±00.0034.91±00.90
C/clang0.918±0.0002.00±00.00 + 0.12±00.0034.47±00.55
Nim/clang0.945±0.0002.75±00.00 + 5.00±00.0038.18±01.09
Crystal1.061±0.0173.75±00.00 + 1.00±00.0043.27±00.84
C/gcc1.117±0.0021.88±00.00 + 0.12±00.0041.76±00.87
Nim/gcc1.348±0.0022.62±00.00 + 4.62±00.0054.45±00.73
D/ldc21.352±0.0023.75±00.00 + 3.38±00.0055.63±00.63
Java1.435±0.00443.04±00.06 + 248.75±00.1260.04±01.08
Kotlin/JVM1.561±0.02145.70±00.10 + 252.06±00.3168.06±02.36
Go1.592±0.0074.31±00.06 + 0.00±00.0066.55±01.37
Scala1.594±0.05063.26±00.10 + 315.25±04.1968.76±01.29
V/gcc1.641±0.0032.75±00.00 + 2383.00±01.4462.77±03.08
Vala/clang1.646±0.0035.62±00.00 + 0.12±00.0064.21±01.67
Vala/gcc1.647±0.0025.75±00.00 + 0.12±00.0065.43±02.11
Ruby1.861±0.00512.38±00.00 + 43.13±00.4973.89±01.16
Perl (MIME::Base64)1.997±0.00814.88±00.00 + 0.00±00.0081.05±01.87
Ruby (--jit)2.033±0.00546.26±00.04 + 64.14±00.0479.75±00.49
C#/.NET Core2.372±0.02135.25±00.06 + 5.34±00.0697.09±01.95
F#/.NET Core2.445±0.07639.86±00.06 + 20.22±00.9498.47±03.45
Tcl2.749±0.0035.25±00.00 + 0.12±00.00115.56±01.50
Go/gccgo3.037±0.00724.56±00.19 + 0.00±00.00144.41±01.00
Julia3.085±0.019258.12±00.02 + 28.20±00.07126.59±02.11
Python3.099±0.00211.25±00.00 + 0.10±00.00126.46±01.27
V/clang3.258±0.0032.62±00.00 + 2388.00±01.12121.78±02.22
Ruby/truffleruby (JVM)3.505±0.036350.81±05.12 + 256.81±12.70182.88±06.01
Python/pypy3.534±0.00359.50±00.12 + 30.36±00.08158.06±01.43
C++/clang++ (libcrypto)3.664±0.0076.38±00.00 + 0.25±00.00152.34±02.81
C++/g++ (libcrypto)3.664±0.0037.00±00.00 + 0.25±00.00153.36±02.88
D/dmd3.760±0.0033.62±00.00 + 3.38±00.00158.04±01.48
Racket3.886±0.00795.64±00.10 + 21.81±00.19156.59±01.21
D/gdc3.976±0.0037.00±00.06 + 3.62±00.00166.06±02.47
Ruby/jruby4.129±0.009240.66±00.49 + 78.50±00.12184.97±01.57
Ruby/truffleruby8.789±0.034196.12±01.06 + 544.31±06.06422.24±01.81
Perl (MIME::Base64::Perl)10.391±0.08016.25±00.12 + 0.19±00.01462.87±10.60

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

Json

LanguageTime, sMemory, MiBEnergy, J
C++/g++ (simdjson On-Demand)0.060±0.000113.38±00.00 + 59.75±00.002.55±00.02
C++/clang++ (simdjson On-Demand)0.062±0.000113.00±00.00 + 59.75±00.002.59±00.03
C++/clang++ (DAW JSON Link NoCheck)0.070±0.000112.82±00.01 + 0.00±00.002.94±00.04
C++/clang++ (DAW JSON Link)0.085±0.000112.76±00.06 + 0.00±00.003.61±00.05
C++/g++ (DAW JSON Link NoCheck)0.087±0.000113.21±00.00 + 0.00±00.003.54±00.05
C++/g++ (DAW JSON Link)0.089±0.000113.21±00.00 + 0.00±00.003.66±00.02
Rust (Serde Custom)0.103±0.000111.62±00.00 + 0.00±00.004.43±00.03
C++/clang++ (simdjson DOM)0.103±0.001112.88±00.00 + 175.06±00.694.72±00.08
C++/g++ (simdjson DOM)0.107±0.002113.44±00.06 + 175.86±00.894.89±00.06
Rust (Serde Typed)0.110±0.000111.75±00.00 + 11.12±00.004.70±00.02
C++/g++ (gason)0.127±0.000113.21±00.00 + 96.75±00.005.16±00.06
C++/clang++ (gason)0.133±0.001112.70±00.00 + 96.75±00.005.35±00.04
D/ldc2 (Mir Asdf DOM)0.133±0.001112.81±00.06 + 61.25±00.005.56±00.07
Scala (jsoniter-scala)0.147±0.001283.94±00.09 + 25.62±00.197.89±00.03
C++/g++ (RapidJSON)0.154±0.001113.21±00.00 + 123.44±00.886.61±00.07
C++/clang++ (RapidJSON)0.194±0.001112.95±00.00 + 128.62±00.008.25±00.16
Go (rjson custom)0.199±0.000114.00±00.06 + 0.00±00.007.68±00.01
Go (Sonic)0.209±0.001121.88±00.12 + 0.00±00.009.14±00.05
C++/g++ (RapidJSON Precise)0.212±0.001113.21±00.00 + 128.75±00.009.15±00.11
D/ldc2 (Mir Amazon's Ion DOM)0.220±0.001113.00±00.00 + 80.75±00.009.29±00.05
Go (rjson)0.224±0.000114.00±00.00 + 0.00±00.008.65±00.03
Zig0.239±0.001111.12±00.00 + 36.44±00.0610.29±00.04
Go (goccy/go-json)0.252±0.000114.50±00.06 + 0.00±00.009.99±00.08
C++/clang++ (RapidJSON Precise)0.262±0.001112.83±00.00 + 128.75±00.0011.08±00.14
C/gcc (yajl)0.362±0.000111.25±00.00 + 0.00±00.0015.71±00.12
C/clang (yajl)0.364±0.000111.25±00.00 + 0.00±00.0015.65±00.08
C++/g++ (RapidJSON SAX)0.366±0.001113.33±00.00 + 0.00±00.0015.95±00.12
C++/g++ (Boost.JSON)0.374±0.002113.47±00.00 + 307.75±00.0015.97±00.15
Node.js0.374±0.001158.06±00.06 + 283.88±03.0018.22±00.13
C++/clang++ (Boost.JSON)0.377±0.002113.00±00.07 + 307.75±00.0016.12±00.12
Nim/clang (jsony)0.386±0.001112.12±00.00 + 159.75±01.9416.19±00.20
Nim/gcc (jsony)0.418±0.001111.75±00.00 + 155.62±01.5017.56±00.10
C++/clang++ (RapidJSON SAX)0.423±0.001195.20±00.00 + 0.00±00.0017.99±00.12
C++/g++ (RapidJSON SAX Precise)0.433±0.000113.34±00.00 + 0.12±00.0019.54±00.20
Go (jsoniter)0.482±0.001114.50±00.12 + 0.00±00.0019.93±00.04
Crystal (Pull)0.484±0.002113.44±00.06 + 18.00±00.0021.76±00.28
C++/clang++ (RapidJSON SAX Precise)0.500±0.001195.20±00.00 + 0.00±00.0022.35±00.40
C#/.NET Core (System.Text.Json)0.500±0.001481.12±00.00 + 119.63±00.0824.55±00.12
Crystal (Schema)0.510±0.001113.38±00.00 + 48.31±00.0622.61±00.39
Rust (Serde Untyped)0.557±0.002111.88±00.12 + 839.75±00.0023.75±00.42
Java (DSL-JSON)0.571±0.020277.81±00.14 + 163.69±00.3829.34±01.43
Python/pypy0.606±0.001280.66±00.07 + 125.44±00.0126.55±00.11
V/clang0.635±0.001112.12±00.00 + 496.00±00.0027.08±00.57
V/gcc0.639±0.005111.88±00.00 + 496.00±00.0027.16±00.35
Nim/clang (Packedjson)0.664±0.002112.38±00.00 + 294.25±00.0028.59±00.36
Nim/gcc (Packedjson)0.674±0.004112.12±00.00 + 294.12±00.0028.60±00.31
Julia (JSON3)0.700±0.001631.76±00.04 + 332.81±01.0029.33±00.22
CPython (UltraJSON)0.700±0.002123.35±00.12 + 478.56±01.8128.00±00.40
Perl (Cpanel::JSON::XS)0.767±0.011125.38±00.00 + 402.88±00.0032.13±00.63
Crystal0.795±0.007113.44±00.06 + 392.12±00.0034.30±00.39
Go0.834±0.001114.00±00.06 + 0.00±00.0034.46±00.07
PHP0.836±0.001127.36±00.12 + 517.82±00.0036.28±00.23
Python0.936±0.003121.30±00.06 + 325.88±00.0039.40±00.44
Ruby (--jit)1.014±0.001126.75±00.02 + 210.64±00.0342.62±00.35
Ruby1.031±0.002122.00±00.12 + 212.08±00.2043.59±00.44
Nim/gcc1.036±0.005112.12±00.00 + 1000.88±00.3843.26±00.27
Nim/clang1.092±0.004112.38±00.00 + 999.00±00.0045.82±00.27
Clojure1.128±0.017476.94±02.22 + 405.56±14.1259.15±01.15
C#/.NET Core1.169±0.005488.06±00.06 + 41.46±00.1757.40±00.51
C++/clang++ (Nlohmann)1.182±0.002112.95±00.00 + 359.88±00.0050.72±00.62
C++/clang++ (json-c)1.185±0.002112.96±00.00 + 1215.88±00.0049.83±00.62
C++/g++ (json-c)1.187±0.004113.35±00.00 + 1215.88±00.0049.45±00.40
Go/gccgo1.256±0.003139.25±00.12 + 0.00±00.0051.58±00.27
C++/g++ (Nlohmann)1.266±0.003113.33±00.00 + 447.88±00.0053.77±00.41
F#/.NET Core (System.Text.Json)1.494±0.003490.33±00.11 + 133.64±00.4370.35±00.46
Ruby (YAJL)1.565±0.019121.69±00.06 + 233.88±00.0068.39±01.94
D/ldc21.633±0.002113.12±00.06 + 638.62±00.0070.66±00.51
C31.959±0.003111.88±00.00 + 795.88±00.0082.11±00.61
Haskell2.019±0.013115.75±00.12 + 723.44±00.1286.17±00.72
C++/clang++ (Boost.PropertyTree)2.589±0.004195.33±00.00 + 1232.50±00.00111.50±00.48
Rust (jq)2.637±0.006113.50±00.00 + 905.97±02.12110.08±00.97
D/dmd2.778±0.003113.50±00.00 + 626.00±00.00121.50±00.76
Ruby/jruby2.838±0.016555.29±00.73 + 572.50±77.88152.07±00.83
C++/g++ (Boost.PropertyTree)2.907±0.010113.46±00.00 + 1439.88±00.00124.59±00.96
Vala/gcc2.980±0.017115.25±00.00 + 980.00±00.00127.79±01.75
Vala/clang2.983±0.012115.25±00.00 + 980.00±00.00127.36±00.59
Odin3.026±0.007111.38±00.00 + 20.00±00.00126.12±01.22
D/gdc3.459±0.007116.75±00.00 + 708.88±00.06148.15±01.48
Racket3.861±0.015223.42±00.33 + 288.29±04.07163.00±01.74
Perl (JSON::Tiny)9.034±0.089126.00±00.00 + 528.56±00.00401.98±06.07
Ruby/truffleruby (JVM)10.618±0.148482.80±07.29 + 2212.51±207.00678.79±08.84
Ruby/truffleruby11.341±0.066402.19±11.19 + 1941.56±30.56665.04±03.89

Matmul

Testing allocating and multiplying matrices.

Matmul

LanguageTime, sMemory, MiBEnergy, J
D/ldc2 (lubeck)0.042±0.00116.38±01.31 + 47.88±01.384.27±00.05
Python (NumPy)0.066±0.00132.78±02.50 + 52.17±02.295.62±00.07
Nim/gcc (Arraymancer)0.073±0.0055.62±00.12 + 58.06±00.195.88±00.23
C++/g++ (Eigen)0.081±0.00010.06±01.62 + 79.75±01.755.68±00.02
C++/clang++ (Eigen)0.081±0.00012.88±00.50 + 77.46±00.566.41±00.04
Java (ND4J)0.082±0.001122.16±01.44 + 92.56±00.196.51±00.11
Rust (ndarray)0.091±0.0012.62±00.00 + 68.36±00.006.20±00.10
Nim/clang (Arraymancer)0.114±0.04518.81±00.44 + 45.62±00.508.31±02.57
Julia (threads: 2)0.120±0.000259.86±00.08 + 56.95±00.026.69±00.05
Julia (threads: 1)0.170±0.000259.82±00.17 + 56.36±00.048.09±00.03
V/clang (VSL)0.224±0.00210.62±00.12 + 51.60±00.0318.12±00.11
V/clang (VSL + CBLAS)0.227±0.00110.62±00.12 + 51.47±00.0918.15±00.23
V/gcc (VSL + CBLAS)0.423±0.0032.50±00.00 + 51.38±00.0032.87±00.19
V/gcc (VSL)0.441±0.0022.50±00.00 + 51.38±00.0030.29±00.18
Julia (no BLAS)1.113±0.017256.38±00.00 + 51.50±00.0049.41±00.63
D/gdc1.505±0.0017.12±00.00 + 4.00±00.0058.37±00.11
D/ldc21.722±0.0013.62±00.00 + 70.38±00.0064.31±00.15
D/dmd1.880±0.0023.56±00.06 + 70.38±00.0072.70±00.56
C/gcc3.034±0.0012.00±00.00 + 68.38±00.00113.24±00.33
V/gcc3.036±0.0002.50±00.06 + 68.75±00.00113.56±00.46
Java3.057±0.00143.14±00.13 + 68.75±00.12123.21±00.95
Vala/clang3.062±0.0015.54±00.04 + 68.38±00.00109.60±00.81
V/clang3.065±0.0002.88±00.00 + 68.75±00.00109.23±00.06
C/clang3.067±0.0012.00±00.00 + 68.38±00.00109.34±01.23
Rust3.067±0.0012.25±00.12 + 68.50±00.00109.32±00.64
Zig3.071±0.0031.88±00.00 + 68.56±00.06109.78±00.28
Nim/gcc3.091±0.0022.62±00.06 + 57.88±00.00115.31±00.37
Swift3.102±0.0018.38±00.12 + 68.62±00.00113.76±00.36
Nim/clang3.121±0.0012.94±00.06 + 57.75±00.00111.65±00.36
Vala/gcc3.131±0.0015.45±00.06 + 68.50±00.00118.43±00.81
Go3.153±0.0014.12±00.12 + 0.00±00.00115.20±00.44
Crystal3.154±0.0013.75±00.00 + 59.50±00.00116.10±00.24
Go/gccgo3.159±0.00124.44±00.12 + 0.00±00.00115.54±00.39
Kotlin/JVM3.160±0.00243.80±00.16 + 70.00±00.12124.32±00.30
Node.js3.212±0.00557.91±00.28 + 70.83±00.19126.71±00.50
Scala3.243±0.00364.43±00.14 + 158.69±00.06119.81±00.30
Python/pypy3.263±0.00260.62±00.12 + 68.94±00.06136.66±00.49
C#/.NET Core4.725±0.00136.62±00.06 + 69.44±00.06193.26±01.10
Ruby/truffleruby (JVM)17.667±0.235417.21±09.41 + 339.96±72.03722.45±10.75
Ruby/truffleruby17.776±0.189324.88±03.31 + 504.50±13.19638.49±08.58
Python136.267±0.43711.62±00.06 + 68.75±00.006131.10±41.35
Perl149.049±1.6338.62±00.06 + 379.75±00.006511.38±65.17
Ruby (--jit)152.991±0.29421.83±01.04 + 68.62±00.066802.85±30.51
Tcl205.116±0.5377.62±00.00 + 400.38±00.009417.77±28.45
Ruby211.868±0.27412.75±00.00 + 69.62±00.009430.93±36.77
Ruby/jruby330.203±16.724299.99±00.72 + 880.65±189.1913544.82±520.08

Primes

Testing:

  • generating primes using the optimized sieve of Atkin;
  • prefix search for their decimal numbers using Trie data structure.

Notes:

  • All languages but V and Python use unordered hashmaps (V and Python don't provide those out of box, and their hashmaps use keys in the insertion order);
  • The results are always sorted (could be unstable or stable though).

Primes

LanguageTime, sMemory, MiBEnergy, J
C++/g++0.064±0.0003.62±00.00 + 85.55±00.442.57±00.06
Go0.070±0.0003.62±00.00 + 0.00±00.003.09±00.09
Zig0.071±0.0001.50±00.12 + 64.69±00.062.94±00.01
C++/clang++0.075±0.0003.25±00.00 + 64.48±00.192.90±00.06
V/clang0.100±0.0002.12±00.00 + 200.44±00.694.18±00.08
Rust0.102±0.0002.00±00.00 + 74.25±00.124.05±00.06
V/gcc0.106±0.0012.25±00.00 + 211.06±02.064.52±00.05
Java0.113±0.00041.79±00.14 + 107.88±03.816.06±00.22
Crystal0.144±0.0003.50±00.00 + 89.88±00.005.91±00.10
Scala0.184±0.00466.51±00.15 + 158.44±02.0010.67±00.30
Node.js0.216±0.00146.88±00.06 + 227.55±04.7710.68±00.15
Nim/gcc0.293±0.0001.75±00.12 + 582.06±01.0011.25±00.17
Nim/clang0.295±0.0012.00±00.00 + 603.44±05.0611.62±00.12
Lua/luajit0.305±0.0012.50±00.00 + 157.22±01.0512.36±00.17
Julia0.419±0.002258.38±00.12 + 215.71±03.0316.24±00.31
Python/pypy0.617±0.00359.62±00.12 + 247.16±00.0425.09±00.34
Racket0.757±0.004109.51±00.71 + 248.35±01.0730.05±00.58
Ruby/truffleruby0.910±0.010197.81±01.44 + 672.44±54.6961.70±00.61
Lua1.230±0.0052.56±00.06 + 283.84±00.7550.66±00.69
Ruby (--jit)1.289±0.01423.96±00.03 + 174.84±00.3154.03±01.54
Ruby/truffleruby (JVM)1.363±0.033350.78±04.33 + 506.71±28.1288.50±02.14
Ruby/jruby1.491±0.021250.45±00.62 + 577.06±02.3180.03±01.19
Ruby1.992±0.02712.12±00.00 + 183.69±00.5085.16±02.19
Python2.096±0.01211.44±00.06 + 181.46±01.0090.37±01.30

Tests Execution

Environment

CPU: Intel(R) Xeon(R) E-2324G

Base Docker image: Debian GNU/Linux trixie/sid

LanguageVersion
.NET Core9.0.304
C#/.NET Core4.14.0-3.25359.3 (6dbcfd2f)
C30.7.4
Chez Scheme10.0.0
Clojure"1.12.1"
Crystal1.17.1
D/dmdv2.111.0
D/gdc14.2.0
D/ldc21.41.0
Elixir1.18.3
F#/.NET Core13.9.303.0 for F# 9.0
Gogo1.25.0
Go/gccgo14.2.0
Haskell9.10.2
Idris 20.7.0
Java24.0.2
Juliav"1.11.6"
Kotlin2.2.0
Lua5.4.7
Lua/luajit2.1.1748495995
MLton20241230
Nim2.2.4
Node.jsv24.5.0
OCaml5.3.0
Odindev-2025-08-nightly
PHP8.4.11
Perlv5.40.1
Python3.13.5
Python/pypy7.3.20-final0 for Python 3.11.13
Racket"8.17"
Ruby3.4.5p51
Ruby/jruby10.0.2.0
Ruby/truffleruby24.2.2
Rust1.89.0
Scala3.7.2
Swift6.1.2
Tcl8.6
V0.4.11 b9e5757
Vala0.56.18
Zig0.14.1
clang/clang++19.1.7 (3+b1)
gcc/g++14.2.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where <cmd> is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);
  • primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a ./run.sh that could be used to simplify Docker usage:

  • ./run.sh build (build the image);
  • ./run.sh make versions (run the image with the versions command);
  • sudo ./run.sh shell (run the image with the `shell' command, sudo is required to read energy levels).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

  • JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.

README update

TOC is regenerated using git-markdown-toc:

./run.sh toc