Performance Test Result and Comparison

November 13, 2024 ยท View on GitHub

Quark Container is a secure container runtime with OCI interface. There are 2 other open source project provides similar functions:

  1. Gvisor
  2. Kata Containers

Here we compare the startup time and memory overhead of container runtime.

  1. Startup time: Test how much overhead time will take for container runtime between start a container application and the first set of system calls are executed by the application. We are using date to do such test. The test script is as below.
date +%s%N; docker run --rm -it ubuntu /bin/date +%s%N
date +%s%N; docker run --runtime=quark --rm -it ubuntu /bin/date +%s%N
date +%s%N; docker run --runtime=runsc  --rm -it ubuntu /bin/date +%s%N
date +%s%N; docker run --runtime=kata-runtime --rm -it ubuntu /bin/date +%s%N
RuncQuarkgVisorKata
Startup (ms)6076257081747
  1. Memory usage overhead: Test how much memory overhead the container runtime will consume. We are using busybox to test.

We used "busybox" to measure the memory usage of container runtime. The test script is as below. Note: the memory usage only measure the processes other than docker run -it --name some_busybox.

docker run -it --name some_busybox --rm busybox
docker run -it --name some_busybox --runtime=quark --rm busybox
docker run -it --name some_busybox --runtime=runsc  --rm busybox
docker run -it --name some_busybox --runtime=kata-runtime --rm busybox
RuncQuarkgVisorKata
Memory Overhead (MB)011.828.1184.3

We also did performance test and comparison based on some popular open source service benchmarks.

  1. Etcd3.0: The test is based on ETCD benchmark.

The etcd startup script is as below.

"sudo docker run -p 2379:2379 -p 2380:2380 --rm --name etcd0 quay.io/coreos/etcd:v3.0.0 /usr/local/bin/etcd --name my-etcd-1 --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://0.0.0.0:2379 \
                --listen-peer-urls http://0.0.0.0:2380 \
                --initial-advertise-peer-urls http://0.0.0.0:2380 \
                --initial-cluster my-etcd-1=http://0.0.0.0:2380 \
                --initial-cluster-token my-etcd-token \
                --initial-cluster-state new"
"sudo docker run  -p 2379:2379 -p 2380:2380 --runtime=quark --rm --name etcd0 quay.io/coreos/etcd:v3.0.0 /usr/local/bin/etcd --name my-etcd-1 --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://0.0.0.0:2379 \
                --listen-peer-urls http://0.0.0.0:2380 \
                --initial-advertise-peer-urls http://0.0.0.0:2380 \
                --initial-cluster my-etcd-1=http://0.0.0.0:2380 \
                --initial-cluster-token my-etcd-token \
                --initial-cluster-state new"
"sudo docker run --runtime=runsc -p 2379:2379 -p 2380:2380 --rm --name etcd0 quay.io/coreos/etcd:v3.0.0 /usr/local/bin/etcd --name my-etcd-1 --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://0.0.0.0:2379 \
                --listen-peer-urls http://0.0.0.0:2380 \
                --initial-advertise-peer-urls http://0.0.0.0:2380 \
                --initial-cluster my-etcd-1=http://0.0.0.0:2380 \
                --initial-cluster-token my-etcd-token \
                --initial-cluster-state new"
"sudo docker run --runtime=kata-runtime -p 2379:2379 -p 2380:2380 --rm --name etcd0 quay.io/coreos/etcd:v3.0.0 /usr/local/bin/etcd --name my-etcd-1 --listen-client-urls http://0.0.0.0:2379 \
                --advertise-client-urls http://0.0.0.0:2379 \
                --listen-peer-urls http://0.0.0.0:2380 \
                --initial-advertise-peer-urls http://0.0.0.0:2380 \
                --initial-cluster my-etcd-1=http://0.0.0.0:2380 \
                --initial-cluster-token my-etcd-token \
                --initial-cluster-state new"

The benchmark script is as below.

./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  put --key-size=8 --sequential-keys --total=10000 --val-size=256
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  lease-keepalive --total=10000
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  mvcc put --total=10000
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  range 123 1234567890 --total=10000
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  stm --total=10000
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  txn-put --total=10000
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  watch
./benchmark --endpoints=http://localhost:2379 --target-leader --conns=10 --clients=10  watch-get

The result is as below.

RuncQuarkgVisorKata
Put374134712408769
lease-keepalive17883680256474982
mvcc Put12737798250105232101256
range785651171837932
stm7877354423792059
txn-put493333592189746
watch24368811621613468628035
watch-get3418194110721774
  1. Redis The Redis startup scripts are as below.
docker run -p 6379:6379  --name some-redis --rm  -it redis
docker run -p 6379:6379  --runtime=quark --name some-redis --rm  -it redis
docker run -p 6379:6379  --runtime=runsc --name some-redis --rm  -it redis
docker run -p 6379:6379  --runtime=kata-runtime --name some-redis --rm  -it redis

The benchmark execution scripts are as below.

redis-benchmark -n 100000 -c 20

Test result is as below.

RuncQuarkgVisorKata
PING_INLINE3847636791820431446
PING_BULK3757938240848133602
SET3859537864834834698
GET3862437425837434048
INCR3852037037831634867
LPUSH3923137509821433840
RPUSH3895537950825932393
LPOP3735537341821232948
RPOP3827038431826934506
SADD3937039354829735310
HSET3954137821820533579
SPOP3734137878846734782
LPUSH3937037921820633211
LRANGE_1002517724142695524703
LRANGE_3001315913504501412980
LRANGE_5009090957342319044
LRANGE_6008328793733847176
MSET3840237707788430147
  1. Nginx: We use Nginx to test http get performance

The Nginx startup script is as below.

docker run --name some-nginx -it --runtime=runc --rm -p 80:80  --rm -v /home/brad/website/:/usr/share/nginx/html:ro nginx
docker run --name some-nginx -it --runtime=quark --rm -p 80:80  --rm -v /home/brad/website/:/usr/share/nginx/html:ro nginx
docker run --name some-nginx -it --runtime=runsc --rm -p 80:80  --rm -v /home/brad/website/:/usr/share/nginx/html:ro nginx
docker run --name some-nginx -it --runtime=kata-runtime --rm -p 80:80  --rm -v /home/brad/website/:/usr/share/nginx/html:ro nginx

The index.html is this

The benchmark script is as below.

ab -n 10000 -c 10 http://localhost/index.html

The test is as below.

RuncQuarkgVisorKata
RPS (Request Per Second)61362274.741296.7928.96
  1. dd: We use dd to test disk write performance

The test script is as below.

docker run -P --mount type=bind,source="/home/brad/rust/quark/test",target=/test --rm -it ubuntu /bin/dd if=/dev/zero of=/test/fio-rand-read bs=4k count=2500
docker run -P --runtime=quark --mount type=bind,source="/home/brad/rust/quark/test",target=/test --rm -it ubuntu /bin/dd if=/dev/zero of=/test/fio-rand-read bs=4k count=2500
docker run -P --runtime=runsc --mount type=bind,source="/home/brad/rust/quark/test",target=/test --rm -it ubuntu /bin/dd if=/dev/zero of=/test/fio-rand-read bs=4k count=2500
docker run -P --runtime=kata-runtime --mount type=bind,source="/home/brad/rust/quark/test",target=/test --rm -it ubuntu /bin/dd if=/dev/zero of=/test/fio-rand-read bs=4k count=2500

The test result is as below.

RuncQuarkgVisorKata
MB/Sec39717930.5103
  1. MariaDB initialization time: MariaDB's initialization is complex as it includes much IO/CPU/Memory operation. So we use as benchmark.

The test script is as below.

sudo docker run --net=host --rm --name some-mariadb -e MYSQL_ROOT_PASSWORD=123 -it mariadb
sudo docker run --net=host  --runtime=quark  --rm --name some-mariadb -e MYSQL_ROOT_PASSWORD=123 -it mariadb
sudo docker run --runtime=runsc  --rm --name some-mariadb -e MYSQL_ROOT_PASSWORD=123 -it mariadb
sudo docker run --runtime=kata-runtime --rm --name some-mariadb -e MYSQL_ROOT_PASSWORD=123 -it mariadb

The test result is as below.

RuncQuarkgVisorKata
Sec8101412
  1. Mysql initialization time: We also measure the MySql initialization time.
sudo docker run -P --runtime=runc  --rm --name some-mysql -e MYSQL_ROOT_PASSWORD=123 -it mysql
sudo docker run -P --runtime=quark  --rm --name some-mysql -e MYSQL_ROOT_PASSWORD=123 -it mysql
sudo docker run -P --runtime=runsc --rm --name some-mysql -e MYSQL_ROOT_PASSWORD=123 -it mysql
sudo docker run -P --runtime=kata-runtime --rm --name some-mysql -e MYSQL_ROOT_PASSWORD=123 -it mysql

The test result is as below. The Kata can't run mysql with error "ERROR 14 (HY000) at line 147118: Can't change size of file (OS errno 2 - No such file or directory)"

RuncQuarkgVisorKata
Sec182032N/A