Benchmarks

April 22, 2026 ยท View on GitHub

We all love benchmarks but we know it's difficult to do fair comparisons. That's why it's important to be very transparent about the protocol of the benchmark.

Crawler Benchmark

Methodology

The benchmark uses Amiibo demo website. The program will crawl the first page and then all the subpages to retrieve 933 URLs in total.

The crawler program is written in Go, is uses chromedp lib to control the browsers. The source code is available in chromedp/crawler/.

This benchmark will get the pages through internet connexion.

Measuring memory and CPU usage is not easy. Chrome uses many threads. We use smem for memory which take shared memory pages into account. We use ps for CPU utilization.

We run the follwing script with watch -n0.1 during the benchmark to get stats every 100ms. And we take the peak memory with its CPU percent in account.

#!/bin/sh

PROGRAM=\$1

MEM=`smem -c "pss" -P "^$PROGRAM" -t -k | tail -1`
CPU=`ps aux | grep $PROGRAM | grep -v grep | awk '{sum+=\$3} END {print "Total CPU: " sum "%"}'`

echo "$MEM\t$CPU"

Preparation

Test machine

The tests are run in an AWS m5.xlarge (x86_64) with a fresh Ubuntu install.

AWS m5.xlarge (x86_64) neofetch

Browsers

We use Google Chrome version 143.0.7499.169

$ google-chrome-stable --version
Google Chrome 143.0.7499.169

And Lightpanda commit 7b1f157cf8ccbdded7db6ab47e94577495ee87e4.

Crawler

Clone the demo repository.

We use a Go program to crawl the website.

$ cd chromedp
$ go build -o crawler crawler/main.go
$ ./crawler/main https://demo-browser.lightpanda.io/amiibo/

Summary

Benchdurationmemory peak% CPUPages
Lightpanda 1 process0:51.6827.2M11.1%933
Lightpanda 2 process0:29.7931.7M26.6%933
Lightpanda 5 process0:11.7043.9M71.6%933
Lightpanda 10 process0:06.7663.7M135.3%933
Lightpanda 25 process0:04.81123.0M204.7%933
Lightpanda 100 process0:05.23410.2M202.8%933
Chrome 1 tab1:22.831.3G124.9%933
Chrome 2 tabs0:53.111.3G202.3%933
Chrome 5 tabs0:45.661.6G237%933
Chrome 10 tabs0:45.621.7G241.6%933
Chrome 25 tabs0:46.702.0G254%933
Chrome 100 tabs1:09:374.2G229%933

memory usage comparison

Single tab/process

Run the crawler with one tab for Chrome and one process for Lightpanda.

1 process memory usage comparison 1 process cpu usage comparison

Chrome

Start Chrome on port 9222. It displays the websocket connection URL to use.

$ rm -fr /tmp/bench_chrome; \
        /opt/google/chrome/chrome --headless=new --remote-debugging-port=9222 --user-data-dir=/tmp/bench_chrome

$ /usr/bin/time -v ./crawler/main --pool 1 --cdp ws://127.0.0.1:9222/devtools/browser/46425034-faf3-40e4-8b00-55224d96ecc2  https://demo-browser.lightpanda.io/amiibo/

        Command being timed: "./crawler/main --pool 1 --cdp ws://127.0.0.1:9222/devtools/browser/46425034-faf3-40e4-8b00-55224d96ecc2  https://demo-browser.lightpanda.io/amiibo/"
        User time (seconds): 5.89
        System time (seconds): 3.64
        Percent of CPU this job got: 11%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:22.83
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 15852
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 5973
        Voluntary context switches: 293725
        Involuntary context switches: 39304
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Lightpanda

$ ./lightpanda serve

$ /usr/bin/time -v ./crawler/main --pool 1 https://demo-browser.lightpanda.io/amiibo/

        Command being timed: "./crawler/main --pool 1 https://demo-browser.lightpanda.io/amiibo/"
        User time (seconds): 2.12
        System time (seconds): 1.12
        Percent of CPU this job got: 6%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:51.68
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 15388
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 67
        Minor (reclaiming a frame) page faults: 4133
        Voluntary context switches: 108859
        Involuntary context switches: 1166
        Swaps: 0
        File system inputs: 15152
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Multi tabs/processes

Run the crawler with multiple tabs for Chrome and multiple processes for Lightpanda.

Chrome with 25 tabs and Lightpanda with 25 processes

25 process memory usage comparison 25 process cpu usage comparison

Chrome with 100 tabs and Lightpanda with 100 processes

100 process memory usage comparison 100 processes cpu usage comparison

Chrome

Start Chrome on port 9222. It displays the websocket connection URL to use.

$ rm -fr /tmp/bench_chrome; \
        /opt/google/chrome/chrome --headless=new \
        --remote-debugging-port=9222 --user-data-dir=/tmp/bench_chrome

$ /usr/bin/time -v ./crawler/main --pool 100 \
        --cdp ws://127.0.0.1:9222/devtools/browser/4443644c-c476-4597-bd59-cf7ad38ec226 \
        https://demo-browser.lightpanda.io/amiibo/

        Command being timed: "./crawler/main --pool 100 --cdp ws://127.0.0.1:9222/devtools/browser/4443644c-c476-4597-bd59-cf7ad38ec226 https://demo-browser.lightpanda.io/amiibo/"
        User time (seconds): 9.13
        System time (seconds): 5.07
        Percent of CPU this job got: 20%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:09.37
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 44884
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 12816
        Voluntary context switches: 390124
        Involuntary context switches: 91106
        Swaps: 0
        File system inputs: 0
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Lightpanda

Lightpanda can't create multi-tabs. So instead we start 100 browser process, each on it's own port. The crawler program has -fork and -lpd-path options to enable this mode.

$ /usr/bin/time -v ./crawler/main --pool 100 --fork \
        --lpd-path ../../lightpanda \
        https://demo-browser.lightpanda.io/amiibo/

        Command being timed: "./crawler/main --pool 100 --fork --lpd-path ../../lightpanda https://demo-browser.lightpanda.io/amiibo/"
        User time (seconds): 11.80
        System time (seconds): 2.17
        Percent of CPU this job got: 266%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:05.23
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 51400
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 2
        Minor (reclaiming a frame) page faults: 253270
        Voluntary context switches: 31865
        Involuntary context switches: 16069
        Swaps: 0
        File system inputs: 48
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Campfire e-commerce Benchmark

Methodology

The benchmark uses a homemade demo web page. This web page is a fake e-commerce product offer page loading product details and reviews in JSON with two XHR requests.

We decided to use a homemade page because Lightpanda browser is not yet fully compliant and we wanted to be sure it would be able to execute the web page correctly to be comparable with Google Chrome.

Moreover, using this web page allows us to run the test with a local web server, reducing network request impacts to the bench.

Metrics and tools

We compare now multiple page loads and js evaluations using Puppeteer, which connects to the browser using CDP (Chrome Debug Protocol).

We run the follwing script with watch -n0.1 during the benchmark to get stats every 100ms. And we take the peak memory with its CPU percent in account.

#!/bin/sh

PROGRAM=\$1

MEM=`smem -c "pss" -P "^$PROGRAM" -t -k | tail -1`
CPU=`ps aux | grep $PROGRAM | grep -v grep | awk '{sum+=\$3} END {print "Total CPU: " sum "%"}'`

echo "$MEM\t$CPU"

Preparation

Dependencies

To run the benchmark, you need to install nodejs.

Once nodejs is installed, please run a npm install to install nodejs dependencies, mainly Puppeteer.

You have also to install Google Chrome and Lightpanda browser.

Demo web page

Clone the demo web page and expose the public/ directory locally with a web server.

We use the simple Go program to expose the public/ dir. By default it exposes the public dir using the 1234 port.

$ go run runner/main.go -serve

Test machine

The tests are run in an AWS m5.xlarge (x86_64) with a fresh Ubuntu install.

AWS m5.xlarge (x86_64) neofetch

Running the benchmark

The puppeteer/cdp.js benchmark accepts multiple env vars to be configured.

  • BROWSER_ADDRESS is the address of the running browser listening the CDP protocol, by default ws://127.0.0.1:9222.
  • BASE_URL is the base url of the running web reser to request, by default http://127.0.0.1:1234.
  • RUNS is the number of pages loaded by the benchmark, default is 100.

npm run bench-puppeteer-cdp starts a Puppeteer process instance and load the page to extract data 100 times.

$ npm run bench-puppeteer-cdp

Results

Google Chrome

We use Google Chrome version 143.0.7499.109

You have to start the browser first.

$ google-chrome --headless=new --remote-debugging-port=9222

Then you can run the benchmark.

$ BROWSER_ADDRESS=http://127.0.0.1:9222 npm run bench-puppeteer-cdp

> demo@1.0.0 bench-puppeteer-cdp
> node puppeteer/cdp.js

................................................................................
....................
total runs 100
total duration (ms) 18551
avg run duration (ms) 185
min run duration (ms) 164
max run duration (ms) 205

$ watch -n0.1 "./stat.sh chrome |tee -a chrome"
# peak is
402.1M        Total CPU: 158.6%

Lightpanda browser

We use Lightpanda commit 7b1f157cf8ccbdded7db6ab47e94577495ee87e4.

You have to start Lightpanda browser.

$ ./lightpanda serve

Then you can run the benchmark.

$ npm run bench-puppeteer-cdp

> demo@1.0.0 bench-puppeteer-cdp
> node puppeteer/cdp.js

.................................................................................
...................
total runs 100
total duration (ms) 1698
avg run duration (ms) 16
min run duration (ms) 12
max run duration (ms) 47

$ watch -n0.1 "./stats.sh lightpanda |tee -a lightpanda.stat"
# peak is
21.2M        Total CPU: 4.6%