Benchmark

November 11, 2022 ยท View on GitHub

The document is the performance test report for EaseProbe.

1. Test Environment

  • OS: Ubuntu 20.04 LTS

  • VM: AWS EC2 C5.2xlarge - 8 vCPUs, 16 GiB RAM, EBS Volume - 100 GiB

  • Config: Everything is default configured. only enlarge the max open file limit.

    • in /etc/sysctl.conf, add

      fs.file-max = 1000000

    • in /etc/security/limits.conf, add

      * hard nofile 1000000
* soft nofile 1000000

session required pam_limits.so

    • DNS server is 8.8.8.8

      Note: the DNS server 1.1.1.1 would be a problem if we query DNS too many, 1.1.1.1 would block the DNS query.

  • Build: the following PRs are needed for this test and we must ensure they are included (already merged in main):

    • PR #157 - Equally distributing start the probers.
      • without this PR, each prober would start to work a same time.
      • with it the prober would start in evenly distribute with in 1 minute
    • PR #159 - Set the SO_LINGER=0 to tcp, http, tls, ssh probe
      • without this PR, EaseProbe leave a TIME_WAIT tcp connection.
      • with this PR, EaseProbe will close the TIME_WAIT tcp connection.
    • PR #162 - Fix the concurrent map access crashing problem
      • without this PR, EaseProbe will crash during the test.
      • with this PR, EaseProbe will not crash during the test.

2. Test Cases

Only ONE Single EaseProbe Process for this performance test.

  • Only test HTTPS and TCP probe
  • with 100, 1000, 2000, 5000 & 10,0000 probers
  • with 5s, 10s, 30s, 60s probe interval
  • With 5 mins running, and check the CPU, Memory, and Network connection.

Using the the following command to observe the performance:

  • top and htop to check the CPU and Memory usage.
  • netstat -nat | awk '{print \$6}' | sort | uniq -c | sort -n to check the Network connection.

3. Test Data

The test data is taken from the top 1M websites (https://majestic.com/reports/majestic-million)

The test data can be found under the EaseProbe Test folder.

4. Test Result

4.1 With 100 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s2.0%50MB0
https10s1.3%50MB0
https30s0.7%45MB0
https60s0.3%45MB0
KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s0.7%30MB0
tcp10s0.7%30MB0
tcp30s0.2%30MB0
tcp60s0.2%30MB0

4.2 With 1,000 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s18.7%105MB0
https10s10.1%98MB0
https30s4.0%75MB0
https60s2.0%75MB0
KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s3.3%105MB0
tcp10s1.3%100MB0
tcp30s0.7%80MB0
tcp60s0.7%79MB0

4.2 With 2,000 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s38.5%183MB0
https10s20.2%150MB0
https30s8.0%125MB0
https60s5.3%114MB0
KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s5.3%165MB0
tcp10s2.7%150MB0
tcp30s1.3%145MB0
tcp60s0.7%142MB0

4.3 With 5,000 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s88.4%580MB0
https10s43.6%380MB0
https30s18.7%270MB0
https60s10.0%254MB0
KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s12.3%380MB0
tcp10s8.7%380MB0
tcp30s2.6%370MB0
tcp60s2.3%350MB0

4.4 With 10,000 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s100.0% *2.5GB *0
https10s88.6%960MB0
https30s30.7%570MB0
https60s17.8%450MB0

Note: https with 5s interval reaches the CPU limitation for all 8 CPU cores, the memory usage is abnormal as well, EaseProbe does not work properly (there are thousands of SYNC_SENT and CLOSE_WAIT ).

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s20.7%800MB0
tcp10s10.3%760MB0
tcp30s3.3%717MB0
tcp60s2.7%492MB0

4.5 With 20,000 Probers

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
https5s---
https10s---
https20s88.8%1.6GB0
https30s58.7%1.3GB0
https60s32.6%1.0GB0

Note: https cannot make the 5s and 10s interval.

KindIntervalCPU UsageMemory Usage (RES)# TIME_WAIT
tcp5s49.7%1.5GB0
tcp10s18.3%1.5GB0
tcp30s4.7%1.5GB0
tcp60s3.4%1.5GB0