Criterium 0.5.x

The goal is to make criterium more data driven, and usable in more contexts.

Status

Early alpha. Breaking changes will be made.

Quick Start

Installation

Add criterium as an alias in your deps.edn:

{:aliases
 {:bench {:extra-deps {org.hugoduncan/criterium {:mvn/version "0.5.153-ALPHA"}}
          ;; JDK 17+ options for optimal dead-code elimination
          :jvm-opts ["-XX:+UnlockExperimentalVMOptions"
                     "-XX:CompileCommand=blackhole,criterium.blackhole.Blackhole::consume"]}}}

For benchmarks with generated arguments, also add the arg-gen library:

{:aliases
 {:bench {:extra-deps {org.hugoduncan/criterium {:mvn/version "0.5.153-ALPHA"}
                       org.hugoduncan/criterium.arg-gen {:mvn/version "0.5.153-ALPHA"}}
          :jvm-opts ["-XX:+UnlockExperimentalVMOptions"
                     "-XX:CompileCommand=blackhole,criterium.blackhole.Blackhole::consume"]}}}

To time an expression use criterium.bench/bench.

(require '[criterium.bench :as bench])
(bench/bench (criterium.jvm/wait 10000)) ; 10us busy wait

If you are using portal, try:

(require 'criterium.viewer.portal)
(require '[criterium.bench :as bench])
(bench/bench (criterium.jvm/wait 10000)
  :viewer :portal
  :bench-plan criterium.bench-plans/histogram)

New features

• improved accuracy on very fast functions.
• charting via portal
• thread memory allocation collection
• agent to allow capture of all allocations during the execution of an expression.

Using the Native Agent

The native agent provides detailed allocation tracking. Due to JVMTI limitations, it must be loaded at JVM startup:

# Get the agent path
clojure -Sdeps '{:deps {org.hugoduncan/criterium {:mvn/version "0.5.153-ALPHA"}}}' \
  -e '(require '"'"'[criterium.agent :as agent]) (println (first (agent/jvm-opts)))'

# Restart with the agent (use the path from above)
clojure -J-agentpath:/tmp/criterium-agent-macos-x64-abc123.dylib -M:dev

In your REPL:

(require '[criterium.agent :as agent])

;; Verify agent is loaded
(agent/loaded?)
;; => true

;; Benchmarks now include allocation tracking
(require '[criterium.bench :as bench])
(bench/bench (reduce + (range 1000)))

See projects/agent/README.md for complete documentation.

Argument Generation

The criterium.arg-gen namespace (in the separate org.hugoduncan/criterium.arg-gen artifact) provides benchmarks with generated arguments using test.check generators. This is useful for benchmarking functions with realistic, varied inputs.

(require '[criterium.arg-gen :as arg-gen])
(require '[clojure.test.check.generators :as gen])
(require '[criterium.bench :as bench])

;; Benchmark with generated arguments
(bench/bench-measured
  (arg-gen/measured
    [a gen/large-integer
     b gen/large-integer]
    (+ a b)))

;; With options for size and reproducibility
(bench/bench-measured
  (arg-gen/measured {:size 50 :seed 12345}
    [coll (gen/vector gen/small-integer)]
    (reduce + coll)))

The measured macro uses let-style bindings where each right-hand side is a test.check generator. The body is measured with fresh generated values for each sample.

Design

The bench cli is a thin layer over the following.

Criterium separates metrics collection from the processing of the metrics to analyse and display them.

We provide multiple ways to collect metrics; running a collection plan, instrumenting a function, or triggering collection on events. You can also provide metrics by any other means you like.

Benchmarks are pipelines that can be run with different reporting targets.

Metrics Collection

Controlled sampling is based on the concept of a "measured". This consists of a pair of functions, a state constructor, used to capture arguments, and a measure function, that takes the output of the state constructor as an argument, together with an evaluation count, runs the benchmark subject the given number of times, and returns the total time taken.

This design addresses both time resolution and argument capture concerns.

Other ways of sampling are provided. You can instrument a function to collect samples, or can use a trigger which collects samples based on deltas between successive firing of the trigger.

Benchmark

Once samples have been collected they are passed to the benchmark, which provides analysis and viewing.

Analysis

The analysis is based on a set of analysis functions that read and write paths in a data map. Each analysis reads and writes default paths in the map, but these can be explicitly specified.

View

Viewing is based on a set of viewing function that read paths from the data map. Each view uses default paths, but these can be explicitly specified.

The built in views support different viewers. The default viewer, :print, prints results in human readable format. There are also :pprint and :portal viewers.

The :portal viewer is capable of displaying charts.

Development Plan

Features that will probably get added:

• add KDE to improve estimation of the PDF.
• add bi-modal detection, probably based on KDE.
• add a percentile sampling mode based oh HDRHistogram
• add charting and regressions around change in metrics with a varied parameter
• add charting to compare alternate implementations of expressions
• add fitting of different distributions (e.g. hyperbolic secant)