CleitonForge

[English] | Português

CleitonForge measures what a test is able to detect, and uses that to find faults in quantum compilers and simulators. Written in Rust; the CLI binary is cforge.

The order is the point. Most tools report what they found. This one first measures what it could have found, by injecting faults on purpose and counting which of them the oracle catches. A campaign that reports nothing is only worth reading if you know what it would have caught, and a campaign that reports something is only worth trusting if you know what it misses.

Two things have come out of that so far.

A soundness bug in Qiskit's transpiler, issue #16594: the CommutativeCancellation pass cancelled sxdg sxdg sx, which is an X, down to an empty circuit at optimization level 2 and above. No error, no warning, and the measured result changes. A core maintainer confirmed it and it was fixed in Qiskit 2.5.1.

And a result about quantum error correction, written up as a preprint with a DOI, 10.5281/zenodo.22802524: the detector-based oracles that circuit transformations in QEC are usually checked against are not weak but empty in the regime where a compiler pass has to be validated. Over 335 mutants that provably change what a circuit computes, they detect none. Two negative controls, equivalent on every trial with no false positive, are what give that number its credit.

The differential fuzzer described below is how the measuring gets done.

---

The problem

Simulators and compilers are hard to test because there is usually nothing to compare an answer against. A wrong amplitude is still a well formed amplitude. Nothing raises, nothing logs, and the number you get back looks exactly like a correct one. You only notice when you already know what the answer should be, which is the case you did not need the tool for.

Differential testing gets around this. Instead of asking "is this right", it asks "do these two agree". Two implementations of the same specification should return the same state for the same circuit. When they do not, one of them is wrong, and the disagreement is the signal.

That is the whole idea. The rest is making it precise enough to be worth reading:

• The generator aims at the numeric thresholds a compiler branches on: multiples of pi/2, angles just under a synthesis cutoff, sequences that collapse to identity. Uniform random angles almost never land on those.
• The oracle compares modulo global phase, because global phase is not observable and flagging it would bury the real findings under noise. The harnesses for external compilers can hand the verdict to MQT QCEC instead, which reasons symbolically rather than simulating.
• The shrinker reduces a failure to a 1-minimal witness, meaning removing any single gate makes the bug disappear. #16594 came out of it as three gates on one qubit.
• The triage cross-checks every verdict against the exact operator before calling anything a bug, and tells a known class from a new one by accumulated rotation instead of by gate name.

The hit rate, honestly

Roughly one real bug per hundred thousand circuits. These stacks are mature and well tested, and most disagreements turn out to be a declared transformation that the oracle had not been told about, not a fault. Campaigns against tket, PennyLane and Cirq came back clean.

So the value here is not volume. It is that the findings which survive triage are real, minimal, and reproducible by someone who does not trust you. They live in bug-zoo/, one JSON file each, with the gate list, an OpenQASM 2 reproducer and a record of how it was found. The backend divergences also carry the generator seed and the distance at each oracle level.

Why Rust

• The generator, the state vector and the shrinker are all in the hot path of a campaign that runs tens of thousands of circuits.
• Qiskit's transpiler is itself moving to Rust, and the bug above is in Rust code. It helps to read the language of the thing you are fuzzing.
• For the benchmarking side, predictable low overhead matters, because measurement noise is what is being measured.

The benchmarking side

CleitonForge started as a cross-backend benchmarking layer, and that part still works and is documented below. cforge run executes the same OpenQASM file on more than one backend and reports time, memory, depth and fidelity side by side. It is now a component of the fuzzer rather than the point of the project.

Two rules kept from that period, because they are what makes the comparison mean anything:

• Never optimized to favor a specific simulator.
• Plugin architecture (SimulationBackend trait), so a backend can be added without touching the core IR.

---

Architecture

CleitonForge/                    Rust workspace
├── cforge-core/                 Canonical IR: Circuit, GateKind, Operation
├── cforge-parser/               OpenQASM 2 + OpenQASM 3 parsers
├── cforge-backends/             SimulationBackend trait + implementations
├── cforge-metrics/              Fidelity, depth, timing, memory measurement
├── cforge-fuzz/                 The fuzzer
│   ├── generator.rs             Weighted circuit generator
│   ├── oracle.rs                Divergence oracles (N1 amplitude, N2 probability)
│   ├── shrinker.rs              Greedy reduction to a 1-minimal witness
│   ├── triage.rs                Known class vs new finding
│   └── zoo.rs                   JSON output for bug-zoo/
├── cforge-cli/                  `cforge` binary (clap + comfy-table)
│   └── examples/
│       └── compare_grover.rs   Grover algorithm, pure Rust API example
├── cforge-py/                   Python bindings (PyO3)
├── tools/                       Harnesses for external compilers
│   ├── fuzz_qiskit*.py          Qiskit: full pipeline, isolated passes, wide
│   ├── fuzz_pytket*.py          tket
│   ├── fuzz_pennylane*.py       PennyLane devices and transforms
│   ├── fuzz_cirq.py             Cirq
│   ├── oracle_qcec.py           MQT QCEC as a three way verdict
│   └── triage_known.py          Triage by accumulated rotation
├── bug-zoo/                     Minimized counterexamples, one JSON each
└── examples/
    └── bell.qasm               Bell state in OpenQASM 2

Data flow

.qasm file
    │
    ▼
cforge-parser  ──►  Circuit (canonical IR)
                         │
              ┌──────────┴──────────┐
              ▼                     ▼
  NativeStateVector          QuantRS2Backend
     Backend                       │
              └──────────┬──────────┘
                         ▼
              cforge-metrics: fidelity, time, memory
                         │
                         ▼
              cforge-cli: table / JSON output

---

Quick Start

Prerequisites

• Rust 1.96+ (rustup update stable)
• No external simulators required. All dependencies are pure Rust crates.

Build

git clone https://github.com/cleitonaugusto/CleitonForge.git
cd CleitonForge
cargo build --release

The binary lands at target/release/cforge. The commands below call it by that path so they work straight after the build. To type just cforge anywhere, install it onto your PATH:

cargo install --path cforge-cli

Run a circuit on both backends

./target/release/cforge run --circuit examples/bell.qasm --backends statevector,quantrs2 --shots 1024

Circuit: 2 qubits  |  2 gates  |  depth 2  |  seed 0xdeadbeefcafebabe
┌──────────────────────┬───────────┬────────┬───────┬───────┬──────────┬───────┐
│ Backend              │ Time (ms) │ Memory │ Depth │ Gates │ Fidelity │ Shots │
╞══════════════════════╪═══════════╪════════╪═══════╪═══════╪══════════╪═══════╡
│ statevector-native   │ 0.002     │ 64 B   │ 2     │ 2     │ 1.000000 │ 1024  │
├──────────────────────┼───────────┼────────┼───────┼───────┼──────────┼───────┤
│ statevector-quantrs2 │ 0.004     │ 64 B   │ 2     │ 2     │ 1.000000 │ 1024  │
└──────────────────────┴───────────┴────────┴───────┴───────┴──────────┴───────┘

Run the benchmark suite as a CI gate

Six standard algorithms, each checked against a known expected outcome and, with two or more backends, against each other by state fidelity. Exits 1 if anything fails, so it drops straight into a pipeline step:

./target/release/cforge bench --backends statevector,quantrs2
./target/release/cforge bench --format json > fidelity-report.json

The QAOA MaxCut case reports a fidelity of ~0 and still passes. That is the documented Rz sign divergence between the two backends: the disagreement is the result, not a regression, so it is shown and not gated. Everything else is gated at --threshold (0.9999 by default).

Validate a circuit without simulation

./target/release/cforge validate --circuit examples/bell.qasm

File   : examples/bell.qasm
Qubits : 2
Gates  : 2
Depth  : 2
By gate:
  cx       1
  h        1
Status : OK

Export results as JSON (for scripts / CI)

./target/release/cforge run --circuit examples/bell.qasm --shots 1024 --format json

{
  "circuit": { "file": "examples/bell.qasm", "qubits": 2, "gates": 2, "depth": 2 },
  "shots": 1024,
  "seed": 16935479246996842942,
  "results": [
    {
      "backend": "statevector-native",
      "time_ms": 0.456,
      "memory_bytes": 64,
      "depth": 2,
      "gates": 2,
      "fidelity": 1.0,
      "shots": 1024,
      "error": null
    }
  ]
}

Grover's algorithm end-to-end example

3-qubit Grover search for |101⟩, 2 iterations, both backends:

cargo run --release --example compare_grover -p cforge-cli

Target state : |101⟩  (index 5)
Circuit      : 43 gates  |  depth 21

Backend  : statevector-native
  Top state  : |101⟩  prob = 0.9453 (94.5 %)   [theory: sin²(5θ) ≈ 94.8 %]
  Fidelity   : 1.00000000

Cross-backend fidelity (native vs quantrs2): 1.00000000
Both backends agree: YES ✓

---

CLI Reference

cforge run
  --circuit <path>          OpenQASM 2 or 3 file (auto-detected)
  --backends <list>         Comma-separated: statevector, quantrs2  [default: both]
  --shots <n>               Measurement shots; 0 = statevector only  [default: 0]
  --seed <u64>              PRNG seed for reproducible counts  [default: 0xdeadbeef_cafebabe]
  --format <table|json>     Output format  [default: table]

cforge validate
  --circuit <path>          Parse and report circuit statistics; exit 1 if invalid

---

Supported Gate Set

CleitonForge implements the full OpenQASM 3 stdgates.inc gate set:

Category	Gates
Single-qubit, no params	id x y z h s sdg t tdg sx sxdg
Single-qubit, parametric	rx(θ) ry(θ) rz(θ) p(θ) u(θ,φ,λ)
Two-qubit	cx cy cz ch csx crx cry crz cp cu swap
Three-qubit	ccx (Toffoli) cswap (Fredkin)
Aliases	cnot→cx u1→p u2(φ,λ)→u(π/2,φ,λ) u3→u ccnot→ccx fredkin→cswap

---

Supported Input Formats

Format	Auto-detected by	Notes
OpenQASM 2.0	everything else	include resolved from file directory; no stdgates disk file needed
OpenQASM 3	OPENQASM 3 header	Parses stdgates.inc gate calls via oq3_semantics

Both parsers support whole-register gate application: h q; on a 3-qubit register expands to h q[0]; h q[1]; h q[2];

---

Rust API

use cforge_core::{Circuit, GateKind, Operation};
use cforge_backends::{DEFAULT_SEED, NativeStateVectorBackend, SimulationBackend};
use cforge_metrics::{compute_stats, measure};

// Build a Bell state circuit
let mut circuit = Circuit::new(2);
circuit.push(Operation::new(GateKind::H,  vec![0], vec![]));
circuit.push(Operation::new(GateKind::Cx, vec![0, 1], vec![]));

// Run and measure
let stats = compute_stats(&circuit);
let result = NativeStateVectorBackend.run(&circuit, 1024, DEFAULT_SEED)?;

println!("depth = {}", stats.depth);               // 2
println!("|00⟩ prob = {:.3}", result.statevector[0].norm_sqr()); // 0.500

---

Adding a New Backend

Implement the SimulationBackend trait (one method):

use cforge_backends::{BackendError, SimulationBackend, SimulationResult};
use cforge_core::Circuit;

pub struct MyBackend;

impl SimulationBackend for MyBackend {
    fn name(&self) -> &str { "my-backend" }

    fn run(
        &self,
        circuit: &Circuit,
        shots: usize,
        seed: u64,
    ) -> Result<SimulationResult, BackendError> {
        // ... your simulation logic
    }
}

Then pass --backends my-backend to the CLI after registering it in cforge-cli/src/main.rs.

---

Memory Measurement

For statevector simulations the peak memory is:

$ 2^\text{n\_qubits} \times 16 \text{bytes} (\text{two} \text{f64} \text{per} \text{Complex128} \text{amplitude}) $

On Linux, CleitonForge measures the actual RSS delta via /proc/self/status while the statevector is live. For circuits smaller than one OS page (4 KiB) the theoretical value is used instead. Both values are included in JSON output.

Qubits	Theoretical peak
10	16 KB
20	16 MB
22	64 MB (max)

---

Status

Phase	Crate	Status
0	workspace setup	✓
1	cforge-core	✓
2	cforge-parser	✓
3	cforge-backends	✓
4	cforge-metrics	✓
5	cforge-cli	✓
6	examples + docs	✓
7	cforge-py	✓
8	cforge-fuzz	✓

Python bindings are published: pip install cleitonforge.

Planned: additional backends (qoqo, q1tsim), noise modeling, extended OpenQASM 3 gate coverage.

---

Contributing

Issues and pull requests are welcome. The architecture is intentionally modular. Adding a backend, a new metric, or a new input format does not require touching the core IR.

Run the test suite:

cargo test --workspace
cargo clippy --workspace

The books

Two books come out of this work. They are not documentation — the README and cforge --help cover that. They are the long form: how the tooling was built, and what the measurements actually say.

Quantum Computing for Rust Developers builds the whole thing from nothing in Rust. A statevector simulator to 25 qubits, an OpenQASM 2 and 3 parser that handles circuits exported from real hardware, Grover and QAOA verified rather than assumed, and the fuzzer itself: generator, oracle, shrinker, triage. The chapter on finding faults in real compilers uses the issue numbers above, including the one where I got the root cause wrong and was corrected in public. The campaigns that came back clean are in there too, with the mutation testing that says what they could have caught.

• Amazon — paperback and Kindle
• Leanpub — PDF and EPUB

Quantum Circuit Benchmarking in Rust is the short one. It is the field report of the Rz sign convention that four Rust simulators did not agree on, and of why a benchmark scored on measurement counts cannot see that class of fault at all.

• Leanpub — PDF and EPUB

If you read one and it was useful, an honest review is the thing that helps most. Neither book has a publisher behind it.

Sponsoring

CleitonForge is developed and maintained by a solo independent researcher. If this tool saved you time or helped you catch a bug, consider sponsoring:

➜ github.com/sponsors/cleitonaugusto

---

License

Licensed under the Apache License, Version 2.0. See LICENSE.