CNTI Test Suite CLI Usage Documentation

September 3, 2026 · View on GitHub

Overview

The CNTI Test Suite can be run in production mode (using an executable) or in developer mode (using crystal lang directly). See the pseudo code documentation for examples of how the internals of WIP tests might work.

Syntax for running any of the tests

# Production mode
./cnti-testsuite [options] <task> [<task> ...]

# Developer mode
crystal src/cnti-testsuite.cr -- [options] <task> [<task> ...]

Options are GNU-style and may appear anywhere on the line, as --name VALUE or --name=VALUE; several task names run in order. --skip <task> leaves a task out of a suite. ./cnti-testsuite help lists every option; an unknown option or task is a usage error (exit 64) with the closest match suggested.

./cnti-testsuite cnf_install --cnf-config ./cnti-testsuite.yaml --timeout 120
./cnti-testsuite all --skip resilience --strict
./cnti-testsuite liveness readiness

:star: *Note: All usage commands in this document will use the production (binary executable) syntax unless otherwise stated.

:heavy_check_mark: indicates implemented into stable release
:bulb: indicates Proof of Concept
:memo: indicates To Do
:x: indicates WARNINGS*

Results Output

Every run prints a per-test line to stdout and writes a full results file.

On stdout

:heavy_check_mark: PASSED — the test met best practice; points awarded.
:x: FAILED — the test failed; no points.
⏭ SKIPPED — the test was not executed (a reason is printed); no points.
⏭ N/A — the test does not apply to this CNF (the feature under test is absent); excluded from scoring.
💥 ERROR — the test errored while running.

Failed/errored tests also print indented detail lines beneath the result: > impacted: <resource>, > remediation: <guidance>, and free-form > <note> lines.

Results file

Each run writes a timestamped YAML file into the results directory — cnti/results/ under the current working directory unless redirected (see below):

cnti/results/cnti-testsuite-results-<YYYYMMDD-HHMMSS-mmm>.yml

A new file is created per run, and cnti/results/latest.yml always points at the newest one — a relative symlink, repointed whenever a results file is created (a copy, kept current, on filesystems without symlinks). Scripts should read latest.yml rather than sorting the directory. Every run also ends with one stable line naming both paths:

Results: cnti/results/cnti-testsuite-results-<timestamp>.yml (latest: cnti/results/latest.yml)

To write somewhere else, pass --results-dir PATH on the command line or set the CNTI_TESTSUITE_RESULTS_DIR environment variable; the option wins over the variable. The timestamped file and latest.yml both go there, and delete_results honours the same setting.

A machine-readable JSON Schema describes the file (matching the current schema_version); use it to validate output or generate types.

Structure

name: cnti testsuite
testsuite_version: v1.2.3
schema_version: 1                 # version of this results-file schema
status: failed                    # overall run verdict: passed | failed | error
command: ./cnti-testsuite all
exit_code: 1                      # 0 = run met its objective, 1 = did not, 2 = >=1 test errored
summary:                          # aggregate numbers for the whole run
  total: 18                       # tests executed
  passed: 12
  failed: 2
  skipped: 3
  na: 1
  error: 0
  max_passed: 14                  # maximum tests that could have passed (the denominator)
  essential_passed: 8
  essential_max_passed: 10
  points: 42
  maximum_points: 90
items:
  - name: privileged_containers
    status: failed                # passed | failed | skipped | na | error
    message: Found 2 privileged containers
    type: essential               # the test's declared type (essential | normal | bonus)
    points: 0
    start_time: "2026-07-07T10:00:00.000000000Z"   # RFC 3339
    end_time: "2026-07-07T10:00:03.000000000Z"
    task_runtime: 3.0                              # seconds
    remediation:                  # optional; present only when non-empty
      - Set securityContext.privileged=false on the offending containers
    impacted_resources:           # optional; present only when non-empty
      - kind: Deployment
        name: coredns-coredns
        namespace: cnti-default    # optional (omitted for cluster-scoped resources)
        container: coredns        # optional
        reason: privileged container
  - name: reasonable_startup_time
    status: failed
    message: CNF had a startup time over the limit
    type: normal
    points: 0
    start_time: "2026-07-07T10:00:04.000000000Z"
    end_time: "2026-07-07T10:00:49.000000000Z"
    task_runtime: 45.0
    details:                      # optional; free-form reasons/evidence
      - "CNF had a startup time of 45 seconds (limit: 30 seconds)"

Top-level fields

Field	Description
`name`	Always `cnti testsuite`.
`testsuite_version`	Version of the test suite that produced the file.
`schema_version`	Integer version of this results-file schema; bumped on breaking changes.
`status`	Overall run verdict, derived from `exit_code`: `passed` (0), `failed` (1), `error` (2).
`command`	The command line that produced this file.
`exit_code`	Process exit code, answering "did this run meet its objective?". `2` at least one test errored (raised) - the suite itself broke, which always wins. Otherwise, for a run with a pass criterion (`cert`): `0` the criterion was met, `1` it was not. For a run without one (`all`, `workload`, `platform`): `0` no test failed (passed/skipped/na), `1` at least one test failed. Additionally, the process exits with `64` (usage error) on unknown or malformed command-line arguments, before any test runs. See Exit codes for the full table.
`summary`	Aggregate numbers for the whole run (see below).
`items`	One entry per test that ran (see below).

`summary` fields

All counts/scores are scoped to the tests that actually ran.

Field	Description
`total`	Number of tests executed.
`passed` / `failed` / `skipped` / `na` / `error`	Count of items in each status.
`max_passed`	Maximum number of tests that could have passed (denominator for "X of Y tests passed").
`essential_passed` / `essential_max_passed`	Passed vs. maximum-passable among `essential`-tagged tests.
`points`	Total points scored.
`maximum_points`	Maximum points achievable by the tests that ran.
`criteria`	Success-criterion verdict per task group that ran (see below). Absent when no such group ran.

Test scoring

A test declares everything the suite knows about it at its own definition:

scored_task "sysctls", emoji: "🔓🔑" do |t, args|          # an ordinary test
  ...
end

scored_task "liveness",
  type: CNFManager::TestType::Essential,                  # counts toward cert
  emoji: "⎈🧫" do |t, args|
  ...
end

The type decides the point value — essential 100, normal 5, bonus 1 — so no test restates its own score, and every essential test is by definition a certification test. normal is the default, so only the tests carrying a policy decision say anything about their type. Everything else is read back rather than declared: a test's scope from the namespace it registers in, and its category from the aggregate that runs it. Individual pass:/fail: overrides exist for the rare test that needs one.

There is no points.yml. The suite used to write one into the working directory and read it straight back, so it needed a writable directory to read data it was compiled with, and edits to the file were silently reverted on the next run.

Group success criteria

Every task group — the suites (all, workload, cert) and the categories (security, configuration, ...) — declares a success criterion in Points::GROUP_CRITERIA. These are group-level policy — what the suite certifies, and what it demands of each category — so they live together in one table:

suite_task "cert", [...],
  scope: "essential",     # whose tests count
  min_passed: 15,         # how many of them must pass
  max_failed: CNFManager::NO_FAILURE_LIMIT   # the threshold accounts for failures

A group passes when every threshold it declares holds:

threshold	default	meaning
`min_passed`	`0`	at least this many tests in scope passed
`min_ratio`	none	at least this share of the tests that exist in scope passed
`max_failed`	`0`	at most this many failed; `NO_FAILURE_LIMIT` for no limit

min_ratio counts against how many tests are declared in scope rather than how many a run reached, because excluding tests shrinks the latter — and cert takes an exclude argument, so a ratio over it could be met by running less.

With the defaults — no floor, no ratio, no failures tolerated — a group passes when nothing in it failed, which is what these groups have always demanded. Only cert declares anything else.

Each evaluated group is recorded under summary.criteria in evaluation order:

summary:
  criteria:
    - group: security
      scope: security
      min_passed: 0
      max_failed: 0
      passed_count: 17
      max_passed: 19
      failed_count: 2
      passed: false

A group's dependencies run before its body, so the last entry is the outermost group, and its verdict is what exit_code reflects.

The same verdict is printed to stdout in a stable, greppable form:

Security: FAILED (2 of 19 tests failed)
Certification: PASSED (17 of 19 essential tests passed, threshold 15)

`items[]` fields

Field	Description
`name`	Test name (e.g. `privileged_containers`).
`status`	`passed`, `failed`, `skipped`, `na`, or `error`.
`message`	One-line verdict for the test.
`type`	The test's declared type (`essential`, `normal`, `bonus`).
`points`	Points awarded for this test.
`start_time` / `end_time`	RFC 3339 timestamps for the test's start and end.
`task_runtime`	Test duration in seconds (number).
`details`	(optional) Free-form reason/evidence strings; omitted when empty.
`remediation`	(optional) Guidance on how to fix the failure; omitted when empty. A failed test that provides none of its own carries the Remediation section of its entry in TEST_DOCUMENTATION.
`impacted_resources`	(optional) Structured list of offending resources; omitted when empty.

Each impacted_resources entry has kind and name, plus optional namespace, container, pod, and reason (present only when known).

Logging Parameters

CNTI_TESTSUITE_LOG_LEVEL environment variable: sets minimal log level to display: error (default); info; debug.
CNTI_TESTSUITE_LOG_PATH environment variable: if set - all logs would be appended to the file defined by that variable.

Output streams

Result lines, scores and summaries are the suite's output and go to stdout; log messages are diagnostics and go to stderr (or to the CNTI_TESTSUITE_LOG_PATH file when set). This means ./cnti-testsuite <test> > results.txt 2> debug.log cleanly separates the two even with CNTI_TESTSUITE_LOG_LEVEL=debug. When capturing the streams separately, their relative ordering is not guaranteed — merge them with 2>&1 if strict interleaving matters. Cursor-control progress rewrites and colors are only emitted when stdout is a terminal.

Exit codes

The exit code answers one question: did the invocation do what was asked?

Code	Meaning	When
`0`	Yes	A test run met its objective — for a run with a pass criterion (`cert`) the criterion was met, for any other run no test failed. A setup, install or uninstall command completed.
`1`	No	At least one test failed, or the criterion was not met. Or a command could not do its job: a precondition was missing (no kubeconfig, no CNF installed, a CNF already installed, a tool failed to install) or an operation failed. A message says which.
`2`	The suite itself broke	A test raised an unexpected exception, or the process crashed outside a test. Always wins over `1`.
`64`	Usage error	The command line was wrong: an unknown task, flag or argument; a malformed value; a required argument missing; an argument naming a file that does not exist. Detected before the requested operation starts. (sysexits `EX_USAGE`.)

A script that only needs pass/fail can test for 0. A script that wants to tell "the CNF failed" (1) apart from "the suite or the invocation is broken" (2, 64) can branch on the code.

Common Example Commands

Building the executable

This is the command to build the binary executable if in developer mode or using the source install method (requires crystal):

crystal build src/cnti-testsuite.cr

Validating a cnti-testsuite.yaml file:

./cnti-testsuite validate_config --cnf-config [PATH_TO]/cnti-testsuite.yaml

Installing a cnf:

./cnti-testsuite cnf_install --cnf-config ./cnti-testsuite.yaml

Specify a timeout for resource readiness during installation:

./cnti-testsuite cnf_install --cnf-config ./cnti-testsuite.yaml --timeout 1800

Skip waiting for resource readiness during installation:

./cnti-testsuite cnf_install --cnf-config ./cnti-testsuite.yaml --skip-wait-for-install

Uninstalling a cnf:

./cnti-testsuite cnf_uninstall

Specify timeout for resource removal during uninstallation

./cnti-testsuite cnf_uninstall --timeout 60

Skip waiting for resource removal during uninstallation:

./cnti-testsuite cnf_uninstall --skip-wait-for-uninstall

Running all of the tests:

./cnti-testsuite all --cnf-config <path_to_your_config_file>/cnti-testsuite.yaml

Running all of the tests (including proofs of concepts)

./cnti-testsuite all --poc --cnf-config <path_to_your_config_file>/cnti-testsuite.yaml

Running all of the workload tests

crystal src/cnti-testsuite.cr -- workload --cnf-config <path_to_your_config_file>/cnti-testsuite.yaml

Running certification tests

./cnti-testsuite cert
./cnti-testsuite cert --essential
./cnti-testsuite cert --skip increase_decrease_capacity --skip single_process_type

Running the workload tests:

./cnti-testsuite workload

Get available options and to see all available tests from command line:

help prints a usage page: the typical workflow, every option with its meaning, and how to skip tasks or run several. -h and a bare invocation print the same page.

./cnti-testsuite help
./cnti-testsuite -h

To list every task, grouped by category:

./cnti-testsuite help tasks

To see what a single task does, what it runs first, and which suites it belongs to:

./cnti-testsuite help liveness

An unknown task name is answered with the closest match, e.g. ./cnti-testsuite livenes suggests liveness.

Shell completion:

completion prints a completion script for bash (default) or zsh, generated from the same task and option registries help uses, so it always matches the binary that printed it: namespaced task paths (setup:install_kubescape), every --option with its value (file names after --cnf-config and the other path options, task names after --skip, -l log levels), and further task names.

source <(./cnti-testsuite completion bash)     # this shell only
source <(./cnti-testsuite completion zsh)      # zsh (uses bashcompinit)

To make it permanent, add that line to ~/.bashrc / ~/.zshrc, or for bash drop the script where bash-completion picks it up:

./cnti-testsuite completion bash > ~/.local/share/bash-completion/completions/cnti-testsuite

The script completes the binary name it was generated by (cnti-testsuite for the release binary). To remove it: complete -r cnti-testsuite.

Print the version:

./cnti-testsuite --version

Clean up the CNTI Test Suite, the K8s cluster, and upstream projects:

./cnti-testsuite uninstall_all

Logging Options

Update the loglevel from command line:

# cmd line
./cnti-testsuite -l debug test

If in developer mode, make sure to use - - if running from source:

crystal src/cnti-testsuite.cr -- -l debug test

You can also use env var for logging:

CNTI_TESTSUITE_LOG_LEVEL=DEBUG ./cnti-testsuite test

:star: Note: When setting log level, the following is the order of precedence:

CLI or Command line flag
Environment variable
CNTi Test Suite Config file — ./config.yml in the working directory, else config.yml in the suite home (~/.cnti-testsuite, relocatable via CNTI_TESTSUITE_DIR)

Note: Available log levels are: trace, debug, info, notice, warn, error and fatal.

Environment variables for timeouts:

Timeouts are controlled by these environment variables, set them if default values aren't suitable:

CNTI_TESTSUITE_GENERIC_OPERATION_TIMEOUT=60
CNTI_TESTSUITE_RESOURCE_CREATION_TIMEOUT=120
CNTI_TESTSUITE_NODE_READINESS_TIMEOUT=240
CNTI_TESTSUITE_POD_READINESS_TIMEOUT=180
CNTI_TESTSUITE_LITMUS_CHAOS_TEST_TIMEOUT=1800
CNTI_TESTSUITE_NODE_DRAIN_TOTAL_CHAOS_DURATION=90
CNTI_TESTSUITE_LABEL_RESOURCE_SLEEP=5

Network retries:

Network-bound fetches (chart pulls, repo adds, binary downloads) are retried on transient failures — resets, timeouts, DNS blips, 5xx — while a missing chart or auth error still fails immediately:

CNTI_TESTSUITE_NETWORK_RETRY_ATTEMPTS=3   # attempts per fetch
CNTI_TESTSUITE_NETWORK_RETRY_BACKOFF=2    # base seconds between attempts (attempt N waits N x backoff)

Other environment variables:

CNTI_TESTSUITE_DIR: the suite home (default ~/.cnti-testsuite): where config.yml is looked up and where the tools the suite installs for itself are cached under tools/. Each cached tool carries a <tool>.version marker with the version it was installed at; a suite that pins a different version reinstalls it on the next run, so the cache never serves a stale tool.
CNTI_TESTSUITE_FORCE_INSTALL: set (to any value) to reinstall the suite-managed local Helm even when an installation is already present.
CNTI_TESTSUITE_ENV: set to TEST for test-mode shortcuts (smaller samples, quicker checks). Used by the spec suite; not meant for normal runs.
CNTI_TESTSUITE_RESULTS_DIR: redirect the results directory; see Results file.

Every variable that configures the suite's own behavior carries the CNTI_TESTSUITE_ prefix. Credentials and ecosystem-wide standards deliberately keep their conventional names, so one export serves every tool that honors them: KUBECONFIG, GITHUB_TOKEN, DOCKERHUB_USERNAME/DOCKERHUB_PASSWORD (shared by docker login, helm registry login, and the suite's registry queries — see INSTALL.md), NO_COLOR, and HOME.

Running The Linter

Ameba (https://github.com/crystal-ameba/ameba) is a static code linter for crystal-lang. To run Ameba, testsuite needs to be installed in developer mode (Source Install) and Ameba needs to be installed using source method, which is mentioned in Ameba readme.md:

git clone https://github.com/crystal-ameba/ameba && cd ameba
make install

After that, follow the usage guidelines from the Ameba repository.

Usage for categories and single tests

It's located in TEST_DOCUMENTATION, Check for needed category or test there.