Testkube Agent Architecture

September 21, 2026 · View on GitHub

The Testkube Agent is 100% Open Source and can be run in two modes:

  • In Standalone Mode (free), the Agent manages results/artifact storage, scheduling, triggering, etc.
  • In Connected Mode (commercial), core functionality is delegated to the Testkube Control Plane and the Agent primarily runs Workflows scheduled by the Control Plane and reports results back to it.

You can read more about the differences between the two deployment modes in the Testkube Documentation

This document describes the high-level architecture of the Testkube Agent when run in Standalone Mode

Table of Contents

Core Components

1. API Server

Entry Point: cmd/api-server/main.go

The API server is the main service that:

  • Exposes REST (HTTP) and gRPC APIs for managing tests, workflows, and executions
  • Handles TestWorkflow execution requests
  • Manages storage connections (MongoDB/PostgreSQL, MinIO, NATS)
  • Runs Kubernetes controllers for watching CRDs
  • Processes events and webhooks

Key Packages:

2. Kubernetes Controllers

Location: pkg/controller/

Controllers watch Kubernetes Custom Resource Definitions (CRDs) and trigger actions:

  • TestWorkflowExecution Controller (testworkflowexecutionexecutor.go) - Watches TestWorkflowExecution CRDs and schedules TestWorkflow executions when CRDs are created/updated

Controllers are enabled via ENABLE_K8S_CONTROLLERS=true and use controller-runtime.

GitOps sync controllers (Connected Mode only): internal/sync/controller/ holds a second set of reconcilers — one each for TestWorkflow, TestWorkflowTemplate, TestTrigger, WorkflowTrigger, Webhook, and WebhookTemplate — that push Kubernetes resources into the Control Plane over the gRPC SyncService (internal/sync/grpc/). They are registered separately in cmd/api-server/main.go behind GITOPS_KUBERNETES_TO_CLOUD_ENABLED. Because the Control Plane grants exclusive ownership of a synced resource to a single GitOps agent, a sync it rejects as an ownership conflict is returned as a reconcile.TerminalError instead of being retried, so one agent cannot overwrite another's resources and cannot spin on a conflict it has no way to resolve. See AGENTS.md for the full ownership contract.

3. TestWorkflow Execution Runtime

Testkube uses Test Workflows as an abstraction layer for running any kind of test inside Kubernetes.

TestWorkflow Init: cmd/testworkflow-init/

  • Initializes TestWorkflow execution containers
  • Orchestrates TestWorkflow step groups and parallel execution
  • Handles container lifecycle and coordination

TestWorkflow Toolkit: cmd/testworkflow-toolkit/

  • Runtime utilities for TestWorkflow containers
  • Artifact collection and upload
  • Log streaming and aggregation

Execution Logic: pkg/testworkflows/

  • Core TestWorkflow executor (testworkflowexecutor/)
  • TestWorkflow processing and step execution
  • Result aggregation and status management
  • Execution worker (executionworker/): applies the workflow to Kubernetes, watches the job and pod, and stops executions. When a caller aborts or cancels an execution, it passes an actor code, a reason code, and an optional detail in DestroyOptions, and the worker writes them into the job annotations testkube.io/termination-actor, testkube.io/termination-reason, and testkube.io/termination-detail. The result reader renders the words for both codes, so the stored error message names the component that stopped the execution and, when there is one, the cause.
  • Runner gRPC client (pkg/runner/grpc/): receives execution starts from the control plane. When the runner cannot start an execution, the client reads the reason code from the StartError in the error chain and declines the execution with the code and the error text, so the control plane stores the cause on the execution.

4. Execution Lineage and Reruns

A rerun descends from a specific earlier execution, and TestWorkflowExecutionLineage (baseId, rootId, attempt) is what records that. It is written for every execution, not only reruns: an original run is its own root at attempt 1, so "every execution of chain R" is one predicate and includes the original. In SQL that predicate is COALESCE(lineage_root_id, id) = R, not the bare column: a row written before lineage existed carries NULL there and is its own root, so a bare-column query would find a rerun of a legacy execution but not the original it descends from. The chain index is built on the same expression.

Propagation: only the base execution id travels on the wire, as ScheduleRequest.base_execution_id - a caller able to assert a root or an attempt could forge a chain. The scheduler derives the rest:

ScheduleRequest.base_execution_id
  -> Enqueuer.deriveLineage         (loads the base; root carried down, attempt + 1)
  -> execution record               (lineage_base_id / lineage_root_id / lineage_attempt)
  -> ExecutionStart.lineage         (every writer must read it back off the record)
  -> ExecutionConfig.Lineage        (the pod's internal config)
  -> RerunExecutionId()             (resolves the reserved execution("rerun") reference)

Loading the base is also where the Control Plane confirms it belongs to the caller's environment, as proto/service.proto requires: the results repository is scoped to the organization and environment, so a base outside them comes back not-found and the request is refused.

The same path exists in the connected-mode scheduler in testkube-cloud-api, which keeps its own copy of the test_workflow_executions schema and reads executions through its own queries. Both halves have to carry lineage; a writer that omits it sends the pod nothing, execution("rerun") stops resolving, and nothing reports it.

Storage: three scalar columns rather than JSONB, because they are queried - an ordered range scan behind the organization/environment prefix, which a GIN containment index could neither order nor compose with. Indexed with a non-partial expression index on COALESCE(lineage_root_id, id), so both chained rows and legacy/original rows participate in the same ordered scan, and on COALESCE(lineage_attempt, 1) rather than the bare attempt, so a legacy row - NULL attempt, meaning 1 - orders as the chain root it is instead of sorting last. A chain query has to spell both expressions the same way to get the index.

Legacy rows: executions written before the columns existed carry NULL, and are never backfilled. They mean exactly what an original run means, and TestWorkflowExecution.EffectiveLineage() synthesizes that - no base, itself as the root, attempt 1. That accessor is the single source of the default: the expression machine behind {{ execution.lineage.* }} applies the same field-by-field fallbacks, so an old execution cannot report one lineage through the API and a different one to its own workflow.

Resolved in the pod, not while scheduling: IntermediateExecution.Resolve substitutes execution.* into the spec and the result is stored as ResolvedWorkflow, which a rerun replays verbatim unless it is asked for the latest definition. So anything resolved there is frozen at the values of the run that produced the snapshot. For an id or a number that is correct - the snapshot is a record of that run - but for lineage it is fatal: the original's "no base, attempt 1" would be baked in, every rerun of that snapshot would take the original branch, and execution("rerun") would never be reached. CreateSchedulingExecutionMachine therefore omits the accessor, which leaves {{ execution.lineage.* }} dynamic rather than empty (an accessor nothing matches resolves to itself), and the pod resolves it against the execution actually running - including in step conditions, which ResolveCondition evaluates through data.Expression. Do not add lineage back to that machine to make it usable in a pod-spec field; that is the trade-off, not an oversight. TestResolveLeavesLineageForThePod is the regression.

Reserved references: execution("parent") and execution("rerun") resolve before the registry of executions a workflow scheduled, so a child aliased - or a workflow named - parent/rerun cannot shadow them. A collision is refused rather than resolved either way, since preferring the reserved meaning would instead make that child unreachable by name.

5. Storage Layer

PostgreSQL (Future Primary Database, currently in Preview)

MongoDB (Current Primary Database)

MinIO (Object Storage)

  • Stores TestWorkflow execution artifacts (logs, reports, files)
  • Buckets: testkube-artifacts, testkube-logs
  • Storage interface: pkg/storage/

NATS (Message Queue)

6. Event System

Location: pkg/event/

The event system publishes and listens to TestWorkflow execution events:

7. REST API

Testkube exposes REST APIs for interacting with core resources and functionality - Read More.

OpenAPI Definition: api/v1/testkube.yaml

  • Defines the complete REST API contract
  • Used for client code generation and documentation
  • Generated models: pkg/api/v1/testkube/
  • The actor codes and the reason codes, with their words, live in the same package as hand-written files, so the worker, the runner, and the control plane share one definition

Framework: Uses Fiber web framework for HTTP routing and middleware

Route Registration: internal/app/api/v1/server.go - TestkubeAPI.Init()

Handler Implementation: internal/app/api/v1/

  • Handlers: testworkflows.go, testworkflowexecutions.go, webhook.go, etc.
  • Each handler function (e.g., ListTestWorkflowsHandler()) returns a Fiber handler
  • Handlers interact with repositories, executors, and event emitters

Response Formats: Supports JSON and YAML (via Accept header)

  • Default: application/json
  • Alternative: text/yaml or application/yaml

Port: HTTP API listens on port 8088 (configurable via environment variables)

8. Prometheus Metrics Endpoint

Endpoint: GET /metrics

The API server exposes Prometheus metrics at /metrics for monitoring and observability - Read More.

Metrics Implementation: internal/app/api/metrics/metrics.go

Server Setup: The metrics endpoint is registered in pkg/server/httpserver.go using Prometheus's standard HTTP handler (promhttp.Handler()).

Access: Metrics are accessible at http://localhost:8088/metrics (or the configured API server port).

9. Logging and Telemetry

Logging

Framework: Uses zap structured logging library

Implementation: pkg/log/log.go

Configuration:

  • Log Level: Controlled via DEBUG environment variable
    • Default: InfoLevel
    • Set DEBUG=true for DebugLevel
  • Output Format: Controlled via LOGGER_JSON environment variable
    • Default: Production format (JSON)
    • Set LOGGER_JSON=true for Development format (human-readable)

Usage:

  • Default Logger: log.DefaultLogger - Singleton logger used throughout the codebase
  • Logger Methods:
    • Info(), Infow() - Information messages
    • Debug(), Debugw() - Debug messages
    • Error(), Errorw() - Error messages
    • Warn(), Warnw() - Warning messages
  • Structured Logging: Use Infow(), Errorw(), etc. for structured logs with key-value pairs
    • Example: log.DefaultLogger.Infow("connected to database", "host", dbHost, "port", dbPort)

Timestamps: Logs include RFC3339 formatted timestamps

Telemetry

Implementation: pkg/telemetry/

Telemetry collects usage analytics to help improve the product. It can be disabled by users.

Telemetry Backends:

  • Segment.io (sender_sio.go) - Primary analytics backend
  • Google Analytics (sender_ga4.go) - Alternative analytics backend
  • Testkube Analytics (sender_tka.go) - Internal analytics

Heartbeat: cmd/api-server/services/telemetry.go

  • Sends a testkube_api_start event on startup and a testkube_api_heartbeat event every hour
  • Both events include the detected cluster type and agent capabilities
  • Capability tags come from cmd/api-server/services/capabilities.go and cover the agent persona, connection mode, enabled features, and whether this is a Testkube-provisioned hosted runner (hosted-runner) rather than a user-deployed one

10. Kubernetes Custom Resource Definitions (CRDs)

Definition Location: api/ Generated CRDs: k8s/crd/

Testkube extends Kubernetes with Custom Resource Definitions to enable declarative TestWorkflow management. CRDs are defined using Kubebuilder annotations and generated from Go types.

CRD Generation: Run make generate-crds to regenerate CRDs after modifying types in api/.

Legacy CRDs are no longer supported by Testkube but still included to avoid deletion of corresponding resources on deployment.

TestWorkflow CRDs

  • TestWorkflow (testworkflows.testkube.io/v1)

    • Definition: api/testworkflows/v1/testworkflow_types.go
    • Purpose: Defines a TestWorkflow with setup, steps, and after phases
    • Features: Template inclusion, parallel execution, service dependencies, PVCs
    • Status: Tracks latest execution and health metrics

  • TestWorkflowTemplate (testworkflows.testkube.io/v1)

  • TestWorkflowExecution (testworkflows.testkube.io/v1)

Webhook CRDs

  • Webhook (executor.testkube.io/v1)

    • Definition: api/executor/v1/webhook_types.go
    • Purpose: Defines webhooks triggered by TestWorkflow execution events
    • Targeting: Supports a target field (commonv1.Target) to control which agents execute the webhook

  • WebhookTemplate (executor.testkube.io/v1)

    • Definition: api/executor/v1/webhook_types.go
    • Purpose: Reusable webhook templates with configurable payloads
    • Targeting: Supports a target field (commonv1.Target) for agent-level targeting

Other CRDs

  • TestTrigger (tests.testkube.io/v1)
    • Definition: api/testtriggers/v1/testtrigger_types.go
    • Purpose: Automatically triggers tests/workflows based on Kubernetes events
    • Features: Watches Pods, Deployments, Services, etc. and triggers executions; supports git-content based triggers reconciled by the git informer
    • Event forms: spec.event (single) and spec.events (list) are mutually exclusive; validation in api/testtriggers/v1/validation.go enforces exactly one, and consumers normalize both forms through EffectiveEvents() (CRD spec and API model each expose it) — new event consumers must use the normalized list, never read spec.event directly
    • Leader behavior: Git informer reconciliation is registered as a leader-coordinated task in cmd/api-server/main.go, so only the elected leader performs git polling/pulls

Deprecated CRDs

  • A number of now-deprecated CRDs are still in the codebase to avoid the removal of corresponding Kubernetes resources.
    • Test (tests.testkube.io/v1, v2, v3)
    • TestExecution (tests.testkube.io/v1)
    • TestSource (tests.testkube.io/v1)
    • TestSuite (tests.testkube.io/v1, v2, v3)
    • TestSuiteExecution (tests.testkube.io/v1)
    • Executor (executor.testkube.io/v1)
    • Template (tests.testkube.io/v1)
    • Script (tests.testkube.io/v1, v2)

CRD Lifecycle

  1. Definition: CRDs are defined in Go using Kubebuilder annotations (+kubebuilder:object:root=true)
  2. Generation: controller-gen generates CRD YAML files in k8s/crd/
  3. Post-processing: CRD files are optimized to reduce size (for Kubernetes annotation limits)
  4. Deployment: CRDs are installed via the Helm chart (k8s/helm/testkube/)
  5. API Server: Kubernetes API server validates and stores CRD instances
  6. Controllers: Controllers watch CRDs and take actions (see Kubernetes Controllers)

Kubernetes Deployment

Helm Chart: k8s/helm/testkube/

The Helm chart deploys:

  • API server deployment
  • MongoDB or PostgreSQL (via subchart) - MongoDB is default but will be deprecated.
  • MinIO (via subchart)
  • NATS (via subchart)
  • Kubernetes RBAC and service accounts

Configuration: See k8s/helm/testkube/values.yaml for deployment configuration.

CLI

Entry Point: cmd/kubectl-testkube/main.go

The Testkube CLI (kubectl-testkube, typically invoked as testkube) is a kubectl plugin that provides a command-line interface for managing tests, workflows, and executions.

Architecture

Completion Command: [cmd/kubectl-testkube/commands/completion.go] (custom implementation that generates zsh completion under the actual binary name kubectl-testkube instead of testkube to ensure proper shell integration)

Command Structure: cmd/kubectl-testkube/commands/

  • Root command and command groups (testworkflows, webhooks, artifacts, etc.)
  • Common utilities: cmd/kubectl-testkube/commands/common/
  • Client abstraction: Works with both standalone API and control plane APIs

Context Resolution: Commands that act on a Control Plane environment resolve their target through cmd/kubectl-testkube/commands/common/orgenv.go. The precedence is an explicit --org-id/--env-id, then --org-name/--env-name resolved against the Control Plane's organization and environment listings, then an interactive selector when the terminal allows one. Name matching is exact, with a slug fallback for environments, and an ambiguous name is an error rather than an arbitrary pick. The resolved ids are persisted to ~/.testkube/config.json, which is the context every later command reads.

Client Layer:

Configuration: The CLI stores configuration in ~/.testkube/ directory, including:

  • API server endpoints (standalone or control plane)
  • Authentication tokens
  • Contexts (for multi-environment setups)

External Integration: License Event Reporting

The CLI reports installation lifecycle events to the Testkube license service so the install funnel can be tracked as telemetry.

Endpoint: POST https://license.testkube.io/events (the license worker's /events handler). The URL is defined as LicenseEventsURL in pkg/diagnostics/validators/license/client.go.

Client: Client.ReportEvent(license, event) in the same file marshals { "license": <key>, "event": <name> } and POSTs it with a short (5s) request timeout.

Events (constants in client.go):

  • cli_install_started — emitted right before the Helm install begins.
  • cli_install_finished — emitted right after the install succeeds.

Flow: During testkube init demo (cmd/kubectl-testkube/commands/init.go), the reportLicenseEvent helper wraps ReportEvent. It is:

  • Telemetry-gated — it is a no-op when the user has disabled telemetry (config.Data.TelemetryEnabled == false).
  • Non-blocking — each call runs in a background goroutine so a slow or unreachable endpoint never stalls the install path.
  • Best-effort / non-fatal — failures are logged at debug level only and never abort the installation.

Authentication: the license key sent in the request body is itself the credential — the license worker validates the key (against Keygen) before recording anything, so no separate shared secret ships in the public CLI. Recording is scoped to that license's own plan on the worker side.