Testing Guide

May 3, 2026 · View on GitHub

Overview

The TNS CSI Driver is tested comprehensively using real infrastructure - not mocks, simulators, or virtual TrueNAS instances. Every commit triggers automated tests against actual hardware and software.

Testing Infrastructure

Real Hardware, Real Tests

Main integration suite — GitHub-hosted runners with QEMU:

integration.yml runs on GitHub Actions ubuntu-24.04 runners
Each protocol job boots its own Ubuntu Noble cloud-image VM via QEMU/KVM (.github/actions/qemu-vm composite action)
k3s is installed inside the VM with full kernel-module support (nvme-tcp, iscsi_tcp, nfs, cifs)
The host runner and VM both join a Tailscale tailnet so they can reach the private TrueNAS

Auxiliary workflows — currently disabled:

encryption.yml, scale.yml, snapclone-stress.yml, snapshot-clone-matrix.yml, snapshot-debug.yml, compatibility.yml, distro-compatibility.yml previously ran on a dedicated self-hosted runner (label new); that runner has been retired
Files are renamed with a .yml.disabled suffix in .github/workflows/ so GitHub Actions ignores them but the migration inventory stays visible
Pending port to the same QEMU composite-action pattern as integration.yml

Real TrueNAS Scale Server:

Physical TrueNAS Scale 25.10+ installation (Akamai/Linode)
Reachable from CI via Tailscale tailnet (TRUENAS_HOST_TAILSCALE secret) — never exposed to the public internet
Real storage pools with ZFS
Actual NFS shares, SMB shares, NVMe-oF subsystems, iSCSI targets
Real network I/O and protocol operations

Real Protocol Testing:

NFS: Actual NFS mounts from TrueNAS to Kubernetes nodes
NVMe-oF: Real NVMe-oF TCP connections and block device operations
WebSocket: Live API connections to TrueNAS with authentication
Full end-to-end data path testing

Why Real Infrastructure?

Testing against real infrastructure catches issues that mocks cannot:

Network timing and race conditions
Actual protocol behavior and error modes
TrueNAS API quirks and edge cases
Real-world performance characteristics
Connection resilience and recovery
Cleanup and resource management

Test Framework

All integration tests use Ginkgo v2 and Gomega for BDD-style testing with:

Structured Describe/It blocks for clear test organization
Eventually for robust async waiting (no brittle sleep calls)
LIFO cleanup stack ensuring resources are always cleaned up
Parallel test execution support
JUnit report generation for CI integration

Automated Test Suite

CSI Specification Compliance

Sanity Tests:

Uses kubernetes-csi/csi-test v5.4.0
Validates full CSI specification compliance
Tests all CSI RPC calls and error conditions
Location: tests/sanity/
Run on: Every CI build

Ginkgo E2E Integration Tests

Every push to main triggers comprehensive integration tests organized into protocol-specific test suites:

NFS Test Suite (`tests/e2e/nfs/`)

Test File	Description
`basic_test.go`	Volume provisioning, mounting, I/O operations, deletion
`access_modes_test.go`	RWO/RWX access mode validation
`adoption_test.go`	Volume adoption for GitOps workflows
`clone_test.go`	Volume cloning from snapshots
`concurrent_test.go`	5 simultaneous volume creations
`delete_strategy_retain_test.go`	Volume retention on PVC deletion
`detached_snapshot_test.go`	Snapshot lifecycle without attached pods
`persistence_test.go`	Data survives pod restarts
`statefulset_test.go`	StatefulSet with VolumeClaimTemplates
`zfsprops_test.go`	Custom ZFS properties via StorageClass

NVMe-oF Test Suite (`tests/e2e/nvmeof/`)

Test File	Description
`basic_test.go`	Volume provisioning, mounting, I/O operations, deletion
`access_modes_test.go`	RWO access mode validation
`block_mode_test.go`	Raw block device mode testing
`clone_test.go`	Volume cloning from snapshots
`concurrent_test.go`	5 simultaneous volume creations
`delete_strategy_retain_test.go`	Volume retention on PVC deletion
`detached_snapshot_test.go`	Snapshot lifecycle without attached pods
`persistence_test.go`	Data survives pod restarts
`statefulset_test.go`	StatefulSet with VolumeClaimTemplates
`zfsprops_test.go`	Custom ZFS properties via StorageClass

iSCSI Test Suite (`tests/e2e/iscsi/`)

Test File	Description
`basic_test.go`	Volume provisioning, mounting, I/O operations, deletion
`access_modes_test.go`	RWO access mode validation
`block_mode_test.go`	Raw block device mode testing
`clone_test.go`	Volume cloning from snapshots
`concurrent_test.go`	5 simultaneous volume creations
`delete_strategy_retain_test.go`	Volume retention on PVC deletion
`detached_snapshot_test.go`	Snapshot lifecycle without attached pods
`persistence_test.go`	Data survives pod restarts
`statefulset_test.go`	StatefulSet with VolumeClaimTemplates
`zfsprops_test.go`	Custom ZFS properties via StorageClass

Shared Test Suite (`tests/e2e/`)

Test File	Description
`snapshot_restore_test.go`	Snapshot creation and restoration (all protocols)
`detached_snapshot_advanced_test.go`	Detached snapshots via zfs send/receive, DR scenario testing
`stress_test.go`	Volume stress testing
`name_templating_test.go`	Custom volume naming templates
`error_handling_test.go`	Error condition handling
`dual_mount_test.go`	Simultaneous NFS + NVMe-oF mounting
`connection_resilience_test.go`	WebSocket reconnection testing

Test Execution

Tests run via GitHub Actions workflow (.github/workflows/integration.yml):

# NFS tests (~25 minutes)
ginkgo -v --timeout=25m ./tests/e2e/nfs/...

# NVMe-oF tests (~40 minutes)
ginkgo -v --timeout=40m ./tests/e2e/nvmeof/...

# iSCSI tests (~40 minutes)
ginkgo -v --timeout=40m ./tests/e2e/iscsi/...

# Shared tests (~25 minutes)
ginkgo -v --timeout=25m ./tests/e2e/

Each test run:

Deploys CSI driver via Helm to a fresh namespace
Creates test resources (PVCs, pods, snapshots)
Validates operations with Eventually for robustness
Cleans up all resources automatically (LIFO stack)
Verifies TrueNAS backend cleanup

View test results: Test Dashboard

Test Results and History

CI/CD Badges

- Unit tests and sanity tests
- Full Ginkgo E2E test suite

Test Dashboard

Interactive test results dashboard with history and metrics:

https://fenio.github.io/tns-csi/dashboard/
Shows pass/fail status for all tests
Tracks test duration over time
Identifies flaky tests and patterns

Running Tests Locally

Prerequisites

Go 1.21+
Ginkgo CLI: go install github.com/onsi/ginkgo/v2/ginkgo@latest
Access to a TrueNAS Scale 25.10+ server
Kubernetes cluster (k3s recommended)
TrueNAS API key with admin privileges
For NFS: nfs-common installed
For NVMe-oF: nvme-cli installed, kernel modules loaded
For iSCSI: open-iscsi installed, iscsid service running

Environment Variables

export TRUENAS_HOST="your-truenas-ip"
export TRUENAS_API_KEY="your-api-key"
export TRUENAS_POOL="your-pool"
export KUBECONFIG="$HOME/.kube/config"

CSI Sanity Tests

cd tests/sanity
./test-sanity.sh

Ginkgo E2E Tests

# Run all E2E tests
ginkgo -v --timeout=60m ./tests/e2e/...

# Run NFS tests only
ginkgo -v --timeout=25m ./tests/e2e/nfs/...

# Run NVMe-oF tests only
ginkgo -v --timeout=40m ./tests/e2e/nvmeof/...

# Run iSCSI tests only
ginkgo -v --timeout=40m ./tests/e2e/iscsi/...

# Run shared tests only
ginkgo -v --timeout=25m ./tests/e2e/

# Run specific test by name
ginkgo -v --focus="expand" ./tests/e2e/nfs/...

# Run with verbose output
ginkgo -v -vv ./tests/e2e/nfs/...

# Generate JUnit report
ginkgo -v --junit-report=junit-nfs.xml ./tests/e2e/nfs/...

Note: E2E tests assume a Kubernetes cluster with kubectl access. They will deploy the CSI driver, run tests, and clean up.

Using Makefile Targets

# Run all E2E tests
make test-e2e

# Run NFS E2E tests
make test-e2e-nfs

# Run NVMe-oF E2E tests
make test-e2e-nvmeof

# Run iSCSI E2E tests
make test-e2e-iscsi

Test Coverage

What's Tested

CSI Spec Compliance - Full CSI spec validation via csi-test
Volume Lifecycle - Create, attach, mount, unmount, detach, delete
Volume Expansion - Dynamic resizing (NFS, NVMe-oF & iSCSI)
Snapshots - Create, restore, clone (NFS, NVMe-oF & iSCSI)
StatefulSets - VolumeClaimTemplates and pod identity
Data Persistence - Data survives pod restarts
Concurrent Operations - Race condition detection
Connection Resilience - WebSocket reconnection
Resource Cleanup - Orphaned resource detection
ZFS Properties - Custom compression, recordsize, etc.
Block Mode - Raw block device support (NVMe-oF)

What's Not Yet Tested

Multi-node scenarios - Tests run on single-node k3s
Network partitions - Not tested yet
Storage pool failures - Not tested yet
Long-running workloads - No soak tests yet
Performance benchmarks - No formal performance testing

Contributing Tests

When adding new features:

Add unit tests in pkg/*/
Add Ginkgo E2E test in appropriate tests/e2e/ subdirectory:
- tests/e2e/nfs/ for NFS-specific tests
- tests/e2e/nvmeof/ for NVMe-oF-specific tests
- tests/e2e/ for shared/cross-protocol tests
Follow the existing test patterns (see tests/e2e/README.md for templates)
Ensure tests run on real infrastructure (no mocks for E2E tests)
Update this documentation if adding new test categories