SCT - Scylla Cluster Tests
September 10, 2026 · View on GitHub
SCT tests are designed to test Scylla database on physical/virtual servers under high read/write load. Currently, the tests are run using built in unittest These tests automatically create:
- Scylla clusters - Run Scylla database
- Loader machines - used to run load generators like cassandra-stress
- Monitoring server - uses official Scylla Monitoring repo to monitor Scylla clusters and Loaders
Quickstart
Option 1 - Config AWS using OKTA (preferred option)
Option 2 - Config AWS using AWS credentials
# install aws cli
sudo apt install awscli # Debian/Ubuntu
sudo dnf install awscli # Redhat/Fedora
# or follow amazon instructions to get it: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
# Ask your AWS account admin to create a user and access key for AWS) and then configure AWS
> aws configure
AWS Access Key ID [****************7S5A]:
AWS Secret Access Key [****************5NcH]:
Default region name [us-east-1]:
Default output format [None]:
# if using OKTA, use any of the tools to create the AWS profile, and export it as such,
# anywhere you are gonna use hydra command (replace DeveloperAccessRole with the name of your profile):
export AWS_PROFILE=DeveloperAccessRole
# Install hydra (docker holding all requirements for running SCT)
sudo ./install-hydra.sh
# if using podman, we need to disable enforcing of short name usage, without it monitoring stack won't run from withing hydra
echo 'unqualified-search-registries = ["registry.fedoraproject.org", "registry.access.redhat.com", "docker.io", "quay.io"]
short-name-mode="permissive"
' > ~/.config/containers/registries.conf
Run a test
Disable Argus (if not needed for testing)
To disable Argus reporting during testing, set the following environment variable:
export SCT_ENABLE_ARGUS=false
Logs Location
All logs generated during test runs can be found in the ~/sct-results directory.
Example running test using Hydra using test-cases/PR-provision-test.yaml configuration file
Run test locally with AWS backend:
export SCT_SCYLLA_VERSION=5.2.1
# Test fails to report to Argus. So we need to disable it
export SCT_ENABLE_ARGUS=false
# configuration is needed for running from a local development machine (default communication is via private addresses)
hydra run-test longevity_test.LongevityTest.test_custom_time --backend aws --config test-cases/PR-provision-test.yaml --config configurations/network_config/test_communication_public.yaml
# Run with IPv6 configuration
hydra run-test longevity_test.LongevityTest.test_custom_time --backend aws --config test-cases/PR-provision-test.yaml --config configurations/network_config/all_addresses_ipv6_public.yaml
Run test using SCT Runner with AWS backend:
hydra create-runner-instance --cloud-provider <cloud_name> -r <region_name> -z <az> -t <test-id> -d <run_duration>
export SCT_SCYLLA_VERSION=5.2.1
# For choose correct network configuration, check test jenkins pipeline.
# All predefined configurations are located under `configurations/network_config`
hydra --execute-on-runner <runner-ip|`cat sct_runner_ip> "run-test longevity_test.LongevityTest.test_custom_time --backend aws --config test-cases/PR-provision-test.yaml"
Run test locally with GCE backend:
export SCT_SCYLLA_VERSION=5.2.1
export SCT_IP_SSH_CONNECTIONS="public"
hydra run-test longevity_test.LongevityTest.test_custom_time --backend gce --config test-cases/PR-provision-test.yaml
Run test locally with Azure backend:
export SCT_SCYLLA_VERSION=5.2.1
hydra run-test longevity_test.LongevityTest.test_custom_time --backend azure --config test-cases/PR-provision-test.yaml
Run test locally with docker backend:
If you wish to run the tests on a local machine, ensure you have Docker installed and running. If necessary, set a higher value for asynchronous I/O by:
echo "fs.aio-max-nr=3000000" | sudo tee /etc/sysctl.d/99-sct-aio.conf
sudo sysctl --system
For the purpose of debugging a simple logic of a nemesis, for example, it would be recommended to use Docker backend.
# **NOTE:** user should be part of sudo group, and setup with passwordless access,
# see https://unix.stackexchange.com/a/468417 for example on how to setup
# example of running specific docker version
export SCT_SCYLLA_VERSION=5.2.1
hydra run-test longevity_test.LongevityTest.test_custom_time --backend docker --config test-cases/PR-provision-test-docker.yaml
Run test with ScyllaDB Cloud (xcloud) backend:
export SCT_SCYLLA_VERSION=2025.3.0
export SCT_XCLOUD_PROVIDER=aws
export SCT_XCLOUD_ENV=lab
hydra run-test longevity_test.LongevityTest.test_custom_time --backend xcloud --config test-cases/PR-provision-test.yaml
For more details on xcloud backend, see xcloud backend documentation
You can specify a specific scylla version by:
# Simple version (release)
export SCT_SCYLLA_VERSION=2025.1
# Branch version (nightly)
export SCT_SCYLLA_VERSION=master:latest
# Full version tag (specific build with commit hash)
export SCT_SCYLLA_VERSION=2024.2.5-0.20250221.cb9e2a54ae6d-1
For detailed information on full version tag support, see docs/full-version-tag-usage.md
For debugging a standard nemesis setup you can simply use a default nemesis setup. Use the yaml files as in https://github.com/scylladb/scylla-cluster-tests/blob/master/jenkins-pipelines/oss/nemesis/longevity-5gb-1h-AbortRepairMonkey-docker.jenkinsfile:
hydra run-test longevity_test.LongevityTest.test_custom_time --backend docker \
-c configurations/nemesis/longevity-5gb-1h-nemesis.yaml \
-c configurations/nemesis/AbortRepairMonkey.yaml \
-c configurations/nemesis/additional_configs/docker_backend.yaml
For debugging a specific nemesis setup you can edit the nemesis configuration to run. Change the relevant parameters in test-cases/PR-provision-test-docker.yaml like below:
test_duration: 60
stress_cmd: "cassandra-stress write cl=QUORUM duration=5m -schema 'replication(strategy=NetworkTopologyStrategy,replication_factor=3) ' -mode cql3 native -rate threads=10 -pop seq=1..100000 -log interval=5"
n_db_nodes: 4
nemesis_class_name: 'SisyphusMonkey'
nemesis_selector: 'DecommissionMonkey' # Filter to run only DecommissionMonkey
nemesis_interval: 5
The nemesis_class_name specifies the runner (e.g. SisyphusMonkey), while nemesis_selector filters which nemesis classes to include using boolean flag expressions or class names.
For more details on nemesis architecture, flags, and configuration, see the Nemesis Developer Guide.
For docker backend supported nemesis, check docker backend specifics.
#### You can also enter the containerized SCT environment using:
```bash
hydra bash
List resources being used by user:
# NOTE: Only use `whoami` if your local use is the same as your okta/email username
hydra list-resources --user `whoami`
Reuse already running cluster:
export SCT_REUSE_CLUSTER=$(cat ~/sct-results/latest/test_id)
hydra run-test longevity_test.LongevityTest.test_custom_time --backend aws --config test-cases/PR-provision-test.yaml --config configurations/network_config/test_communication_public.yaml
More details on reusing a cluster can be found in reuse_cluster
Clear resources:
hydra clean-resources --user `whoami`
# by default, it only cleans aws resources
# to clean other backends, specify manually
hydra clean-resources --user `whoami` -b gce
Clear resources being used by the last test run:
SCT_CLUSTER_BACKEND= hydra clean-resources --test-id `cat ~/sct-results/latest/test_id`
Install local development environment
Frequently Asked Questions (FAQ)
Contribution instructions
HTTP retry & timeout conventions
AI code review on PRs
Claude code review is on-demand. To get a review, comment @claude please review this PR
on the pull request. Available to scylladb org members and collaborators.
See How does the Claude AI code review work? for details.
Supported backends
-
aws- the mostly used backed, most longevity run on top of this backend -
gce- most of the artifacts and rolling upgrades run on top of this backend -
azure- -
docker- should be used for local development -
baremetal- experimental — can be used to run with an already-provisioned cluster -
xcloud- ScyllaDB Cloud managed clusters -
k8s-eks- -
k8s-gke- -
k8s-local-kind- used for run k8s functional test locally -
k8s-local-kind-gce- used for run k8s functional test locally on GCE -
k8s-local-kind-aws- used for run k8s functional test locally on AWS
Configuring test run configuration YAML
Take a look at the test-cases/PR-provision-test.yaml file. It contains a number of
configurable test parameters, such as DB cluster instance types and AMI IDs.
In this example, we're assuming that you have copied test-cases/PR-provision-test.yaml
to test-cases/your_config.yaml.
All the test run configurations are stored in test-cases directory.
Important: Some tests use custom hardcoded operations due to their nature,
so those tests won't honor what is set in test-cases/your_config.yaml.
Configuration Documentation
- SCT Configuration Guide - Comprehensive guide on how the configuration system works and how to add new options
- Configuration Options Reference - Auto-generated index of all configuration options, grouped by backend, test type and concern
- Cross-Cloud Instance Sizing - Constraint-based instance selection across AWS, GCE, Azure, and OCI
- Running against minicloud - Local QEMU/KVM cloud emulation: run SCT tests without real cloud credentials or cost
Diagnostics Documentation
- Kernel Panic Detection - How SCT detects kernel panics from cloud serial consoles, and how to troubleshoot false positives
- Collected Log Files - What each log archive contains after a test run
Development Plans
Active implementation plans for SCT are tracked in docs/plans/MASTER.md. For guidelines on creating new plans, see docs/plans/INSTRUCTIONS.md.