README.md
May 4, 2026 ยท View on GitHub
What is system-tests?
Having trouble? Reach out on slack: #apm-shared-testing
System-tests is a black-box testing workbench for Datadog tracer libraries. It runs the same tests against every tracer implementation -- Java, Node.js, Python, PHP, Ruby, C++, .NET, Go, and Rust -- so shared features stay consistent across languages.
Key principles:
- Black-box testing -- only component interfaces are checked, no assumptions about internals. "Check that the car moves, regardless of the engine."
- Cross-language -- one test validates all tracer libraries.
Quick start
You need bash, Docker (20.10+), and Python 3.12.
# 1. Build images for the language you want to test
./build.sh python # or: java, nodejs, ruby, php, dotnet, golang
# 2. Run the tests
./run.sh # run all default tests
./run.sh SCENARIO_NAME # run a specific scenario
./run.sh tests/test_smoke.py::Test_Class::test_method # run a single test
Having trouble? Check the troubleshooting page.
To understand the test output, see test outcomes and the glossary.
Documentation
All detailed documentation lives in the docs/ folder. Here is a guided reading order:
Understand system-tests
| Topic | Description |
|---|---|
| Architecture overview | Components, containers, data flow |
| Scenarios | End-to-end, parametric, SSI, K8s -- what each one tests |
| Weblogs | The test applications instrumented by tracers |
| Glossary | Definitions of pass, fail, xpass, xfail, etc. |
Run tests
| Topic | Description |
|---|---|
| Build | Build options, weblog variants, image names |
| Run | Run options, selecting tests, scenarios, timeouts |
| Logs | Understanding the logs folder structure |
| Test outcomes | Reading test results |
| Replay mode | Re-run tests without starting the containers |
| Custom tracer versions | Testing with local tracer builds |
| Troubleshooting | Common issues and how to fix them |
Write and edit tests
| Topic | Description |
|---|---|
| Add a new test | Step-by-step guide to adding tests |
| Add a new scenario | Creating new test scenarios |
| Enable / disable tests | Activating tests for a library version |
| Manifests | How test activation is declared per library |
| Skip tests | Decorators for conditional skipping |
| Features | Linking tests to the feature parity dashboard |
| Formatting | Linter and code style |
| Troubleshooting | Debugging tips for test development |
CI integration
| Topic | Description |
|---|---|
| CI overview | Adding system-tests to your CI pipeline |
| GitHub Actions | GitHub Actions workflow details |
| System-tests CI | How the system-tests own CI works |
Internals
| Topic | Description |
|---|---|
| Internals overview | Deep-dive index for maintainers |
| End-to-end lifecycle | How e2e scenarios execute step by step |
| Parametric lifecycle | How parametric scenarios execute |
| Interface validation | API reference for validating intercepted traces |
AI tooling
| Topic | Description |
|---|---|
| AI integration guide | Built-in rules for AI-assisted development |
Additional requirements
Specific scenarios may require additional tools:
- Kubernetes tests -- require Kind/Minikube for local K8s clusters. See K8s docs.
- AWS SSI tests -- require AWS credentials and Pulumi setup. See AWS SSI docs.
Contributing
Before submitting a PR, always run the linter (./format.sh). Here are the most common types of contributions, ordered by frequency:
| What you want to do | Guide |
|---|---|
| Activate or deactivate a test for a library | Manifests, enable a test, skip tests |
| Add or edit a test | Add a new test, editing overview |
| Add or edit a scenario | Scenarios guide, scenarios overview |
| Add or edit a weblog | Weblog spec, build options |
| Other changes | Full editing docs, internals |
For testing against unmerged tracer changes, see enable-test.md and binaries.
Ownership
For file ownership see .github/CODEOWNERS.
Test ownership is defined using the @features decorator (see the feature doc and utils/_features.py).
Need help?
Drop a message in #apm-shared-testing -- we're happy to help!