End-to-End Tests

July 8, 2026 · View on GitHub

This directory contains end-to-end tests for ToolHive, including both CLI and HTTP API tests.

Overview

These tests validate ToolHive functionality by exercising the full application stack:

CLI Tests: Test command-line interface operations (run, list, stop, restart, etc.)
API Tests: Test HTTP API endpoints with a real API server instance
Integration Tests: Test interactions between different components

Structure

Test Files

*_test.go - Individual test files organized by feature
e2e_suite_test.go - Ginkgo test suite setup
api_helpers.go - Helper functions for starting API server and making HTTP requests
helpers.go - General helper functions for e2e tests
mcp_client_helpers.go - MCP client helper utilities
oidc_mock.go - Mock OIDC server for authentication tests
run_tests.sh - Test runner script

Operator-specific end-to-end tests live in subdirectories, such as the VirtualMCPServer E2E tests that run against a real Kubernetes cluster.

Test Categories

Tests are organized using Ginkgo labels for parallelization and filtering:

Core CLI Tests (Label: `core`)

Client management (client_test.go)
Group operations (group_*.go)
Server restart (restart_test.go)
Export functionality (export_test.go)
THVIgnore support (thvignore_test.go)

MCP Run Tests (Label: `mcp-run`)

MCP server operations (fetch_mcp_server_test.go, osv_mcp_server_test.go)

MCP Protocol Tests (Label: `mcp-protocol`)

Streamable HTTP (osv_streamable_http_mcp_server_test.go)
Remote MCP servers (remote_mcp_server_test.go)
Protocol builds (protocol_builds_e2e_test.go)
Inspector functionality (inspector_test.go, inspector_autocleanup_test.go)

Note: Both mcp-run and mcp-protocol tests also carry the parent mcp label, so LABEL_FILTER=mcp still runs all MCP tests locally.

Proxy Tests (Label: `proxy`)

Stdio proxy (proxy_stdio_test.go)
OAuth authentication (proxy_oauth_test.go)
Tunnel functionality (proxy_tunnel_e2e_test.go)
Streamable HTTP proxy (stdio_proxy_over_streamable_http_mcp_server_test.go)
SSE endpoint rewriting (sse_endpoint_rewrite_test.go)
Network isolation (network_isolation_test.go)

Middleware & Stability Tests (Label: `middleware || stability`)

Audit middleware (audit_middleware_e2e_test.go)
Authorization (osv_authz_test.go, http_pdp_authz_test.go)
Telemetry middleware (telemetry_middleware_e2e_test.go, telemetry_metrics_validation_e2e_test.go)
Stability tests (unhealthy_workload_test.go, health_check_zombie_test.go)

API Registry Tests (Label: `api-registry`)

Registry CRUD operations (api_registry_test.go)

API Workloads Tests (Label: `api-workloads`)

Workload endpoints (api_workloads_test.go)
Workload lifecycle (api_workload_lifecycle_test.go)

API Clients Tests (Label: `api-clients`)

Client management (api_clients_test.go, api_clients_validation_test.go)
Skills API (api_skills_test.go)

API Misc Tests (Label: `api-misc`)

Discovery API (api_discovery_test.go)
Groups API (api_groups_test.go)
Health check API (api_healthcheck_test.go)
Version API (api_version_test.go)
Secrets API (api_secrets_test.go)

Note: All api-* tests also carry the parent api label, so LABEL_FILTER=api still runs all API tests locally.

Running Tests

Prerequisites

Go installed
Ginkgo CLI installed: go install github.com/onsi/ginkgo/v2/ginkgo@latest
Docker, Podman, or Colima container runtime
ToolHive binary built (for CLI tests): task build

Run All Tests

cd test/e2e
./run_tests.sh

Run Tests by Label

cd test/e2e

# Run only core CLI tests
E2E_LABEL_FILTER=core ./run_tests.sh

# Run only API tests
E2E_LABEL_FILTER=api ./run_tests.sh

# Run only MCP protocol tests
E2E_LABEL_FILTER=mcp ./run_tests.sh

# Run proxy tests
E2E_LABEL_FILTER=proxy ./run_tests.sh

# Run middleware and stability tests
E2E_LABEL_FILTER='middleware || stability' ./run_tests.sh

Run with Ginkgo Directly

cd test/e2e

# Run all tests
ginkgo run --vv .

# Run specific label
ginkgo run --label-filter="api" .

# Run specific test file
ginkgo run --focus-file="api_healthcheck_test.go" .

Run from Project Root

# Run all e2e tests
task test-e2e

# Run with custom label filter
E2E_LABEL_FILTER=api task test-e2e

GitHub Actions Integration

The e2e tests run in parallel in GitHub Actions using label filters. The workflow:

Builds the ToolHive binary once and shares it across jobs
Runs tests in parallel using matrix strategy with 9 label-based buckets:
- core: Core CLI functionality (~57 specs)
- mcp-run: MCP server run tests (~33 specs)
- mcp-protocol: MCP protocol & inspector tests (~37 specs)
- proxy: Proxy tests (~25 specs)
- middleware: Middleware & stability tests (~28 specs)
- api-registry: Registry API tests (~41 specs)
- api-workloads: Workloads API tests (~56 specs)
- api-clients: Clients & skills API tests (~44 specs)
- api-misc: Discovery, groups, health, version, secrets API tests (~50 specs)
Uploads test results as artifacts

See .github/workflows/e2e-tests.yml for the full configuration.

Writing Tests

Adding New CLI Tests

Create a new test file (e.g., feature_test.go)
Add appropriate labels for categorization
Use existing helper functions from helpers.go
Follow the pattern of existing tests

Example:

var _ = Describe("Feature Name", Label("core", "e2e"), func() {
    It("should do something", func() {
        // Test implementation
    })
})

Adding New API Tests

Create a new test file (e.g., api_workloads_test.go)
Use the api label along with specific labels
Use e2e.StartServer() helper to start the API server
Make HTTP requests using the server's methods

Example:

var _ = Describe("Workloads API", Label("api", "workloads"), func() {
    var apiServer *e2e.Server

    BeforeEach(func() {
        config := e2e.NewServerConfig()
        apiServer = e2e.StartServer(config)
    })

    It("should list workloads", func() {
        resp, err := apiServer.Get("/api/v1beta/workloads")
        Expect(err).ToNot(HaveOccurred())
        defer resp.Body.Close()
        Expect(resp.StatusCode).To(Equal(http.StatusOK))
    })
})

Troubleshooting

Container Runtime Not Available

Ensure Docker, Podman, or Colima is running:

docker ps
# or
podman ps
# or
colima status

Binary Not Found (CLI Tests)

Build the ToolHive binary:

task build
# Binary will be at ./bin/thv

Set the binary path if needed:

export THV_BINARY=/path/to/thv

Test Timeouts

Increase the timeout:

TEST_TIMEOUT=20m ./run_tests.sh

Port Conflicts (API Tests)

API tests use random available ports by default. If you encounter port binding issues, the system will automatically find an available port.

Test Best Practices

Use descriptive labels - Make it easy to filter and run related tests
Clean up resources - Use DeferCleanup or AfterEach to clean up
Use unique names - Use GenerateUniqueServerName() for server names
Avoid hardcoded ports - Use random ports for API tests
Test isolation - Ensure tests can run independently
Meaningful assertions - Add context messages to assertions
Use Serial when needed - Mark tests as Serial if they can't run in parallel