Testing Guide -- cortex

August 9, 2026 · View on GitHub

Unit tests

cargo test

Shortcut: just test

Coverage summary:

just coverage       # cargo llvm-cov nextest --summary-only
just coverage-html  # cargo llvm-cov nextest --html

Install cargo-llvm-cov first if the command is missing:

cargo install cargo-llvm-cov --locked

Tests are colocated with source code in #[cfg(test)] modules:

Module	Tests cover
`src/config.rs`	Env var overrides, defaults, validation (host format, storage budget relationships, pool size)
`src/db.rs`	Schema init, insert/search/tail/errors/hosts/stats, FTS5 queries, retention purge, storage budget enforcement, batch retry
`src/syslog.rs`	RFC 3164/5424 parsing, UniFi CEF extraction, severity mapping, facility mapping, malformed input
`src/mcp.rs` + `src/mcp/`	Health endpoint, auth middleware (valid/invalid/missing token, no-auth mode), RMCP tool dispatch, timestamp validation, MCP lifecycle
`src/main.rs`	Background interval timing

Running specific tests

cargo test test_search           # Run tests matching "test_search"
cargo test config::tests         # Run config module tests only
cargo test -- --nocapture        # Show println/tracing output

Database tests use tempfile::TempDir for isolated SQLite instances. Each test gets a fresh database, preventing cross-test contamination. The StorageConfig::for_test() helper provides minimal config with pool_size=1 and WAL mode disabled.

Live smoke tests

Live tests run against a running cortex server:

just test-live
# or: bash tests/test_live.sh

The smoke test (scripts/smoke-test.sh) exercises all cortex actions via mcporter. tests/test_live.sh additionally covers live UDP and TCP syslog ingest, MCP tool calls, CLI parity between local SQLite and --http, REST surface parity, and a deterministic admin POST /api/file-tails status/list path when CORTEX_API_ADMIN_TOKEN is set. Docker log ingest is split by operational path: host-local agent Docker streaming is covered by agent deployment parity tests, while the legacy central pull path uses the mocked Docker HTTP fixture in src/docker_ingest/client_tests.rs. Compose diagnostics are non-mutating and are validated only for redacted shape, so the smoke test can pass on either Docker-backed or non-Docker deployments. When seeding is enabled, the smoke scripts import tests/fixtures/ai-session-smoke.jsonl and assert that sessions, search_sessions, abuse, abuse_incidents, abuse_investigate, ai_correlate, and project_context can retrieve real AI transcript rows, not just empty response envelopes. scripts/smoke-ai-mcp.sh additionally seeds a temporary transcript and calls the HTTP MCP endpoint for search_sessions, abuse, abuse_incidents, abuse_investigate, usage_blocks, project_context, list_ai_tools, and list_ai_projects. The AI smoke scripts resolve CORTEX_BIN first, then cortex on PATH, then the repo-local debug binary at target/debug/cortex, so repo-local builds do not require an installed shell binary.

Action registry covered by live/script references: search, filter, tail, errors, hosts, map, host_state, fleet_state, correlate_state, topic_correlate, sessions, search_sessions, abuse, abuse_incidents, abuse_investigate, ai_correlate, usage_blocks, project_context, list_ai_tools, list_ai_projects, correlate, stats, status, apps, source_ips, timeline, patterns, context, get, ingest_rate, silent_hosts, clock_skew, anomalies, compare, compose_status, compose_doctor, unaddressed_errors, ack_error, unack_error, notifications_recent, file_tails, notifications_test, llm_invocations, similar_incidents, incident_context, graph, skill_events, skill_incidents, skill_investigate, mcp_events, mcp_incidents, mcp_investigate, hook_events, hook_incidents, hook_investigate, help.

mcporter-based testing

# List available tools
mcporter list cortex --config config/mcporter.json

# Call actions through the single cortex tool
mcporter call --config config/mcporter.json cortex.cortex action=stats
mcporter call --config config/mcporter.json cortex.cortex action=status
mcporter call --config config/mcporter.json cortex.cortex action=tail n=10
mcporter call --config config/mcporter.json cortex.cortex action=search query=error limit=5
mcporter call --config config/mcporter.json cortex.cortex action=hosts
mcporter call --config config/mcporter.json cortex.cortex action=host_state host_id=host-id
mcporter call --config config/mcporter.json cortex.cortex action=sessions
mcporter call --config config/mcporter.json cortex.cortex action=abuse terms=ai-smoke-authentication limit=5
mcporter call --config config/mcporter.json cortex.cortex action=abuse_incidents limit=3
mcporter call --config config/mcporter.json cortex.cortex action=abuse_investigate limit=1
mcporter call --config config/mcporter.json cortex.cortex action=correlate_state reference_time=2026-01-01T00:00:00Z window_minutes=10
mcporter call --config config/mcporter.json cortex.cortex action=ai_correlate project=/tmp/cortex-sessions-smoke limit=2 events_per_anchor=3
mcporter call --config config/mcporter.json cortex.cortex action=apps
mcporter call --config config/mcporter.json cortex.cortex action=source_ips
mcporter call --config config/mcporter.json cortex.cortex action=timeline
mcporter call --config config/mcporter.json cortex.cortex action=patterns
mcporter call --config config/mcporter.json cortex.cortex action=context host=host timestamp=2026-01-01T00:00:00Z
mcporter call --config config/mcporter.json cortex.cortex action=get id=1
mcporter call --config config/mcporter.json cortex.cortex action=ingest_rate
mcporter call --config config/mcporter.json cortex.cortex action=silent_hosts
mcporter call --config config/mcporter.json cortex.cortex action=clock_skew
mcporter call --config config/mcporter.json cortex.cortex action=file_tails op=status
mcporter call --config config/mcporter.json cortex.cortex action=anomalies
mcporter call --config config/mcporter.json cortex.cortex action=compare a_from=2026-01-01T00:00:00Z a_to=2026-01-01T01:00:00Z b_from=2026-01-01T01:00:00Z b_to=2026-01-01T02:00:00Z
mcporter call --config config/mcporter.json cortex.cortex action=compose_status
mcporter call --config config/mcporter.json cortex.cortex action=compose_doctor
mcporter call --config config/mcporter.json cortex.cortex action=graph mode=entity entity_type=host key=example-host
mcporter call --config config/mcporter.json cortex.cortex action=graph mode=around entity_type=host key=example-host depth=1
mcporter call --config config/mcporter.json cortex.cortex action=graph mode=explain entity_type=host key=example-host depth=2
mcporter call --config config/mcporter.json cortex.cortex action=graph mode=evidence evidence_id=12345

For graph proof UX smoke, use a real bounded evidence id from graph around --limit 5, then assert:

relationship.src_entity_id and relationship.dst_entity_id are preserved,
relationship.src_entity and relationship.dst_entity are present,
source_log_summary is either compact and bounded or null with missing_source_reason,
response JSON contains no raw field and no metadata_json,
auth-like strings, URL userinfo, home paths, private-key markers, and terminal controls are redacted/escaped.

CLI-based testing (abuse investigation workflow)

The deterministic abuse incident/investigation workflow is also exercisable directly through the cortex binary. All outputs are bounded; the investigation findings are rule-based and local-only (never an external LLM analysis).

# Group abuse hits into scored incident candidates (bounded; capped result set)
cortex sessions incidents --limit 3 --json

# Expand the top incidents into deterministic evidence bundles + findings
cortex sessions investigate --limit 1 --json

# Heartbeat fleet state parity commands
cortex state host --host nashost --json
cortex state fleet --json
cortex correlate state --reference-time 2026-01-01T00:00:00Z --window-minutes 10 --json

# Graph lookup commands
cortex entity host nashost --json
cortex graph around host nashost --limit 25 --json
cortex graph explain host nashost --depth 2 --json

curl-based testing

# Health check
curl http://localhost:3100/health

# Tail recent logs
curl -s -X POST http://localhost:3100/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"tail","n":10}}}'

# Search
curl -s -X POST http://localhost:3100/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"search","query":"error","limit":5}}}'

# Stats
curl -s -X POST http://localhost:3100/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"stats"}}}'

# Status
curl -s -X POST http://localhost:3100/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"cortex","arguments":{"action":"status"}}}'

Testing checklist

CI configuration

Tests run automatically in CI via GitHub Actions:

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: dtolnay/rust-toolchain@stable
      - run: cargo test

Test coverage

See tests/TEST_COVERAGE.md for detailed coverage documentation.