Monitoring and Debugging

GARM provides built-in tools for monitoring, live log streaming, event watching, and an interactive terminal dashboard.

• Monitoring and Debugging
  • Prometheus metrics
    • Enable metrics
    • Generate a metrics token
    • Prometheus configuration
    • Metrics reference
      • Health
      • Webhooks
      • Entities (repositories, organizations, enterprises)
      • Providers
      • Pools
      • Scale sets
      • Runner instances
      • Jobs
      • GitHub/Gitea API
  • Live log streaming
    • Filtering logs
  • Database events
    • Event structure
    • Programmatic access
  • Interactive dashboard
  • Job monitoring
  • Reverse proxy considerations

Prometheus metrics

Enable metrics

In config.toml:

[metrics]
enable = true
disable_auth = false

Generate a metrics token

garm-cli metrics-token create

The token validity matches the time_to_live in [jwt_auth].

Prometheus configuration

scrape_configs:
  - job_name: "garm"
    scheme: https
    static_configs:
      - targets: ["garm.example.com"]
    authorization:
      credentials: "your-metrics-token"

Metrics reference

All metrics use the garm_ namespace. Metrics fall into two groups:

• Snapshot metrics are reset and recomputed on every tick (default every 60s, configured via period in [metrics]). These reflect the current state: pools, instances, entities, jobs.
• Cumulative metrics are counters or gauges updated as GARM operates: webhooks received, provider operations, GitHub API calls, rate limits.

Health

Metric	Type	Labels
garm_health	Gauge	metadata_url, callback_url, webhook_url, controller_webhook_url, controller_id

Set to 1 if GARM is healthy, 0 otherwise. Useful for alerting.

Webhooks

Metric	Type	Labels
garm_webhook_received	Counter	valid, reason

Increments on every webhook received from GitHub/Gitea. The valid label is true/false; reason explains why invalid webhooks were rejected.

Entities (repositories, organizations, enterprises)

Metric	Type	Labels
garm_repository_info	Gauge	name, id
garm_repository_pool_manager_status	Gauge	name, id, running
garm_organization_info	Gauge	name, id
garm_organization_pool_manager_status	Gauge	name, id, running
garm_enterprise_info	Gauge	name, id
garm_enterprise_pool_manager_status	Gauge	name, id, running

The _info gauges are always set to 1; the labels are what carry the information. The pool_manager_status gauges are 1 when the pool manager for that entity is running.

Providers

Metric	Type	Labels
garm_provider_info	Gauge	name, type, description

Pools

Metric	Type	Labels
garm_pool_info	Gauge	id, image, flavor, prefix, os_type, os_arch, tags, provider, pool_owner, pool_type
garm_pool_status	Gauge	id, enabled
garm_pool_max_runners	Gauge	id
garm_pool_min_idle_runners	Gauge	id
garm_pool_bootstrap_timeout	Gauge	id

Scale sets

Metric	Type	Labels
garm_scaleset_info	Gauge	id, scaleset_id, name, image, flavor, prefix, os_type, os_arch, tags, provider, runner_group, scaleset_owner, scaleset_type
garm_scaleset_status	Gauge	id, enabled, state
garm_scaleset_max_runners	Gauge	id
garm_scaleset_min_idle_runners	Gauge	id
garm_scaleset_desired_runner_count	Gauge	id
garm_scaleset_bootstrap_timeout	Gauge	id

The id label is GARM's internal scale set ID; scaleset_id is the numeric ID assigned by GitHub. garm_scaleset_desired_runner_count reflects the runner count GitHub has requested for the scale set (unique to scale sets, since GitHub drives scheduling).

Runner instances

Metric	Type	Labels
garm_runner_status	Gauge	name, status, runner_status, pool_owner, pool_type, pool_id, scaleset_id, provider
garm_runner_operations_total	Counter	operation, provider
garm_runner_errors_total	Counter	operation, provider

garm_runner_status covers both pool-owned and scale-set-owned runners. For any given series, exactly one of pool_id / scaleset_id is populated. pool_owner and pool_type describe the owning entity (repo/org/enterprise) and apply to both.

The operation label on garm_runner_operations_total / garm_runner_errors_total takes one of these values:

Operation	Description
CreateInstance	Create a new compute instance
DeleteInstance	Delete a compute instance
GetInstance	Get details about a compute instance
ListInstances	List all instances for a pool
RemoveAllInstances	Remove all instances created by a provider
Start	Boot up an instance
Stop	Shut down an instance

Jobs

Metric	Type	Labels
garm_job_status	Gauge	job_id, workflow_job_id, scaleset_job_id, workflow_run_id, name, status, conclusion, runner_name, owner, repository, requested_labels

GitHub/Gitea API

Metric	Type	Labels
garm_github_operations_total	Counter	operation, scope
garm_github_errors_total	Counter	operation, scope
garm_github_rate_limit_limit	Gauge	credential_name, credential_id, endpoint
garm_github_rate_limit_remaining	Gauge	credential_name, credential_id, endpoint
garm_github_rate_limit_used	Gauge	credential_name, credential_id, endpoint
garm_github_rate_limit_reset_timestamp	Gauge	credential_name, credential_id, endpoint

The scope label is Repository, Organization, or Enterprise. The operation label takes one of the values listed below.

GitHub client operations (hooks, runners, registration tokens):

Operation	Description
ListHooks	List webhooks on an entity
GetHook	Get a single webhook
CreateHook	Create a webhook
DeleteHook	Delete a webhook
PingHook	Ping a webhook
ListEntityRunners	List runners for an entity
ListEntityRunnerApplicationDownloads	List runner application downloads
RemoveEntityRunner	Remove a runner from an entity
CreateEntityRegistrationToken	Create a runner registration token
ListOrganizationRunnerGroups	List organization runner groups
ListRunnerGroups	List enterprise runner groups
GetEntityJITConfig	Generate a JIT runner configuration
GetRateLimit	Fetch API rate limit information

Scale set operations (scale set management and message queue):

Operation	Description
GetRunnerScaleSetByNameAndRunnerGroup	Look up a scale set by name and runner group
GetRunnerScaleSetByID	Look up a scale set by ID
ListRunnerScaleSets	List all scale sets
CreateRunnerScaleSet	Create a scale set
UpdateRunnerScaleSet	Update a scale set
DeleteRunnerScaleSet	Delete a scale set
GetRunnerGroupByName	Look up a runner group by name
GenerateJitRunnerConfig	Generate a JIT runner config for a scale set
GetRunner	Get a runner by ID
ListAllRunners	List all runners
GetRunnerByName	Get a runner by name
RemoveRunner	Remove a scale set runner
AcquireJobs	Acquire jobs for a scale set
GetAcquirableJobs	Get acquirable jobs for a scale set
GetActionServiceInfo	Get actions service admin info
CreateMessageSession	Create a message queue session
DeleteMessageSession	Delete a message queue session
RefreshMessageSession	Refresh a message queue session token
GetMessage	Get a message from the message queue
DeleteMessage	Delete a message from the message queue

Live log streaming

Stream GARM logs to your terminal in real time:

garm-cli debug-log

This requires enable_log_streamer = true in [logging].

Filtering logs

# Only ERROR level and above
garm-cli debug-log --log-level ERROR

# Filter by attribute
garm-cli debug-log --filter "pool_id=9daa34aa-..."

# Filter by message content
garm-cli debug-log --filter "msg=creating instance"

# Multiple filters (OR by default)
garm-cli debug-log --filter "pool_id=abc" --filter "pool_id=def"

# Multiple filters with AND
garm-cli debug-log --filter "pool_id=abc" --filter "msg=error" --filter-mode all

Database events

The debug-events command consumes database change events. Whenever an entity is created, updated, or deleted in the database, an event is generated and exported via WebSocket. This endpoint is designed for integration -- external tools can subscribe without polling the API.

Watch real-time entity changes:

# All events
garm-cli debug-events --filters='{"send-everything": true}'

# Only instance create/delete events
garm-cli debug-events --filters='{"filters": [{"entity-type": "instance", "operations": ["create", "delete"]}]}'

Available entity types: repository, organization, enterprise, pool, user, instance, job, controller, github_credentials, gitea_credentials, github_endpoint, scaleset

Operations: create, update, delete

Event structure

Each event is a JSON object:

{
    "entity-type": "instance",
    "operation": "create",
    "payload": { ... }
}

The payload contains the same JSON you would get from the corresponding REST API endpoint. Sensitive data (tokens, keys) is stripped. For delete operations, some entities return the full object prior to deletion while others return only the ID. Assume that future versions will return only the ID for all delete operations.

Programmatic access

The events endpoint is a WebSocket at /api/v1/ws/events. Connect with a JWT token and send a filter message to start receiving events. By default, the endpoint returns no events -- all events are filtered until you send a filter message:

// Receive all events
{"send-everything": true}

// Receive only specific entity/operation combinations
{
  "filters": [
    {"entity-type": "instance", "operations": ["create", "delete"]},
    {"entity-type": "pool", "operations": ["update"]}
  ]
}

See the events documentation for the full filter schema and a Go code example using garm-provider-common.

Interactive dashboard

The top command shows a live terminal dashboard:

garm-cli top

This displays entities, pools, scale sets, runner instances, and jobs in an interactive view, refreshing every 5 seconds.

Job monitoring

View recorded workflow jobs:

garm-cli job list

GARM only records jobs for which it has a matching pool or scale set. Jobs whose labels don't match any configured pool are silently ignored -- there's no point in recording jobs GARM can't act on. If you've set everything up but garm-cli job list is empty, verify that your webhook URLs are correct and that GitHub can reach them (see Controller settings).

Reverse proxy considerations

If GARM is behind a reverse proxy, the WebSocket endpoints need special configuration. For nginx:

location /api/v1/ws {
    proxy_pass http://garm_backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header Host $host;
}

This is required for debug-log, debug-events, top, and the Web UI. A full sample nginx config with TLS termination is available in the testdata folder.