Operations
July 24, 2026 ยท View on GitHub
This section is for the platform team after the first Forge tenant is running. It covers tenant support, image management, artifacts, cleanup, secrets, upgrades, and troubleshooting.
Day-2 Loop
| Cadence | Action | Doc |
|---|---|---|
| Every tenant request | Collect tenant values, add config, run plan, run smoke workflow. | Tenant Onboarding |
| Every image release | Build base/custom images, share AMIs, update tenant runner specs. | Runner Images |
| Weekly | Run example apply/destroy for helpers, infra, platform, integrations. | Workflow Blueprints |
| Weekly | Run cleanup and policy jobs. | Cloud Custodian |
| Monthly | Review module refs, Renovate output, AMI age, stale ECR tags, and secrets. | Upgrades |
| Planned ARC upgrade | Rebuild blue/green EKS clusters and move tenants one at a time. | Move ARC Tenants |
| Incident | Triage queued jobs, failed runner registration, IAM, webhooks, or ARC. | Troubleshooting |
| Incident | Debug Terraform, OpenTofu, or Terragrunt plan/apply that appears stuck. | Terraform/Terragrunt Stuck Runbook |
| Incident | Use Splunk dashboards to identify the failing subsystem and severity. | Splunk Dashboard Runbook |
| Incident | Triage Forge metrics, dependencies, resource pressure, and detectors. | Splunk Observability Dashboard Runbook |
Operating Repos
If you need an end-to-end operating model, copy from Operations Repo Blueprints. The blueprints include real folders for Packer, Ansible, containers, Renovate, Cloud Custodian, Terragrunt, reusable actions, and weekly example deployments.
For Splunk-based operations, start with the Splunk Dashboard Runbook. Use the panel reference when you need to map a dashboard panel back to its operational question.
For Splunk Observability metrics and detectors, use the Splunk Observability Dashboard Runbook and Splunk Observability Dashboard Panel Reference.
For installations that do not deploy Splunk, use Troubleshooting Without Splunk as the baseline support runbook.