Operations Guide

Health Checks

Three health endpoints are exposed:

Endpoint	Purpose	Probe mapping
/livez	Always returns 200 if the HTTP server is responsive.	livenessProbe
/readyz	Returns 503 when session count exceeds HEALTHZ_MAX_SESSIONS.	readinessProbe
/healthz	Deprecated alias of /readyz. Will be removed in a future release.	—

/livez

• Always 200 OK with {"status":"ok"}.
• Use for liveness: the event loop is alive and the server can serve HTTP.

/readyz

• 200 OK with {"status":"ok","sessions":<n>} when healthy.
• 503 Service Unavailable with {"status":"unhealthy","reason":"session_limit_exceeded","sessions":<n>} when active sessions exceed HEALTHZ_MAX_SESSIONS (default: 10000).
• Use for readiness: the pod should be removed from the service rotation (no restart) when overloaded.

/healthz (deprecated)

Alias of /readyz. Retained for backward compatibility during the 0.7.x cycle. Will be removed in 0.8.0.

Kubernetes Probe Configuration

livenessProbe:
  httpGet:
    path: /livez
    port: 3000
  initialDelaySeconds: 5
  periodSeconds: 30
readinessProbe:
  httpGet:
    path: /readyz
    port: 3000
  initialDelaySeconds: 3
  periodSeconds: 10

Environment Variables

Variable	Default	Description
PORT	3000	HTTP listen port
HOST	127.0.0.1	Bind address. Set to 0.0.0.0 to expose to the network — requires AUTH_MODE=oauth, otherwise startup refuses (GHSA-8jr5-6gvj-rfpf).
USE_SSE	false	Enable legacy SSE transport
USE_STREAMABLE_HTTP	false	Enable MCP Streamable HTTP transport
AUTH_MODE	pat (app default) / oauth (chart default)	pat requires loopback bind. oauth accepts any bind and validates Authorization: Bearer per request, with sessionId bound to the originating Bearer hash.
GITLAB_API_URL	https://gitlab.com/api/v4	GitLab API base URL
GITLAB_PERSONAL_ACCESS_TOKEN	—	Required in pat mode
GITLAB_READ_ONLY_MODE	false	Restrict to read-only tools
CORS_ALLOW_ORIGINS	—	Comma-separated allowed origins. Empty = * only when AUTH_MODE=pat AND bind is loopback. Network-exposed binds require an explicit allowlist or get no Allow-Origin header.
HEALTHZ_MAX_SESSIONS	10000	Session count threshold for unhealthy status

Auth × bind matrix

The HTTP transports carry no authentication of their own in AUTH_MODE=pat. The server's safety properties are a function of two axes — see SECURITY.md for the full threat model.

AUTH_MODE	HOST	Outcome
pat	loopback (127.0.0.0/8, ::1, localhost)	OK for local dev. Wildcard CORS permitted.
pat	non-loopback	Refused at startup. CWE-306 / CWE-942.
oauth	any	OK. Authorization: Bearer required on every request. SessionId bound to SHA-256 of the originating Bearer; reuse with a different (or missing) Bearer is rejected with 401.

Troubleshooting

Server returns 401 on every request

• OAuth mode: Verify the upstream gateway is forwarding the Authorization: Bearer <token> header.
• PAT mode: Ensure GITLAB_PERSONAL_ACCESS_TOKEN is set and the token has not expired.

High session count / memory growth

Active sessions are stored in-memory. If session count grows unbounded:

1. Check that clients are properly closing SSE connections or sending DELETE /mcp.
2. Consider lowering HEALTHZ_MAX_SESSIONS and adding a horizontal pod autoscaler based on the /readyz response.

CORS errors in browser console

In OAuth mode, Access-Control-Allow-Origin: * is intentionally not sent. Set CORS_ALLOW_ORIGINS to the exact origin(s) of your frontend application.

Release atomicity & recovery

The release pipeline has two independent legs that can succeed or fail independently:

• build.yml — runs on push/tag and publishes the Docker image to ghcr.io when a tag is pushed. Helm chart is packaged and pushed on tag pushes as well.
• publish.yml — runs only on release.published or workflow_dispatch and publishes the npm package via OIDC Trusted Publishing.

A successful release means both legs landed: npm has the new version, ghcr has the new image, and (on tag pushes) the Helm chart is in the OCI registry. When one leg fails, the other already-published artifact is immutable and must not be rewritten — recovery means re-running the failed leg so the two sides converge.

Recovery paths

npm publish succeeded but Docker push failed

1. Verify npm: npm view @yoda.digital/gitlab-mcp-server@<version>
2. Re-run Build & Publish against the tag:

   gh run list --workflow=build.yml --branch v<version>
   gh run rerun <run-id> --failed
   

3. Verify the image landed:

   docker pull ghcr.io/yoda-digital/mcp-gitlab-server:<version>
   gh api /orgs/yoda-digital/packages/container/mcp-gitlab-server/versions | jq '.[0].metadata.container.tags'
   

Docker push succeeded but npm publish failed

1. Verify the ghcr image is live (commands above).
2. Inspect why publish-npm failed:

   gh run list --workflow=publish.yml --limit 5
   gh run view <publish-run-id> --log-failed
   

   Common cause: Trusted Publishing configuration drift on npmjs.com. Check Package settings → Trusted publishers → verify the workflow filename, environment name, and repository all match. See [reference_npm_publish_404_trap lessons in memory] for an instance where a misleading 404 PUT masked an auth-config issue.
3. Re-trigger publish via workflow_dispatch:

   gh workflow run publish.yml --ref v<version>
   

Both legs failed

1. Investigate the root cause before re-triggering (GitHub outage, registry incident, dependency failure). Don't blindly re-run if the underlying environment hasn't recovered.
2. If the tag + release artifacts are still valid, re-trigger both legs as above.
3. If the tag itself is bad (e.g. version in package.json doesn't match the tag, or the release was published from the wrong SHA): do NOT delete the tag or release — they may be cached downstream. Instead, bump the version, create a new tag, and document the skipped version in CHANGELOG.

Verifying a full release post-recovery

# npm side
npm view @yoda.digital/gitlab-mcp-server@<version> --json | jq '{version, dist: .dist | {tarball, integrity, signatures}}'
# .dist.signatures present = Sigstore provenance attached via Trusted Publishing

# Container side
docker pull ghcr.io/yoda-digital/mcp-gitlab-server:<version>
docker inspect ghcr.io/yoda-digital/mcp-gitlab-server:<version> --format='{{.Config.Labels}}'

Both should report the expected version. Note recovery actions in CHANGELOG.md if the convergence took more than the standard ceremony — operators downstream may need that context when bisecting.

First-publish runbook: ghcr.io package

When ghcr.io/yoda-digital/mcp-gitlab-server is first published, GitHub creates the container package in private mode by default and does not link it to the source repository. Both need to be fixed manually once per package, after the first successful build.yml run that pushes an image.

One-time setup steps

1. After the first successful build, navigate to: https://github.com/orgs/yoda-digital/packages/container/mcp-gitlab-server/settings
2. Under Manage Actions access, add the mcp-gitlab-server repository with the Write role. This is what lets subsequent CI runs push to this package without an additional OAuth dance.
3. Under Danger Zone → Change visibility, set to Public. Without this, anonymous docker pull returns 403.
4. Under Repository, link the package to yoda-digital/mcp-gitlab-server so the README, license, and source link show up on the package landing page.

Verification

# Anonymous pull should succeed
docker pull ghcr.io/yoda-digital/mcp-gitlab-server:latest

# Package metadata should report visibility + source repo
gh api /orgs/yoda-digital/packages/container/mcp-gitlab-server \
  | jq '{visibility, repository: .repository.full_name, html_url: .html_url}'

Expected:

{
  "visibility": "public",
  "repository": "yoda-digital/mcp-gitlab-server",
  "html_url": "https://github.com/orgs/yoda-digital/packages/container/package/mcp-gitlab-server"
}

Failure mode: "denied" on subsequent CI pushes

If build.yml succeeds initially but later runs start failing with denied: permission_denied on the push step, the Actions access from step 2 has dropped. This can happen if:

• The repo was renamed or moved.
• The org's default Actions permissions tightened.
• A maintainer was removed and the package's per-repo grant was tied to their account.

Fix: re-apply step 2.

Verifying the image (cosign + SBOM + provenance)

Every published image carries a Sigstore keyless signature, a SLSA provenance attestation, and a SPDX SBOM attached to its manifest. Downstream operators — compliance, security review, FedRAMP/ENISA procurement — can verify all three against ghcr.io. Verification requires network access to the Sigstore Rekor transparency log; it is not a true offline check.

Prerequisites

# Install cosign (https://docs.sigstore.dev/cosign/system_config/installation/)
go install github.com/sigstore/cosign/v2/cmd/cosign@latest
# or via Homebrew:
brew install cosign

Verify the signature

IMAGE="ghcr.io/yoda-digital/mcp-gitlab-server:latest"

cosign verify \
  --certificate-identity-regexp '^https://github\.com/yoda-digital/mcp-gitlab-server/\.github/workflows/build\.yml@refs/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  "$IMAGE"

Expected output: a JSON payload listing the signed claim, OIDC identity (https://github.com/yoda-digital/mcp-gitlab-server/.github/workflows/build.yml@refs/...), and Rekor transparency-log entry. A non-zero exit means tampering, a Rekor outage, or that you pulled an image not produced by this repo's pipeline.

Download the SBOM

cosign download sbom "$IMAGE" > sbom.spdx.json
jq '.packages[] | {name: .name, version: .versionInfo}' sbom.spdx.json | less

The SBOM is SPDX 2.3 JSON and lists every OS package, Node module, and base-image layer that contributed to the final image. Use it to answer "are we affected by CVE-XXXX-YYYY in dependency Z?" without rebuilding locally.

Inspect the provenance attestation

cosign download attestation "$IMAGE" \
  | jq -r '.payload' \
  | base64 -d \
  | jq '.predicate'

The provenance follows the SLSA in-toto schema (mode=max) and records the workflow source ref, builder identity, and build inputs.

Multi-arch manifest

The published manifest contains both linux/amd64 and linux/arm64. Verify with:

docker manifest inspect "$IMAGE" \
  | jq '.manifests[] | {platform: .platform, digest: .digest}'

Apple Silicon, AWS Graviton, and ARM-based Kubernetes nodes will pull the native arm64 layer automatically.

CI smoke gate

Each build.yml run pushes the image, then immediately runs cosign verify against the freshly-signed digest. On tag releases (the trust boundary), a failed verify hard-aborts the workflow before downstream consumers can pull the bad artifact. On main and feature branches, verify is retried with backoff and soft-warns (non-blocking) — this absorbs the well-known Rekor inclusion lag (10–30s in pathological cases) without freezing developer iteration.

Sigstore outage — temporary release path

When Sigstore (Rekor or Fulcio) is impaired and a release must ship anyway:

1. In .github/workflows/build.yml, comment out the Install cosign, Sign image with cosign (keyless OIDC), and Verify cosign signature (CI smoke with Rekor lag tolerance) steps for the duration of the outage. Do not delete them.
2. Push the workflow change to main BEFORE cutting the tag.
3. In the GitHub Release notes (and CHANGELOG.md under [Unreleased]), prepend:

   Released during Sigstore outage: this image is not cosign-signed. Verify provenance via the SHA256 digest published below and against the GitHub Actions run URL.

4. Publish the image digest from the workflow run summary in the release notes so consumers can pin by digest.
5. Once Sigstore recovers, re-enable the cosign steps in build.yml (revert step 1), then cut a no-op patch release (0.x.y+1 with only a CHANGELOG.md entry) that ships a signed image of the same code. Announce the resumed signing in that release's notes.

Do not paper over a Sigstore outage with --insecure-ignore-tlog — that defeats the supply-chain claim.

Identity rotation (org rename, workflow file move)

The cosign verify identity regex above is anchored on three things downstream consumers will pin:

• Org name: yoda-digital
• Repo name: mcp-gitlab-server
• Workflow path: .github/workflows/build.yml

Changing any one silently invalidates every consumer's cosign verify script. If a move is unavoidable:

1. Before the move: open an issue + CHANGELOG.md entry under [Unreleased] announcing the new identity regex (the post-move URL) one release ahead of time.
2. After the move: tag a no-op release immediately so consumers have a known-good signed image under the new identity. Update the example in this section of OPERATIONS.md.
3. For consumers: maintain a deprecation issue on the old identity for at least 90 days, linking to the new regex. Encourage pinning by cosign verify --certificate-identity-regexp '<old>|<new>' during the transition window.