v1.3

July 28, 2026 · View on GitHub

The durable findings from the v1.3 investigation. The working records with full evidence chains live in tmp/ and are not tracked; this is the part worth keeping.

Three diagnoses made from documentation during this work were wrong, and each was settled in minutes by a live instance. That is why docker-compose.test.yml exists — see TESTING.md.

Cold start: a 9-second hang on every page

Cause: src/routes/+layout.server.js runs on every route, including the unauthenticated /login, and it awaited an outbound ROMM call. When ROMM was unreachable the call had no timeout, so the first byte of every page waited for a TCP timeout.

The rules this produced are in ENGINEERING_RULES.md under Server-side performance rules. The two most easily re-broken:

AbortSignal.timeout() throws TimeoutError, not AbortError. A catch branch checking for AbortError silently misclassifies every timeout.
Streaming needs CSP mode: "nonce" in svelte.config.js. Under "hash", streamed inline <script> chunks are emitted after headers flush and get blocked, so streamed sections never resolve. While 'unsafe-inline' remains in script-src SvelteKit emits neither hashes nor nonces and the mode is inert — but it must be correct before 'unsafe-inline' is removed.

Request coalescing was added, then reverted. Handing later callers an in-flight promise deadlocks if the cached function calls withCache() with the same key — the nested call awaits its own outer promise forever. The root layout did exactly that: it wrapped getUserPermissions() under user-permissions-${auth_type}-${id}, and that function caches internally under the identical key. Every authenticated request hung indefinitely before load() ran, while /login benchmarked at 33 ms throughout, so it went unnoticed until the containerized stack was driven with a logged-in session. Making it safe needs AsyncLocalStorage, unavailable because cache.js is bundled for the browser. Duplicating a rebuild is cheap; deadlocking is not.

ROMM: the library silently disappeared

Cause: POST /api/token returns 403 {"detail":"Insufficient scope"} — authentication fails outright and no token is ever issued. The predicted failure (a 200 with an empty array from /api/roms) was wrong, and so was the follow-up theory that the RomM 5.0 response schema had changed. It had not; the parser had been reading fields RomM never returned.

Nothing was visible anywhere because the error was swallowed by a catch that returned []. Hence engineering rule 3.

Notes that cost time to establish:

Role alone does not revoke library access. RomM coerces viewer to user, and that role still carries roms.read. Use the 5.0 permission API: PUT /api/permissions/users/{id} with {"overrides":[{"entity":"roms","action":"read","granted":false}]}.
RomM enforces CSRF on writes — fetch the romm_csrftoken cookie and echo it in an x-csrftoken header.
Client API Tokens return the secret once, in raw_token. They do not expire, which is why ROMM_API_TOKEN is now the recommended path over the password grant.

OIDC: `AUTH_METHOD=oidc_generic` never worked

The pre-1.3 implementation was hardcoded to Authentik's URL scheme, with no discovery document, no JWKS, and no id_token validation. Claim mapping was hardcoded and actively harmful: an absent groups claim reset is_admin to false, demoting admins on every login against any IdP that does not emit one. Keycloak does not emit groups without an explicit Group Membership mapper, so this reproduced immediately. Issues #4 and #7.

ggr_users.authentik_sub keeps its column name — a deliberate decision, not an oversight. ~15 query sites reference it. Treat it as a generic OIDC subject.

Three auth defects found while tracing this, all fixed in v1.3: an unsigned basic-auth session token, a published JWT signing-key fallback ("your-secret-key"), and csrf.checkOrigin disabled globally.

Upgrade verification: v1.2.5 → 1.3.0

Run twice against the real v1.2.5 tag via make test-upgrade-old FROM=v1.2.5 → make test-upgrade-new — first on the tree at 6e2c15f, then re-run after the deployment-script removal. All eight assertions passed both times, with an identical two-line diff.

An admin was created through v1.2.5's own POST /api/auth/basic/setup, then a game request, a watchlist entry, a system setting and a cache row were written into the v1.2.5 schema. Snapshots of ggr_migrations, per-table counts, every seeded row and the ggr_user_preferences column list were taken before and after the swap.

The whole before/after diff was two added lines:

> 008_add_animated_background_preference.sql | 2026-07-28 02:35:58.876209 | t
> animated_background | boolean | false

#	Assertion	Result
1	App boots on the old database	`/api/health` → 200
2	001 and 002 skipped	`⏭️ Skipping … (already executed)` for both
3	008 runs exactly once	`🎉 Executed 1 migrations successfully!`
4	Nothing re-ran	every pre-existing `executed_at` byte-identical; the 12 v1.2.5 rows stayed at `02:34:37`, only 008 carries `02:35:58`
5	Data survives	request, watchlist entry, setting and admin all unchanged
6	No row loss	every table count identical
7	008 applied	`animated_background boolean DEFAULT false` present
8	Basic auth works after upgrade	fresh login succeeds

Assertion 8 is a fresh login by design — 1.3.0 signs basic-auth session tokens, so pre-upgrade basic_auth_session cookies are deliberately invalidated.

Also smoke-tested on the upgraded database: the new /request page loaded (200) and rendered the request created on v1.2.5, and clearing the games cache emptied ggr_games_cache while leaving the watchlist row and the request's igdb_id intact.

What this does not cover. Only v1.2.5 → HEAD, only AUTH_METHOD=basic, and only the happy path. Installs originating from 1.1.x carry the migrations/legacy/ series in their history and are untested. It also does not exercise the ggr_migrations drop-on-mismatch path — it cannot, because that path never fires on a healthy database. See the assessment below.

The `ggr_migrations` drop: assessed, deferred to 1.3.1

db-manager.js:251-261 drops ggr_migrations (with a gratuitous CASCADE) when the table exists but is missing any of id, migration_name, executed_at, success, checksum. This was assessed as a possible 1.3.0 ship blocker and deliberately deferred. The reasoning, so it is not re-litigated from the one-line summary:

Not a 1.3.0 regression. The code is byte-identical in v1.2.3 (where it was introduced, bf494c2) and v1.2.5. With AUTO_MIGRATE defaulting on, it has run on every Docker boot for two releases with no issue reported.
Unreachable from any shipped schema. Both creators — migrations/001_initial_schema.sql:5 and the CREATE TABLE in db-manager.js:264 — emit the same eight columns, and 001 is identical across v1.0.3, v1.1.0, v1.1.4, v1.2.0, v1.2.3 and v1.2.5. No released version can produce a table that trips the check.
The replay would be harmless. If it did fire, 001 is re-marked executed and 002/008 replay. All three are idempotent: every INSERT carries ON CONFLICT … DO NOTHING, 002 uses DROP CONSTRAINT IF EXISTS and CREATE TABLE IF NOT EXISTS, and 008 is ADD COLUMN IF NOT EXISTS. The cost is a lost audit trail and reset timestamps — not data loss, and not a failed boot.
The entrypoint does not do this. An earlier note claiming docker-entrypoint.js:224-251 drops and recreates the table was wrong; the entrypoint delegates to db-manager.js migrate.

Deferred because the fix touches the migration bootstrap, and the upgrade test cannot exercise the changed path — it never fires on a healthy database. Shipping an unverified change to migration handling to close a hole no released version can reach is the worse trade.

For 1.3.1: make the repair additive (ALTER TABLE … ADD COLUMN IF NOT EXISTS per missing column) instead of dropping, remove the CASCADE, make fixMigrationTable() repair rather than recreate, and add the regression test that does not exist today — seed a ggr_migrations missing checksum and assert the history survives.

Known limitations that are not bugs

Secure cookies on plain HTTP. Session cookies set Secure when NODE_ENV=production, so a deployment with no TLS anywhere — not even terminating at a proxy — fails the OIDC state check, and API-issued session cookies are discarded by the client. Correct behavior, predates v1.3. Note that /api/auth/basic/login sets Secure while the /login/basic form action does not, so the two login paths behave differently over plain HTTP.
PM2 runs one worker per core and each warms its own cache at boot. On a 6-core host that is 6 simultaneous warm-ups, each sweeping stale cache rows plus IGDB calls. The sweep is scoped — a 7-day age cutoff, and it skips anything on a watchlist — so this is wasted work rather than data loss. Not a regression, but N parallel warm-ups risk rate limiting on larger hosts. Worth a single-flight guard across workers.
src/lib/auth.server.js reads env at module top-level, so those values freeze at first import despite the $env/dynamic/private import. Runtime env changes need a restart.

Outstanding

Carried forward from the phase records. None block the v1.3 release.

Item	Detail
ROMM fetch duplication	Three implementations — `romm.server.js`, `admin/api/settings/test-romm`, `api/setup/check` — should collapse into one
No timeout on IGDB in `api/setup/check`	Its two `fetch` calls to Twitch and IGDB still have no `AbortSignal`, violating engineering rule 2. The ROMM call was fixed
PKCE and `nonce`	Neither exists. The `state` compare in `callback/+server.js` is a plain string compare, not constant-time
Permissions cache not invalidated on login	Group changes take up to 15 minutes to apply
Client errors are swallowed	`hooks.client.js` exports a `handleError` with an empty body, so every client-side error is discarded silently
`getPlatforms` assumes a bare array	`romm.server.js` parses `/platforms` as a bare array while the other endpoints were made response-shape tolerant
ROMM health not surfaced in admin	`admin/settings` shows no ROMM status, so a misconfigured integration is only visible in container logs
`db-manager.js fix` drops migration history	The `fix` subcommand (`npm run db:fix`) drops and recreates `ggr_migrations` unconditionally, losing the audit trail. Deliberate operator action only, but it repairs nothing that is broken. See the assessment below
Skeletons only on the home page	`/search`, `/game/[id]`, `/profile` and `/admin` still block on their loads

Closed since

ROMM sessionToken as an unbounded module global. It now tracks expires_in, renews ahead of expiry, single-flights concurrent authentication, and is discarded on 401, 403 or 5xx rather than 401 alone. This was the cause of the recurring "ROMM 500s until the container is restarted" reports: RomM answers an expired JWT with 500, so the only invalidation path never fired.
Negative-cache TTL. The availability snapshot now escalates 5 → 10 → 20 → 30 minutes across consecutive failures, and ROMM requests fail fast while a failure is still cached.