Conventions

Every service depends on these shapes; a mismatch is an integration bug rather than a local one. okf_core is the shared library that encodes most of them — import from it instead of re-implementing.

S3 bundle bucket layout (source of truth)

okf/<data_domain>/
├── _domain/overview.md              # type: Domain (declared-domain concept doc)
└── <dataset>/
    ├── index.md                     # auto-generated (regenerate_indexes)
    ├── datasets/<dataset>.md        # type: Glue Database
    ├── tables/<table>.md            # type: Glue Table (one per table)
    ├── references/<type>/<slug>.md   # type: Reference — canonical fact-typed
    │                                 #   folders: joins/ metrics/ enums/
    │                                 #   named_sets/ glossary/ known_issues/
    │                                 #   recipes/ (mandatory query transforms,
    │                                 #   authored only when a dataset needs one)
    │                                 #   computations/ (type: Attested
    │                                 #   Computation — see "Attested
    │                                 #   Computations" below)
    │                                 #   (one doc per item; see okf-authoring skill)
    ├── external/<d>/<ds>/…           # type: Cross-Dataset Reference — one subtree
    │                                 #   per counterpart dataset, written ONLY by a
    │                                 #   cross-mode harvest (see "Cross-dataset
    │                                 #   references" below); overview.md + the same
    │                                 #   fact-typed folders (joins/ metrics/ …)
    ├── .context/                     # user-uploaded source docs (persisted)
    ├── .metadata/                    # read-only Glue metadata snapshot (per run)
    │                                 #   (+ .metadata/external/<d>/<ds>/ on a cross
    │                                 #   run: the target's snapshot + published docs)
    └── .harvest/state.json           # commit marker (status: complete | in_progress)
                                      #   + graph.json (precomputed /graph artifact)

The _domain/overview.md doc is a derived materialisation of the declared domain's description + context. Written THROUGH the harvest mount (uid 1000) on PUT /domain-defs/{domain} so the <domain>/ directory is established with correct ownership before any dataset-level write. _domain is a reserved pseudo-dataset that parse_bundle_key parses normally (3 segments: domain/_domain/overview.md) and reindex embeds with type=Domain. Hidden from the dataset listing by is_domain_dataset(). Vector key: <domain>/_domain/overview.

• Concept id is the path under okf/<domain>/<dataset>/ minus .md, e.g. tables/races. Use the okf_core.paths helpers.
• S3 object key is okf/<domain>/<dataset>/<concept_id>.md.
• Vector key is <domain>/<dataset>/<concept_id> — the S3 key without the okf/ prefix and .md suffix. Use okf_core.embedding.vector_key.
• .context/, .metadata/, and .harvest/ are dot-prefixed and are not concepts. The reindex worker ignores any key with a dot-prefixed segment below the dataset root, and ignores index.md and log.md.
• .metadata/ is a read-only Glue metadata snapshot the harvest writes ONCE at the start of each run (harvest/metadata_export.py): index.md (manifest), database.md, columns.tsv (one line per table\tcolumn\ttype\tcomment — the cross-table grep target for join/near-synonym discovery), and tables/<t>.md per table. The agent reads it with the built-in read_file/glob/grep (it replaced the old list_concepts/read_concept_raw tools); the OKF write-guard refuses any write into it. Live verification stays on the sample_rows/ run_sql tools. Like .context/, it is a harvest INPUT and is never published, indexed, or embedded; clean_authored_output preserves it (dot-prefixed), and export_metadata rewrites it fresh each run so a dropped table leaves no stale sheet.
• A bundle is consumable only once .harvest/state.json exists with status == "complete".
• .harvest/graph.json is the precomputed link-graph artifact the Control API's GET /bundle/{d}/{ds}/graph serves: {completed_at, nodes, edges}, built by okf_core.graph_json.build_graph_json. finalize_bundle writes it just BEFORE the commit marker, stamped with the SAME completed_at the marker carries — the endpoint serves it iff the two match (status == "complete" and equal timestamps), so every mutating mode (full, scoped/ incremental, annotation, cross) refreshes it by construction, and repromote rewrites it too (restore_snapshot only touches non-dot .md files, so the stale artifact would otherwise linger). Any mismatch — mid-run marker, legacy bundle, failed precompute — falls back to computing the same JSON live from the docs (parallel S3 fetch; one shared builder keeps the two paths identical). Derived data: writes are best-effort and never fail a harvest or repromote.
• .harvest/review/ holds the review workflow's state: clusters.json (the persisted clustering with stable ids — a run_review(cluster_ids=[...]) retry re-runs those clusters on THIS clustering, never a recomputed one; when the run recorded context digests it also carries a context section pairing them into x1..xN groups under the same retry contract) and one report-<id>.md per run_review call (unique name, nothing overwritten) with every reviewer/fixer transcript. Like the rest of .harvest/, it is state — never published, indexed, or embedded. A FULL harvest wipes .harvest/review/ at start (alongside the authored-output wipe): a clustering from a previous run describes docs the run is about to rebuild, so nothing may retry against it.
• .harvest/context/ holds digest-NN.md — the verbatim output (dispatch brief + returned digest) of every context-extractor dispatch, recorded automatically as each completes (harvest/context_digests.py, fed from the QuickJS shim and the static task path). run_review's context-fidelity phase audits the bundle against these after all cluster fixes; no digests means the phase is skipped. Run-scoped on both ends: finalize_bundle deletes the dir right AFTER the commit marker (the audit is over; a run that fails earlier keeps them for debugging), and a full harvest wipes it at start anyway — the guard for a predecessor that crashed before its own cleanup.

Derived artifacts live OFF the mount prefix. Sibling top-level prefixes sit next to okf/ in the same bucket and are deliberately NOT under it, so nothing an LLM role's file tools can reach ever sees them: benchmark/<domain>/<dataset>/ (the gold-carrying questions CSV + report artifacts — see "Benchmark Studio invocation" below), policy/<domain>/<dataset>/ (the policy-check artifacts — see "Policy checks (LLM-judge engine)"), and the export/import staging exports/<domain>/<dataset>/ + imports/<domain>/<dataset>/ (fixed-key zip

• validation record — see "Bundle export & import"). All are derived or transient: the bundle stays the source of truth, and deleting a dataset purges every one of these prefixes along with it (versions included; a lifecycle rule also ages out the staging prefixes' noncurrent versions).

Cross-dataset references (external/)

external/<counterpart_domain>/<counterpart_dataset>/… holds docs representing knowledge that SPANS this dataset and one counterpart (verified cross-dataset joins, cross-dataset metrics, the pair overview) — Roadmap §5's OSS flat-trust mode. Rules:

• The pair docs have exactly ONE home: the bundle of the dataset whose cross harvest authored them. Nothing is ever written into the counterpart's bundle. This is the load-bearing decision: a mirrored copy would make the pair a distributed fact across two independently versioned, independently restorable bundles, so a full harvest OR a repromote of the counterpart (restoring a version from before/after a sync) would silently desynchronize it, with no transaction able to span the two. One home means a dataset's version history is self-contained and pair state cannot drift.
• Written ONLY by a mode="cross" harvest; the write guard confines a cross run to exactly its pair subtree, and every other mode never touches external/.
• The counterpart's discoverability is a DERIVED signal, not a copy — see "Cross-dataset reference signal" below.
• Every doc carries type: Cross-Dataset Reference and a cross_dataset: {source: {data_domain, dataset}, target: {…}} frontmatter block (source = the initiating side, i.e. where the docs live). Prose is symmetric (read by consumers of BOTH datasets) and tables are named as qualified SQL identifiers.
• Links go to BOTH sides — a link is an address, and addresses may go stale. Home-side docs are linked file-relative as usual (from a joins/ doc: ../../../../tables/<t>.md) — these resolve in the per-bundle link graph and stitch the pair subtree into it (backlinks from a table surface its cross-dataset joins). Counterpart docs are linked with the bundle-ESCAPING relative form (../../../../../../<td>/<tds>/tables/<t>.md): all bundles share one okf/ tree, so the address resolves when the tree is browsed as files, and the UI's Browse view follows it into the other dataset. The link resolver (okf_core/links.py) deliberately DROPS bundle-escaping links from the graph (OKF tolerates dangling cross-bundle links), and a re-harvest of the counterpart may dangle the address — accepted; the qualified SQL identifier in the prose is the durable reference.
• The docs are ordinary published concepts: listed, served, embedded, and searchable exactly like the rest of the bundle (concept id external/<d>/<ds>/joins/<slug> etc.). Nothing downstream special-cases them beyond the annotation scope filter below.
• A full harvest deletes external/ along with everything else (clean_authored_output's delete-every-non-dot-entry rule — there is deliberately no keep-list). Re-run the cross harvest to restore the pair docs; vectors and the XREF signal are pruned/rebuilt through the normal reindex event path.
• A cross re-run of the same pair replaces that pair's subtree wholesale; other pairs' subtrees are untouched.

Attested Computations (references/computations/)

Frozen, parameterized, read-only SQL authored at harvest and executed by filling typed @parameter holes — never by editing the statement (design: docs/ATTESTED_COMPUTATIONS.md; pure rules: okf_core/computations.py; shared S3/engine runner: okf_aws/computation_run.py).

• Doc contract (write-guard-enforced): type: Attested Computation, directly under references/computations/ (flat; the filename is the slug); runtime: athena | redshift; parameters list (name/type/required, example required, optional default/enum/min/max/column; an optional parameter must carry a default); ONE SELECT/WITH statement in a ```sql fence under # Computation, every @hole declared and every declared parameter used.
• Content hash: sha256 over the fence text (trailing-whitespace-stripped lines), the canonical JSON of parameters, and the runtime string (computation_sha256). Human verification signs this hash; any edit changes it and the stamp reads stale.
• Verification triple (verified / verified_by / verified_sha256): null until a human acts. Agents can never set them — the guard refuses any write that sets a non-null value it isn't PRESERVING verbatim from the existing doc. A verified computation is FROZEN in the in-place modes (incremental / annotation / cross): the runner resolves the verified set at run start (harvest/verification.frozen_computation_paths — folded stamps ∪ overlay, hash-checked), the guard refuses agent write/edit/delete on those docs, and lint downgrades their findings to warnings ("a human must unverify"). Human Unverify is the only unlock. A full harvest is exempt — clean_authored_output is unconditional (no keep-list) and the agent re-authors from source, so verification returns to the human's queue unless the fence was reproduced verbatim AND its overlay click had not yet folded in. Verify/Unverify flips land in the off-mount overlay verification/<domain>/<dataset>.json ({version, entries: {slug: {slug, sha256, verified, verified_by}}}; unverify writes a revoked tombstone) — never on the doc: the mount is the bundle tree's sole writer. The runtime folds the overlay into doc frontmatter inside finalize_bundle (every mode, after authoring, before the commit marker — harvest/verification.py); serving merges doc + overlay with the overlay winning, and a hash mismatch on either side surfaces as stale.
• Execution surfaces (one runner, three fronts): consumption MCP tools list_computations / describe_computation / run_computation (execution gated by OKF_COMPUTATIONS_ENABLED ⟵ var.enable_attested_computations, default false); chat's run_computation (ALWAYS bound; executes only under var.enable_chat_sql's grants, policy checker when armed); Control API GET/POST /bundle/{d}/{ds}/computations[/{slug}][/run|/verify|/unverify] (verify identity comes from the JWT, never the body). Declared enum/min/max are CONTRACT (refused); profiled domains (.metadata/profile/domains.json, written by the profile pass) are ADVISORY (warn-and-run). Every run returns the receipt: executed_sql (verbatim), computation_sha256, verification, engine_query_id, warnings.
• Lint: the computations step validates doc shape, column bindings against the snapshot, and declared enums against the profile evidence; the EXPLAIN gate re-enters each valid computation with its example values substituted (raw @hole fences classify as templated and are never sent).

Bundle versions & repromote

The bundle bucket is versioned, and finalize_bundle writes .harvest/state.json LAST — so version history needs no manifest: a bundle version is one status: "complete" object version of that marker, identified by the marker's own S3 VersionId and labeled by its completed_at. The file set of a version is reconstructed on read (okf_aws.s3_versions): for every non-dot .md under the dataset prefix, the newest object version with LastModified <= the marker's (absent if that entry is a delete marker). in_progress marker writes delimit nothing and are filtered out — an interrupted (cancelled/crashed) harvest therefore never becomes a version; its half-written live state is inspectable via the diff to=live sentinel and rolled back by repromoting the last good version.

Endpoints (Control API): GET /bundle/{d}/{ds}/versions, GET /bundle/{d}/{ds}/diff?from=&to= (both optional — defaults answer "what changed in the last harvest"; to=live compares against the working files), POST /bundle/{d}/{ds}/repromote {version_id}, and GET .../repromote (the convergence poll). The built-in chat agent additionally gets a get_bundle_diff tool (same module, agent-bounded output) — deliberately NOT registered on the consumption MCP server, so external agents see only the published bundle, never its history.

Repromote and the S3 Files mount. The repromote's two state.json writes (in_progress + the fresh complete marker) are plain PutObject calls from the Control API — they carry none of the POSIX file-mode metadata the harvest runtime's S3 Files mount stores on mount-written objects, so the mount presents the marker READ-ONLY afterward. harvest.fsutil.write_text heals this (EACCES → unlink + rewrite; the parent dir is mount-created and writable), so the next harvest of a repromoted dataset proceeds normally. Restored docs are unaffected: CopyObject preserves the source object's metadata.

Repromote is append-only: every file of the target version is CopyObject-ed from its source VersionId onto the same key (S3 mints NEW current versions — old ids are never resurrected), live docs absent from the target get a delete marker, and a FRESH complete marker is written carrying repromoted_from (the restored marker VersionId) + repromoted_by (caller identity). The untouched reindex pipeline converges the vector index from the resulting object events. Repromote deliberately does NOT touch the freshness table's Glue-version rows: it is a content rollback, not a pin — the next genuine catalog change (or manual harvest) legitimately overwrites it.

Retention: var.bundle_version_retention_days (durable stack, default 90) lifecycle-expires noncurrent bundle versions — this IS the repromote window — while always keeping the 3 newest noncurrent versions per key. Expired versions simply drop out of the reconstructed list (nothing dangles). SAFETY COUPLING: lifecycle expiry emits Object Deleted events with deletion-type "Permanently Deleted" for keys whose live doc is untouched; the reindex worker MUST keep filtering those (only "Delete Marker Created" reaches DeleteVectors) or daily expiry would delete live docs' vectors.

Bundle export & import

A bundle is a portable artifact: export zips the PUBLISHED tree (the same rule the file endpoints serve — non-dot .md only, so .context/.metadata/ .harvest and the zero-byte dir markers stay out, and computation VERIFICATION — off-mount by design — cannot travel), rooted at <dataset>/ with a top-level manifest.json sibling ({format, data_domain, dataset, exported_at, files} — the only machine-checkable provenance: docs carry no dataset identity, which is exactly what makes them portable). POST /bundle/{d}/{ds}/export stages the archive at the FIXED off-mount key exports/<d>/<ds>/okf-bundle.zip (overwritten per export; the versioned bucket keeps history) and returns a presigned GET with an attachment disposition.

Import is repromote with the snapshot swapped for an uploaded zip, three endpoints under POST /bundle/{d}/{ds}/import[...]:

1. presign (/import) — a presigned POST pinned to the off-mount staging key imports/<d>/<ds>/upload.zip, size-capped by the POST policy itself. 404 unless the dataset is REGISTERED (the registry row is the lease anchor); a never-harvested registered dataset is a fine target — that is the point (move a bundle between deployments without re-running the authoring).
2. validate (/import/validate) — dry-runs the staged archive and names every problem at once (the content checks run even when structural blockers already exist — no second refusal round), in two deliberate tiers: blockers (zip traversal/absolute paths, dot-prefixed and agent-scratch segments per is_reserved_rel_segments, non-markdown, non-UTF-8, unreadable/corrupt/encrypted members, duplicate entry names, file-vs-directory path collisions, wrong root folder or manifest dataset, size/count caps) which apply refuses outright, and findings — the SAME offline okf_core.lint steps a harvest gate runs (coverage self-skips: no .metadata travels) plus live EXPLAIN of the archive's runnable ```sql against THIS deployment's catalog (budget- capped, skips and engine TIMEOUTS counted honestly as unchecked — a slow engine is never reported as a defect in the doc's SQL; external/ fences are skipped like the harvest gate skips them — a counterpart's tables are outside this dataset's database) — which are the human's call. Caps bind on ACTUAL decompressed bytes via bounded streaming reads (64MB archive / 4MB per doc / 128MB total / MAX_RESTORE_FILES docs) — the zip's declared sizes are never trusted. The report carries archive_sha256, and validate persists a small record (imports/<d>/<ds>/validate.json: digest + finding counts) that apply requires.
3. apply (/import/apply {acknowledged, archive_sha256}) — refuses blockers (400), refuses an archive with NO validation record matching the staged digest (409 — validate first; the record is what lets the acknowledged gate cover the live-EXPLAIN findings apply cannot re-derive, and a client-supplied archive_sha256 additionally pins what the human reviewed), refuses unacknowledged findings (409), then: harvest lease with mode="import" → in_progress marker → write docs + delete live published .md the archive lacks (dot-dirs survive, and non-markdown strays are LEFT ALONE — version snapshots are .md$-\text{only}, \text{so} \text{deleting} \text{one} \text{would} \text{be} \text{irreversible}) \text{in} \text{parallel} (\text{serial} \text{writes} \times \text{hundreds} \text{of} \text{docs} \text{would} \text{blow} \text{the} 30\text{s} \text{Lambda}) → \text{fill} \text{MISSING} $index.md files deterministically (authored indexes travel and are preserved — the index regen never deletes) → graph.json → ensure_dir_markers over the written keys + the marker key (a marker-less directory mounts READ-ONLY — without this the next harvest dies EACCES) → freshness seeding → fresh complete marker → IMPORT row → release → policy-rebuild signal → staged zip + record deleted (consumed). Reindex needs nothing: the object events re-converge the vector index.

The synthesized marker carries tables (derived from the archive's top-level tables/ docs) and table_versions seeded from the LIVE Glue catalog for exactly the tables the archive documents — apply also writes those tables' TABLE#…/VERSION freshness rows (the item shape incremental.store.put_stored_version owns), which is what stops the nightly reconcile from seeing every imported table as drifted and re-authoring the bundle the import just delivered. Catalog tables the archive does NOT document stay unseeded, so the reconcile authors them; seeding is best-effort (a Glue failure degrades to the re-author-everything behavior, never a failed import). Also on the marker: imported_by (JWT identity), imported_files, imported_sha256, and imported_export (the manifest's exported_at, only when present). Imported computation docs land UNVERIFIED until a human re-verifies.

Policy checks (LLM-judge engine)

A per-dataset policy document (policies.yaml) — one individually trackable, checkable policy per entry — authored from the wiki's reference docs by a harvest-runtime agent, and enforced MID-TURN by the chat runtime's map-reduce fleets of LLM judges, in two tracks selected per run from the composer's Policy feature. Judge flags are advisory: they ride back into the model's own context as hedged <system-reminder>s (the model decides the correction) and surface in the UI as shield timeline steps; nothing blocks or gates.

Underneath the judges sits a deterministic tier ("Deterministic policy rules" below): computational policies may carry declarative rules that are evaluated on the query's own syntax tree BEFORE the engine runs. It is advisory like everything else — a proven violation rides back as an unhedged proof-backed reminder (never a block) — and a policy it decides never reaches the judges. Everything it cannot prove falls through to the fleet unchanged.

Naming note: the ar_ prefixes below (module okf_aws.ar_policy, the registry attributes, mode="ar_rules") are legacy naming from the retired v1 engine (Bedrock Guardrails Automated Reasoning), kept to avoid a data migration. v2+ involves no Bedrock control-plane resources and no region limitation — the judges run wherever the chat model runs.

The document (pure format: okf_core.policy_doc): a YAML mapping with a top-level policies list; every entry has exactly {id, type, condition, action, source} plus, on computational entries only, an optional rules list (see "Deterministic policy rules"). id is P-prefixed, unique, and STABLE across edits (the UI tracks policies over time; the author agent keeps surviving ids and never reuses retired ones). type is exactly one of computational (the violation is visible in a SQL query itself — additivity, grain/collapse, fan-out joins, sentinel decoding; judged by the query-time check) or behavioural (a process rule — ask-before-committing, refuse, require scope; judged against the agent's steps). A hybrid (computational core + disclosure duty) is tagged computational. condition = when the policy applies, plain language over a turn's conduct; action = the checkable obligation; source = the references/….md wiki page it traces to (the author gate refuses dead pages). Count backstop OKF_POLICY_MAX_RULES (default 60) and a 50k-char doc cap guard against enumeration pathology. Migration is the self-heal: a pre-split document (no type) fails the parse, which flags the row stale and re-authors it — no code path needed.

Authoring lifecycle. ALWAYS ON per dataset (the v1-era ar_enrolled opt-in is retired; the only switch left is the deploy-wide OKF_POLICY_BUILD_ENABLED). The triggers, not a flag, bound the work: the runner's post-complete follow-on build authors on every committed bundle change (full, incremental, annotation — AFTER the terminal status write; see "Harvest status"), policy_rebuild events cover the rest (the Reasoning page's manual Sync, the repromote accelerator — which also first-authors a dataset restored to a version without a document — and the chat check's stale discovery), and the nightly reconcile re-verifies ONLY datasets whose lifecycle has begun (ar_build_status present). A dataset predating the feature is therefore never bulk-backfilled: its first document comes from a manual Sync, its next harvest/increment, or a repromote. Authoring runs on the harvest runtime — as the runner's follow-on step (best-effort, AFTER the commit marker AND the terminal status write) and on mode="ar_rules" dispatches — as a small ReAct agent with full reasoning (harvest.ar_author): it reads the sources (with per-file unified diffs against the previous authoring's copies), then submits through a validating write_policies tool. The pipeline is short: gather sources → fingerprint-skip (unchanged sources + live document = zero model calls) → flip building (THE per-dataset lease; conditional UpdateItem with a BUILD_LOCK_STALE_SECONDS takeover so a dead author's row cannot lock authoring out) → author → persist the document + author state → stamp_ready with the gather-time fingerprint, then one freshness re-check (a wiki that moved mid-authoring flags the row stale and queues a rebuild — never a mislabelled document). Authoring IS completion — there is no build workflow and no completion authority.

From-scratch runs are map-reduce (first authoring + forced Sync — any run with no prior document). Measured 2026-08-03 (hifa): three single-pass from-scratch runs over identical sources each produced a DIFFERENT incomplete rule set — extraction recall is an attention problem, not a model-tier one. So: harvest.ar_clusters splits the sources into deterministic topic clusters (path + frontmatter-tag routing; ~8 files / ~25k tokens max, thin topics fold into a catch-all, the confidentiality/disclosure topic and the usage_guardrails.md contract stay ISOLATED however thin — isolation is the remedy for the pages single-pass kept dropping); a fleet of per-cluster extractors (classifier contract: thinking off / reasoning "none", FORCED submit_rules tool call, schema + source-attribution validation with bounded retry, salvage-on-exhaustion) mines candidate rules in parallel; and the authoring agent becomes the SYNTHESIZER — it receives the candidate union, dedupes/merges/prunes (proportionality is its explicit job, backed by the OKF_POLICY_MAX_RULES cap), verifies doubtful candidates via read_source, and owns the single write_policies gate as before. The gate adds a one-shot coverage nudge on full-mode submissions: a valid document that leaves non-index sources uncited is staged AND answered with the uncited list — one extract-or-affirm round trip. Everything is fail-open (fleet failure → plain single-pass authoring); UPDATE runs never fan out (minimal diffs + stable ids are exactly right there). Code-level env: OKF_POLICY_FANOUT (default on; false kills the fleet), OKF_POLICY_EXTRACT_CONCURRENCY (default 4).

The rebuild authority (incremental.ar_rebuild, reached by policy_rebuild events and the nightly reconcile) makes only deterministic decisions: registration/bundle-ready/fingerprint checks, re-dispatch on change/missing document, reaping rows stalled at building past a 1h grace period, and a deterministic recovery — when the persisted document's sources manifest fingerprint matches the live wiki but the ready stamp was lost, it re-stamps instead of re-authoring. Dispatches always mint a FRESH runtime session id (runtime_session_id(..., unique_token=uuid)): an AgentCore session is pinned to the runtime version it started on, so a deterministic id would reattach a post-deploy retry to pre-deploy code. Dedup comes from the building flip, never from session affinity.

Off-mount artifacts — under a top-level prefix SIBLING to okf/ (the benchmark/ precedent): outside the reindex rule's filter and unreachable by any LLM role's file tools.

policy/<data_domain>/<dataset>/
├── policies.yaml          # the authored policy document (the judges' rubric
│                          #   and the Reasoning page's per-policy list)
├── rules_schema.json      # {version, databases: {db: {table: [columns]}}} —
│                          #   the QUALIFICATION schema for `rules:` blocks,
│                          #   snapshotted at authoring time from the harvest's
│                          #   .metadata/columns.tsv (the schema the wiki was
│                          #   authored against, NOT live Glue). Absent = the
│                          #   author gate refuses rules; the chat tier
│                          #   evaluates nothing and the judges carry the load
├── sources_manifest.json  # {fingerprint, files: {rel: sha256}} — what the
│                          #   author last SAW…
└── sources/<rel>          # …and its exact copies: the DIFF BASE. The next
                           #   authoring run hands the agent per-file unified
                           #   diffs against these, so an incremental/
                           #   annotation harvest yields a surgical EDIT of
                           #   policies.yaml (stable ids), not a rewrite

All derived and rebuildable from the bundle; deleting a dataset purges the prefix.

Per-dataset state lives on the registry DATASET# row (pk = "DOMAIN#<data_domain>", sk = "DATASET#<dataset>" — the mapping row), as flat scalars: {ar_build_status, ar_source_hash, ar_pending_source_hash, ar_build_started_at, ar_built_at, ar_build_detail}. Statuses: building (the lease) / ready / failed / stale. An absent ar_build_status means the lifecycle has never begun — the reconcile skips such rows (no bulk backfill) and the chat check's dataset discovery ignores them. Retired attrs (ar_enrolled, the v1 Bedrock AR set) may linger on rows written by older deploys; nothing reads them, and dataset deletion removes the row itself.

The document is a DERIVED artifact of the bundle, exactly like the vector index. S3 markdown is the only truth; a document authored from anything but the CURRENT wiki state must never be judged against. Enforcement is by fingerprint, not by trusting event hooks: the check recomputes the live source hash (okf_core.ar_sources owns which files are policy material and the fingerprint) and compares it to the stamped one. A mismatch renders NO verdict — the dataset reports "rebuild pending", the row is flagged stale, and a policy_rebuild event is published (both best-effort), so the click starts the repair.

The checks (chat/policy_check.py, v3 — the post-turn panel, policy_check request type, POLICY# report rows, and synthesizer are all REMOVED; the shield timeline steps are the only policy surface):

• Per-run opt-in. The composer's "+" menu gains a Policy field with side options Computational / Behavioural / Strict (= both), carried on the run envelope as features: ["sql", "policy:computational" | "policy:behavioural" | "policy:strict"]. The Policy field REQUIRES the SQL feature — the UI disables it while SQL is off and unchecking SQL also unchecks Policy; the server enforces the same dependency independently (chat.sql.normalize_features drops orphaned policy:* values). The deploy flag OKF_CHAT_POLICY_CHECK_ENABLED remains the master gate above the opt-in. No selection ⇒ no checker constructed ⇒ zero cost.
• One turn-scoped checker serves both tracks (policy_check.PolicyChecker, built by the server per run — it needs the thread id): handed to the SQL tool (computational) and the behavioural middleware; shared state = the curated question, policy caches, budgets, an 8-worker pool with a lock over the shared caches (concurrent checks — parallel tool-called queries, a behavioural eval alongside — must not serialize while their callers' wait budgets burn). Fleet mechanics are shared: policies shard ≤ OKF_CHAT_POLICY_SHARD_SIZE (default 10) per mini-judge, packed by SOURCE page (shard_policies keeps one wiki page's policies in one judge's shard — shared vocabulary/context; a group never straddles a boundary, so grouping can cost an extra shard); judges are CLASSIFIERS on EVERY family — thinking OFF + temperature 0 on a Converse (Anthropic) id, reasoning "none" on an openai.* id — with a FORCED report_violations tool choice (legal on Anthropic exactly because thinking is off), single fast pass. They answer through exactly ONE tool, report_violations, whose contract is violated policy IDS ONLY — no evidence, no explanations (saves judge output tokens, and the reminder the main agent reads is built from the AUTHORED policy text — condition/action/source — so judge prose never pollutes its context). Ids outside the judged set are dropped; a judge that won't call the tool after one retry fails open as a missing shard. No second verification round on either track — the flags ship hedged ("CAN include false positives — use your own judgment") and the receiving agent, holding the wiki pages and the live context, is the verifier.

The rolling curated question (both tracks judge against it): turn 1 is the raw question at zero model cost — but it is still PERSISTED as the curated question (the seed the next turn chains from). Curation runs on EVERY gated turn, armed or not (the checker is built whenever the deploy gate is on; an unarmed run's checker exists solely as the chain's keeper), so enabling Policy mid-conversation chains from real state instead of starting over. A later turn runs ONE minimal-effort rewrite — (previous curated question, previous final answer, current raw question) → curated question — kicked ASYNC at run start (the server's prewarm(), right after any clarification fold) so it races the model's own thinking and the queries; evaluators never wait on it — a verdict fired before the rewrite lands judges against the raw question, and the rewrite still persists in the background for the next turn. A previous ANSWER alone (an armed turn that ran no SQL never seeded the question) still triggers the rewrite. An answered ask_human clarification IS folded in, inline within the turn (the resume rebuilds the agent; the server hands the checkpoint's raw question + the Q&A to the fresh checker, which re-runs the rewrite) — the ask-first evidence lives in the behavioural steps track, and the computational judges need the clarified intent. Durability: the state lives on the THREAD row (policy_curated_question, written when the rewrite lands; policy_last_answer, ~2k chars, written at stream end; policy_question_history, the last 3 curated questions rolled forward with each landing), read with one lazy GetItem — so reloading a chat days later still chains context. The history widens the REWRITE's resolution context: the questions before the previous one (at most 2) render as an earlier-questions section in the rewrite input, so a fragment jumping back past the immediately previous topic still curates to the right question. The judges never see them — both tracks judge against the single current curated question. Every piece is best-effort and fail-open: absent attributes mean raw-question (turn-1) semantics.

The computational track (the run_sql race):

• Which dataset's policies: an @-scoped run pins it — which covers every REDSHIFT run by construction (the Redshift engine can only be built from the scope's source descriptor). An UNSCOPED Athena run has no default database, so the model must schema-qualify every table — the checker reads the schemas in FROM/JOIN position (extract_sql_schemas; aliases, CTE names, string literals, and comments can't match) and resolves them against the policy-bearing datasets' Glue map (ar_build_status present; glue_database attr, dataset id fallback — same contract as scope enrichment; one filtered registry scan, cached per turn). A cross-dataset join is judged against each matched dataset; a query touching only policy-less data is skipped for free.
• Race, don't block: run_sql submits the check before dispatching the query (the judges spend the query's own execution time), then waits at most OKF_CHAT_POLICY_QUERY_TIMEOUT_S (default 60) AFTER the results are back — but ONLY for analytical queries (should_wait); exploration probes submit without waiting. The cold start is PREWARMED at run start (the server calls checker.prewarm() right after the clarification fold): each piece — judge-model build, the curated-question rewrite, the freshness gate / policy-dataset map — is its own parallel pool task behind its own once-guard, and nothing ever waits on the rewrite: evaluators fire the moment they are triggered and take the curated question only if it has already landed, else the turn's raw question (_curated_now). A verdict that isn't ready in time is dropped — advisory, never a bottleneck.
• Exploration is free: a deterministic structural gate (is_analytical_sql) judges only queries that compute something the policies govern (aggregates, windows, joins, unions). Row peeks, DISTINCT value enumeration, SHOW/EXPLAIN/DESCRIBE, and information_schema probes are never judged. Identical (whitespace-normalized) SQL is served from a per-turn cache, and at most OKF_CHAT_POLICY_QUERY_MAX_PER_TURN (default 3) queries per turn are judged at all. Only the COMPUTATIONAL subset of the document is judged here.
• Delivery: flagged violations ride back in the SAME tool result as the rows, as a <system-reminder> after the payload (the sql_anomalies channel). The model decides the correction: revise the query, address it in the answer, or set a misread flag aside. For display, the server splits the reminder back OUT of the tool content (split_policy_reminder) into a typed {"type": "policy"} chunk — live and on history reload — which the UI renders as its own thinking-timeline step (shield marker, finding lines only); the model always sees the full string.

The behavioural track (policy_check.BehaviouralPolicyMiddleware, a before_model hook in the same slot as SteeringMiddleware): before_model runs exactly once after ALL parallel tool results return, so two analytical queries fired in parallel produce ONE evaluation covering everything so far — batching for free, no debounce. When new successful analytical run_sql results exist since the last eval, the middleware kicks ONE eval over the steps-so-far (the curated question, the ask_human Q&A verbatim, tool calls + result summaries — never thinking blocks or injected notes), judged against the BEHAVIOURAL subset with committed-conduct framing (computing on an unresolved ambiguity is committed; not-having-asked-YET is not) — then WAITS for the verdict (bounded by the same OKF_CHAT_POLICY_QUERY_TIMEOUT_S budget; asyncio.to_thread on the async path so the event loop never blocks) and injects it before THIS model call as a HumanMessage carrying additional_kwargs["okf_policy"] (the steering pattern — merged into the tool-result user message, never a cache invalidation, never a user bubble on reload), surfacing as the same typed policy chunk / shield step. The wait is the delivery guarantee: the model step right after an analytical batch is usually the one that writes the FINAL answer, so a fire-and-forget eval loses that race and its verdict never reaches the model at all (observed live 2026-08-02). The checker is prewarmed at run start (see "Race, don't block" above; the middleware's first hook remains a second, idempotent caller), so the wait is normally just the fleet. A verdict slower than the budget rolls to the next hook, or drops at turn end (advisory, logged). One eval in flight at a time; a policy flagged once this turn is never re-flagged (no nagging).

Shared invariants: the freshness gate runs once per turn per dataset (stale ⇒ never judged + flag_stale + policy_rebuild publish — the same self-heal that migrates pre-v3 documents), mid-turn framing on both prompts (the answer doesn't exist yet; answer-stage obligations are not violations), the ids-only judge contract, and fail-open everywhere: any failure returns the results/turn untouched.

Control API routes (Cognito-authed, same Lambda):

• GET /reasoning/{domain}/{dataset} — everything the Reasoning page renders: wiki_ready, status, up_to_date (the fingerprint gate for humans), the live source list, and the document's policies (id/condition/action/source).
• POST /reasoning/{domain}/{dataset}/sync — one FORCED policy_rebuild event (refused without a complete wiki). Sync is "author now": the event carries force: true, which bypasses the rebuild-iff-changed skip at BOTH hops (rebuild authority and harvest-side trigger) — a manual Sync's sources may be unchanged while the authoring itself moved on (model/effort/prompt), and without force a Sync on a ready dataset acknowledged "queued" then silently did nothing (live 2026-08-03). A forced run also authors FROM SCRATCH — the prior document is withheld (update mode's "minimally edit" would hand it straight back); automatic rebuilds keep update mode (minimal diffs, stable ids). Doubles as the manual FIRST authoring for a dataset predating the feature (there is no bulk backfill). The building lease still wins (an in-flight run is never preempted; the UI disables Sync while a build is live), and a stalled building row is reaped and re-dispatched.
• GET /reasoning/{domain}/{dataset}/document — {exists, text}: the live policies.yaml verbatim (kept for raw API access; the UI page renders the parsed per-guardrail list from the status call instead).

Deterministic policy rules

A computational policy entry may carry an optional rules: list — the mechanically checkable core of its action, evaluated on the agent's SQL by okf_core.policy_rules (sqlglot) at run_sql call time. The evaluation is SUBMITTED to the checker pool before the engine dispatches (its gate loads race the query, never serialize in front of it) and JOINED after the results return; the judge path waits on the in-flight evaluation so shard exclusions stay deterministic.

Tri-state contract. Per rule: violation (proven — every implicated column resolved to a real (database, table) through the schema and the forbidden shape is present / the required one absent), pass (proven clean), or unknown (anything unprovable: parse/qualify failure, a column flowing through a CTE or derived source whose name overlaps the bindings, OR at the top of a WHERE, a set-operation ORDER BY, no sqlglot). Renames don't evade: same-scope aliases (SUM(points) AS total, ds.points) are resolved away by qualification and still prove violations, while a derived-scope projection that renames a bound column (points AS pts in a CTE, SUM(pts) outside — including expression wrappers like points + 0 AS pts) is tracked as an export alias and reads unknown, never a decisive pass. A policy is violation if any of its rules proves one, else unknown if any is undecidable, else pass. The tier only ever flags-with-proof or stays silent — its worst case is coverage loss, never a wrong flag.

The closed dimension catalog (okf_core.policy_rules.DIMENSIONS — the author may only BIND these; an unknown dimension is a gate error and a policy fitting none stays prose-only): forbidden_aggregation, forbidden_usage, forbidden_function, required_predicate, required_guard, forbidden_grouping, forbidden_sequencing_key, required_distinct. Two evaluator families — expression-local (one ancestor walk after qualification) and scope analysis (conjuncts of WHERE + HAVING + INNER JOIN ON; outer-join ON never satisfies a required predicate, since it does not filter the preserved side). Matching is engine-faithful and equivalence-aware: literals compare the way Athena compares them (numbers numerically, strings case-SENSITIVELY, booleans case-insensitively), NOT (col = v) satisfies a neq, negative literals resolve, function names match Trino spellings underscore-insensitively (date_diff ⇒ sqlglot's datediff), cast types accept ANSI aliases (integer/int, real/float, decimal/numeric — REAL is in the default guard set), comparison conjuncts IMPLY is_not_null (NULL fails every comparison, so d = date(…) satisfies a required not-null — no false flag on bounded queries), required_predicate has a bounded op (any positive equality/range/IN conjunct on the column, whatever the value — the "must pin an explicit window" shape), forbidden_grouping fires on GROUP BY keys only (ordinals and expressions over the target included; reading/ filtering stays legal, DISTINCT deliberately out of scope), || is transparent to usage contexts (MIN(time || 'x') still aggregates the text form), cast chains are walked whole (an inner TRY_CAST never shields an outer forbidden cast, but IS the guard for required_guard), a guard only counts as a positive conjunct or inside the enclosing CASE/IF (an OR-branch or NOT-wrapped guard reads unknown), and COUNT(*) triggers table-scoped required_predicate rules by table presence when no when_columns narrows the trigger.

Two semantics that look like bugs but are not. (a) forbidden_sequencing_key fires on a window ORDER BY and on ORDER BY … LIMIT 1 ("the latest row") but NOT on a plain ordered listing with any other LIMIT — the run_sql tool description asks the model to LIMIT every query, so an any-LIMIT reading would refuse ordinary paging. (b) required_distinct normally carries a when_filtered trigger (the predicate that makes the reading wrong, e.g. the winner predicate) and never fires on COUNT(DISTINCT …) of ANY column — count-distinct is fan-out-safe by construction, and counting a different column is a different, legitimate question.

Every rule carries its own self-test (examples: {violation, pass}) and the author gate EXECUTES it at submit time: the violation example must trip the rule and the pass example must not (unknown fails too), on top of the schema-contract check that every bound table/column exists. A rule cannot enter the document without proving it fires and does not over-fire — the same move as qgen's submit-time gold execution.

Advisory only — rules never refuse. A proven violation rides back after the results as one <system-reminder> per dataset (deterministic wording — no false-positive hedge, unlike the judges' — with the policy's own action as the fix-it). The query always executes; there is no verification lifecycle, no overlay, and no blocking path. The trust anchor is entirely the author gate's executed self-test (above).

Cross-dataset queries get the UNION of both datasets' rules. Bindings are database-qualified via the sidecar, and a rule only fires on columns proven to belong to ITS dataset's tables, so cross-talk is impossible by construction; a table from an unregistered database contributes no schema, which makes bare columns near it unresolvable and lands unknown.

Judge-shard exclusion. Policies the tier decided (violation or pass) for a given normalized SQL are removed from that query's judge shards — the fleet only ever sees unknown and rule-less policies, which also saves judge tokens on every flagged-clean query. run_computation is deliberately UNTOUCHED: a computation is already frozen, human-verified SQL with its own trust lane.

Which statements it evaluates (is_checkable_sql) is deliberately WIDER than the judge track's is_analytical_sql: that gate requires an aggregate/join/window/group-by because a fleet round costs tokens, while several dimensions fire on plain single-table reads (an unguarded numeric CAST, ORDER BY <text time>, ORDER BY <opaque id> … LIMIT 1, a sentinel filter). Reusing the judge gate would disarm the tier for exactly those. Only non-reading statements (SHOW/DESCRIBE/EXPLAIN — opaque Command nodes) and information_schema probes are skipped, which also keeps this synchronous path off the policy gate's I/O.

Athena/Trino only in practice. The sidecar's table names come from the harvest snapshot, and rule targets are table.column (unqualified). A Redshift-backed dataset addresses tables as schema.table, which no target can express and whose 2-part references resolve as db.table — so on Redshift the tier reads unknown everywhere and the judges carry the whole load. Fail-safe, but a real coverage gap: schema-qualified bindings are the follow-on.

Fail-open, everywhere. No sqlglot, no sidecar, a raising evaluator: every path yields unknown/no-op and today's judge-only behavior. A rules block the runtime's own catalog can't parse (okf_core is vendored per artifact, so a newer harvest image can author a dimension an older chat image doesn't know) degrades THAT policy to prose (parse_policies(drop_invalid_rules=True)) instead of silencing the whole document for both tiers. The tier arms under exactly the conditions the computational track already does (deploy gate + the run's policy:computational/policy:strict opt-in) — no new feature flag.

Tracing (live testing). Every tier log line is prefixed [policy-rules] (logger chat.policy_check, INFO+), so ONE CloudWatch filter on the chat runtime log group yields the whole trace per query: the evaluated SQL + dialect/default-db, one line per rule — <label> P010 rule[0] forbidden_aggregation(driverstandings.points) -> VIOLATION: SUM(…) — the per-policy outcome (VIOLATION -> advisory reminder to the model / PASS -> excluded from the judge shard / UNKNOWN -> deferred to the judge fleet), a done in Nms: X violation / Y pass / Z unknown; … tally, and the judge-side hand-off (… already decided deterministically — N policies left for the judges). Every SKIP names its reason at WARNING/INFO — no rule-bearing policies, missing rules_schema.json sidecar (with the fix: Guardrails → Sync), a FAILED sidecar load (transient S3 error — named as such, never misdiagnosed as "never authored"), sqlglot missing from the image — because a silently inert tier is the one failure mode an operator can't diagnose from behavior alone. The evaluate_policies result also carries the per-rule breakdown (rules: [{index, label, dimension, verdict, detail}]) for any future UI surface.

The policy_rebuild event (okf_core.policy_rebuild): custom source okf.policy, detail-type policy_rebuild, detail {data_domain, dataset, reason?, force?}. Rides the SAME EventBridge → SQS → incremental-handler path as the Glue events. Published by the Control API (manual sync / repromote) and the chat runtime (stale discovery). force (manual sync ONLY — omitted otherwise) makes the rebuild unconditional; automatic publishers never force, for them fingerprint equality genuinely means nothing to do. Duplicates are harmless: the runtime's conditional building flip collapses them. Events are a freshness ACCELERATOR; the fingerprint gate is correctness.

S3 Vectors (one bucket, one index)

See okf_core.embedding.

• 512 dims, cosine, float32. Non-filterable metadata keys are title, description, s3_key. These are immutable in S3 Vectors.
• Filterable metadata: data_domain, dataset, table, type, tags.
• Embed text and metadata come from build_embed_text, build_filterable_metadata, and build_non_filterable_metadata.
• Any query that filters or returns metadata needs both s3vectors:QueryVectors and s3vectors:GetVectors.

DynamoDB tables

Four tables (plus the checkpoint table the LangGraph saver owns); names come from env vars, with the defaults shown.

okf-registry — domain registry, harvest status, credentials

Partition key pk (S), sort key sk (S). Item shapes:

Declared domain. pk = "DOMAIN#<data_domain>", sk = "META", attrs {data_domain, description, context, created_at, updated_at}. A first-class, operator-declared entity: domains must be declared before Glue databases can be mapped into them. description is a short one-liner; context is richer prose (used in the harvest prompt and exposed to agents over MCP). Listing (GET /domain-defs): scans pk begins_with "DOMAIN#" with sk = "META". Deletion (DELETE /domain-defs/{domain}) is blocked (409) while DATASET# mappings still exist under the same partition. On declare/update, a derived concept doc is written through the harvest mount at okf/<domain>/_domain/overview.md (see S3 layout below) so the domain is embedded and semantically searchable.

Domain mapping. pk = "DOMAIN#<data_domain>", sk = "DATASET#<dataset>", attrs {data_domain, dataset, source, glue_database, created_at} plus optional dataset-guidance attrs {guidance, guidance_updated_at, guidance_applied_version} (shared authoring instructions; see okf_core.guidance + the harvest payload's dataset_guidance above) plus the optional policy-check attrs (ar_build_status, ar_source_hash, ar_pending_source_hash, ar_build_started_at, ar_built_at, ar_build_detail — the build state and lease described under "Policy checks (LLM-judge engine)" above; retired attrs like ar_enrolled or the v1-era ar_policy_arn/ar_guardrail_id may linger on legacy rows — nothing reads them, and deleting the dataset deletes the row). Requires a pre-existing META row for the same pk (enforced by assert_domain_declared in the upsert adapter).

A legacy recursive_improvement map may still sit on old DATASET# rows; it is dead — the in-harvest recursive-improvement loop is retired (harvests never benchmark; see the Benchmark Studio sections below) and nothing reads or writes the attribute anymore.

source is the first-class, future-extensible source descriptor — a nested map {type, ...type-specific config} naming WHERE the dataset's data lives and how the harvester reads it. The vocabulary lives in okf_core.sources (SUPPORTED_SOURCE_TYPES, DEFAULT_SOURCE_TYPE). Supported types:

• glue — {"type": "glue", "glue_database": "<db>"}.
• redshift — {"type": "redshift", "redshift_database": "<db>"} plus self-describing connection routing: cluster_identifier OR workgroup_name (exactly one) + secret_arn (the Secrets Manager secret that authenticates to it). The operator picks these in the UI, so any cluster/workgroup in the account is harvestable with no deploy-time connection config — the harvest reads the connection entirely from the descriptor. Registration REQUIRES the full connection (target + secret; a 400 otherwise) — a db-only descriptor can't be harvested, so it's rejected at the boundary rather than failing deep in the async run. (normalize_source still tolerates a stored db-only row when READING, so legacy rows never break readers.) The secret must hold a read-only DB user and be named with the deployment's secret prefix (var.redshift_secret_name_prefix, default okf-) — the IAM grants are scoped to that name pattern. See docs/DATA_SOURCES.md.

Config keys are stored generically on the item, so a new source type (BigQuery, …) adds a type + config keys with no item-schema migration. For a glue source the flat top-level glue_database attribute is also written as a back-compat mirror: the harvest invocation payload and the incremental scan (incremental/store.py, which filters on glue_database) read it directly. A non-glue source writes no such mirror, which is what (correctly) scopes the aws.glue-event incremental path to Glue datasets. Readers go through okf_core.normalize_source, which reconciles the nested shape and pre-source rows (flat glue_database only) into one {type, ...config} dict. The Control API validates on write (PUT /domains/{domain}/datasets/{dataset} accepts either a source object or a bare glue_database), rejects an unsupported type with 400, and applies per-source registration rules (assert_source_registrable): a glue dataset name must equal its glue_database and the database must exist; a redshift dataset name is independent of redshift_database, must carry the full connection (target + secret, see above), and gets no live existence probe (the harvest verifies the connection when it first runs).

The UI's mapping dialog fills a Redshift descriptor dynamically: GET /redshift/clusters lists provisioned clusters + Serverless workgroups (control-plane, no DB connection), and GET /redshift/databases?cluster=…|workgroup=…&secret_arn=…[&database=…] lists databases within a chosen target via the Redshift Data API (ListDatabases, which connects, hence the secret; database is the bootstrap DB to connect through — a provisioned cluster's DBName hint from /redshift/clusters — defaulting to dev).

Harvest status. pk = "HARVEST#<data_domain>#<dataset>", sk = "STATUS", attrs {status: queued | running | complete | failed | cancelled, mode, started_at, updated_at, detail, runtime_session_id, model, effort}. The harvest is DONE at the bundle commit marker: the runner flips the row terminal right after finalize_bundle returns, and the policy document then authors as its own follow-on step (still in the same runtime session, but a separate process semantically). The terminal complete/failed writes are conditional on the row still being in flight AND still being THIS run's (runtime_session_id must match when the runner knows its own — a hung run's late write must not clobber a successor that took the lease via the staleness escape, nor trigger a ghost policy build against its mid-write bundle). The session pin alone can't protect the INCREMENTAL path, whose session id is deliberately deterministic per dataset (two successive incremental runs share it) — so the runner also pins on started_at (status.read_run_identity): every lease acquire rewrites it and no later write touches it, making it a per-run identity the condition checks too. That step runs only when the terminal write actually landed (report_status returns False when a cancel won the race — no authoring for a cancelled run), is SKIPPED outright when the policy feature is off (OKF_POLICY_BUILD_ENABLED unset — no flush wait for a no-op), first WAITS for the bundle's S3 flush to settle (the S3 Files mount's write-back can lag the terminal write by a minute — gathering early fingerprints a partial wiki and the fresh document immediately reads "out of date"; the wait polls for the commit marker plus a settle margin, bounded, and matches the marker's completed_at against the one THIS run's finalize wrote — on a re-harvest the PREVIOUS run's complete marker is still the visible object until the mount flushes), and is serialized by ITS OWN lock — the mapping row's ar_build_status = "building" flip with its ar_build_started_at stamp (okf_aws.ar_policy.build_lock_active; staleness escape BUILD_LOCK_STALE_SECONDS = 1h, honored symmetrically by the flip itself, so an abandoned building row neither wedges harvests nor locks authoring out — the reconcile still reaps it). The gate lives INSIDE the Control API's lease acquirers (acquire_harvest_lease / acquire_repromote_lease — every trigger path gets it for free) and answers with 409 ("guardrails are being authored…"); dataset deletion, which takes no lease, calls it directly; the incremental orchestrator checks the shared helper and returns skipped_guardrails_building without recording the new version (the change is re-detected once the build lands). The REVERSE gate also holds: the Reasoning page's Sync refuses (409) while the harvest lease is held — the finished harvest authors on its own (the lease-held mirror honors BOTH staleness escapes below, so a dead repromote frees Sync after 120s, not 8h). The Reasoning status GET reports wiki_rewriting only while the marker reads in_progress AND that lease is live — a failed run leaves the marker at in_progress forever, and without the cross-check the page would promise an auto re-author that is never coming (it reports "the last harvest did not complete" instead, with no freshness verdict). A follow-on build that loses the flip race leaves a policy_rebuild trigger behind, and every authoring run ends with a freshness re-check (a wiki that moved mid-build is flagged stale + re-queued), so the gate stays a courtesy, never a correctness requirement. The status GET exposes the live lock as a top-level guardrails_building boolean — knowably false (and not re-read) while the row itself is still queued/running, since the build only starts post-terminal and an active build would have refused the lease. A mode = "cross" row additionally carries cross_target ("<domain>/<dataset>", stamped at lease time — the counterpart the discovery run is against, surfaced by the status GET so the UI shows WHO, not just the mode; mirrors the repromote rows' repromote_target). Mode strings are wire values — the UI maps them to display labels (e.g. cross renders as "Cross-dataset discovery", annotated as "Apply annotations"). model and effort record the RESOLVED LLM config the run actually used (override or deploy-time default); the runtime stamps them on the running transition (harvest.status.report_status), so they're empty on a still-queued row. cancelled is a terminal status set by the Control API's cancel_harvest (POST /harvest/{domain}/{dataset}/cancel): it StopRuntimeSessions the runtime_session_id and flips the row with a conditional update (status IN (queued, running)) so it never clobbers a complete/failed the runner wrote first. Being terminal, it satisfies the lease-free predicate below, so a retrigger is immediately allowed.

This row also serves as a per-dataset harvest lease. Every path that starts a harvest — the Control API's trigger_harvest and the incremental orchestrator / nightly reconcile's process_event — acquires the lease with a conditional PutItem before invoking the runtime:

attribute_not_exists(pk) OR NOT (status IN (queued, running)) OR started_at < <now − 8h>

If a harvest for the dataset is already in flight, the second one is refused: the Control API returns 409, and the incremental path returns skipped_locked without recording the new Glue version, so the change is picked up again by the next event or the nightly reconcile. This keeps two runs from writing the same bundle directory at once (one run's clean_authored_output deleting files while the other writes them). A lease older than 8 hours (HARVEST_LEASE_STALE_SECONDS, the AgentCore session cap) can be taken over, so a dead job whose final status write was lost doesn't wedge the dataset forever. A failed invoke marks the row failed to release the lease.

Annotations: unanchored + agent-submitted. quote is OPTIONAL on an annotation item: empty = an UNANCHORED note (page-level general feedback), which the orphan sweep resolves only if its doc is gone. concept_id may be the _dataset sentinel (underscore-pseudo, like _domain) for DATASET-level feedback — never orphaned while the dataset exists. submitted_via records provenance: "ui" (default) or "agent" — the chat agent's per-run submit_annotation tool files on the user's behalf (the run's verified sub keys the partition; chat role has PutItem-only on the annotations table).

A cross-dataset run (mode = "cross") takes only ITS OWN dataset's lease — the target is read via a start-time snapshot and never written, so no lease is ever taken on it and no cross-bundle write window exists. Concurrent X→Y and Y→X cross runs are therefore independent by construction (each harvests its own dataset). The initiating bundle's fresh complete marker carries cross_target: "<d>/<ds>" provenance.

A repromote (bundle version restore, below) takes this SAME lease with mode = "repromote" and rides the existing queued → complete | failed lifecycle — no new status value. A bundle import ("Bundle export & import" below) does the same with mode = "import". EVERY acquirer — the Control API's harvest and repromote/import acquires AND the incremental orchestrator's twin — carries one extra takeover clause for these two modes:

OR (mode IN ("repromote", "import") AND status = queued AND started_at < <now − 120s>)

Both run synchronously inside the 30s-capped Control API Lambda, so such a row still queued after 120s is provably dead, and anything (a retry, a harvest start, an incremental event) may take it over immediately instead of waiting out the 8h harvest staleness — harvest rows are unaffected. The mode list and threshold have ONE owner, okf_core.session.SYNC_LEASE_MODES / SYNC_LEASE_STALE_SECONDS, shared by all acquirers and the read-side harvest_lease_held mirror. The repromote row also carries repromote_target (the marker VersionId being restored) so the status GET's stalled_lease answer can offer one-click retry; a stalled IMPORT row is reported on the same GET with can_retry: false (its retry is re-applying the import, not re-POSTing a repromote).

Repromote convergence manifest. pk = "HARVEST#<data_domain>#<dataset>", sk = "REPROMOTE", attrs {started_at, completed_at, target_version_id, new_version_id, requested_by, copied: [vector_key...], deleted: [vector_key...], total} — written once per repromote (overwriting the previous one) after the S3 writes land. It exists because deleted keys are unlistable after the fact: the convergence check needs the exact touched-key set captured at write time. GET /bundle/{d}/{ds}/repromote reports a key converged when its VEC#<key> freshness row's updated_at (which reindex advances only AFTER the vector work succeeds) is >= started_at − 2s; the UI declares a repromote done only when every key converged — matching the definition that current is what the vector index serves.

Import provenance row. Same shape on sk = "IMPORT" for a bundle import ({started_at, completed_at, archive_sha256, new_version_id, requested_by, copied, deleted, total}), written once per apply. No convergence poll reads it yet — it captures the touched-key set at write time (the same deleted-keys-are-unlistable reason) so one can. Dataset deletion removes the REPROMOTE and IMPORT rows along with STATUS (the ghost-row rule: a re-registered same-named dataset must not inherit a previous owner's convergence manifests).

Harvest live step feed. Separate from the coarse status row, the harvest runtime narrates its progress at message granularity. As the agent runs, a LangChain callback (harvest.steps.StepEmitter, attached via config["callbacks"] so it also observes every sub-agent) emits one stdout line per step: OKF_STEP <json> where the JSON is {ts, data_domain, dataset, session_id, seq, kind, label, agent, tool?, ok?, error?, full?, result?}. kind ∈ agent | tool_call | tool_result | subagent | usage; seq is a 1-based monotonic counter; label is a human phrase (tool calls are shaped, e.g. "Reading tables/races", "Started table-author: …") — tool RESPONSE bodies are never emitted, only success/failure, with THREE exceptions, each with its own bound (~500 chars for the error snippet, ~8KB for the lint report and agent full, ~64KB for sub-agent dispatch I/O): a FAILED tool_result (ok: false) carries error, a whitespace-collapsed snippet of the failure text (bounded ~500 chars). That text exists nowhere else (it goes back to the model, not the logs), so without it a failed call — e.g. a provider 400 killing a sub-agent — is undiagnosable after the run. And a SUCCESSFUL lint_bundle tool_result carries lint, the lint gate's structured report {ok, errors, warnings, steps, findings, hidden?, note?} (bounded ~8KB — the emitter drops tail findings past the budget into a hidden count; error/warning totals come from the per-step counters so they survive truncation). The UI badges the feed row with the counts and opens the findings in a modal on click. And a SUB-AGENT DISPATCH carries its I/O (bounded ~64KB each, not 8 — a table-author brief carries its context-digest slice and routinely exceeds 8KB; each event is one CloudWatch line, hard limit 256KB): the task tool_call (and a subagent start event) carries full — the complete dispatch brief, where the label keeps only a teaser — and its successful tool_result carries result, the sub-agent's FINAL answer. QuickJS task() squares get the same two texts via subagent events with phase: "update" (a mid-flight patch carrying full right after start and result right before complete, correlated by sub_id): the library's own lifecycle events truncate the brief to 200 chars and never carry the answer, so harvest.subagent_io shims the dispatch choke point to emit them. Never the sub-agent's internal steps — those stay filtered. The UI opens both in the fleet square's drill-in sheet (Output/Input tabs). An agent event also carries full (the complete markdown of the AIMessage, whitespace preserved, bounded ~8KB) when it exceeds the one-line label; the UI renders label as inline markdown and opens full in a modal on click. tool_call/tool_result share a call_id so the UI folds them into one row. subagent events power the UI's fleet squares (the dynamic reviewer/table-author fan-out): they carry {phase: start|complete|error|update, batch, sub_id, subagent_type?, error?, full?, result?} (an error-phase event carries the same bounded error snippet — the langchain_quickjs SubagentErrorEvent's failure string) where batch is the top-level eval tool-call id grouping one fan-out wave (NOT the event's own eval_id, a REPL-local counter that resets to call_0 on every eval() and so can't tell one wave from the next — the emitter correlates each sub-agent to the current top-level eval call_id) and sub_id is the per-dispatch id. They come from langchain_quickjs's custom stream (the run loop uses .stream(stream_mode=["custom"], subgraphs=True), since .invoke() drops these into a no-op writer). The UI grows a row of squares as sub-agents START (there is no reliable pre-start count — the model builds the fan-out list dynamically). usage events carry a usage object with the cumulative token counts for the whole run — {input, output, cache_read, cache_write, total} (total = input+output) — accumulated across EVERY model turn including sub-agents (they emit no feed row but dominate the spend). Fields mirror LangChain's normalized usage_metadata (cache_write is its cache_creation, the Anthropic prompt-cache WRITE; cache_read is a cache HIT). input is the FULL input count and already INCLUDES cache_read + cache_write (per langchain_aws _extract_usage_metadata, which sums bedrock_input + cacheRead + cacheWrite into input_tokens), so total = input + output and cache is a breakdown of input, never additive — the UI shows cache read/write as indented "of which" children under Input, not sibling rows (listing them alongside double-counts). Counts are absolute, so the UI renders the latest snapshot as a running total (a missed/re-ordered poll can't corrupt it) and shows no feed row for the event. Metering is wired differently from the other kinds: it rides a UsageForwarder callback on the shared model instance (build_harvest_agent(step_emitter=…) → _build_model(callbacks=…)), NOT the run-config StepEmitter. This is deliberate — QuickJS task() sub-agents run on their own asyncio tasks and never reach the parent run's callbacks, but they invoke the same inherited model, so only a model-instance callback sees every turn. (on_llm_end on the run-config emitter must NOT meter, or sub-agent turns are undercounted and supervisor turns double-counted.) AgentCore ships stdout to the runtime's CloudWatch log group, so this reuses existing storage (no new event store). The Control API's GET /harvest/{domain}/{dataset}/events?since=<seq>&since_ts=<ms> reads it back with FilterLogEvents, correlating by the run's runtime_session_id (on the STATUS row), and returns {events, next, next_ts, done} (done once the status is terminal). Two cursors the UI echoes back: since/next is the seq high-water mark (exact dedup); since_ts/next_ts is the highest CloudWatch event timestamp (ms), which bounds FilterLogEvents' startTime so each live poll scans only a recent window instead of the whole run. On first load (since_ts=0) the floor is the run's started_at, so a viewer who opens the page mid-run backfills the whole current run. OKF_STEP is a frozen marker shared by harvest.steps and control_api.handlers.

Benchmark report index (Benchmark Studio). pk = "HARVEST#<data_domain>#<dataset>", sk = "REPORT#<report_id>" — one row per standalone benchmark run (okf_core.benchmark_report owns the shapes; the retired RI loop's BENCH# rows have no successor and old ones are ignorable). Report ids are time-prefixed (r<UTC compact>-<hex>, charset-locked by is_valid_report_id) so the sk RANGE ordering is chronological — the report list is one Query begins_with(sk, "REPORT#"), ScanIndexForward=False, no GSI. Flat scalars only (structure lives in the S3 report JSON): status (queued → running → complete | failed), created_at/started_at/ completed_at/updated_at, detail (failure reason), config summary (checks as a CSV string, runs, solver_model/solver_effort, judge_model/judge_effort, version_id, question_count, count_<check>), runtime_session_id, requested_by, live progress stamps (phase, progress_check, progress_run, total_runs, progress_current, progress_total — throttled UpdateItems from the runtime; the Benchmark list POLLS rows for live progress, there is no benchmark CloudWatch feed), headline KPIs once complete (<check>_raw, <check>_adjusted, <check>_graded, total_tokens, annotation_candidates), and the annotation-aggregation sub-lifecycle (agg_status: idle | running | complete | failed, agg_detail — the aggregation's OWN failure reason, separate from the run's detail so an agg failure never clobbers the run's; cleared on every agg start — and annotation_final_count). The Control API writes the QUEUED row (conditional PutItem) and invokes the runtime; the runtime owns everything after via UpdateItem — terminal statuses (complete/failed, agg included) retry a transient DynamoDB fault a few times, since the terminal write has no later write to correct it. Starting an aggregation flips agg_status with a CONDITIONAL UpdateItem ("not already running unless the updated_at heartbeat is past the 8h stale cutoff" — the same escape as report deletion), so a dead aggregator is retryable and two concurrent POSTs can't both start one. No lease semantics: benchmark runs never touch the STATUS row — they write nothing to the bundle, so they run concurrently with harvests and with each other. Rows persist until the user deletes the report (no TTL).

Cross-dataset reference signal (derived). pk = "DOMAIN#<target_domain>", sk = "XREF#<target_dataset>#<source_domain>#<source_dataset>", attrs {target_data_domain, target_dataset, source_data_domain, source_dataset, updated_at}. One row per documented PAIR, recording that <source>'s bundle holds external/<target_domain>/<target_dataset>/… docs.

Derived, never authored. The reindex worker maintains these rows from the bundle's S3 object events — the same events that drive the vector index (see reindex.handler._upsert_xref / _clear_xref_if_pair_empty): a concept doc under okf/<sd>/<sds>/external/<td>/<tds>/ upserts the row; a delete whose pair prefix no longer holds ANY concept doc (checked with a live listing, so out-of-order events self-correct) removes it — with a CONDITIONAL delete (updated_at older than the listing's start), so a concurrent worker's fresh upsert for newly authored docs can never be erased by a stalled delete-path worker. Pair components that fail OKF segment validation (e.g. a #, which would collide two pairs onto one sort key) produce no row at all. Because it is event-derived it survives full-harvest wipes and repromotes with no writer having to remember it, and it is rebuildable by replay — the same "S3 markdown is truth, everything else is derived" rule as the vectors. Whether the pair prefix is empty is judged by parse_bundle_key, so a leftover generated index.md does not keep a row alive.

Why it exists. Pair docs live only in the initiating bundle, so a consumer scoped to the referenced dataset would otherwise never learn the relationship exists. list_domains (both the Control API's GET /domains and the consumption MCP tool) reads these rows alongside the mapping listing (one entity="xref" Query on the entity index — see "Registry entity index" below) and adds two optional fields per dataset: cross_references (datasets this one holds pair docs FOR) and cross_referenced_by (datasets whose bundle holds pair docs about this one — read them under <that dataset>/external/<this_domain>/<this_dataset>/). Both are omitted when empty. The reindex role therefore holds PutItem + DeleteItem on the registry table (and ListBucket on the bundle bucket).

MCP credential. pk = "CRED#<client_id>", sk = "META", attrs {name, client_id, created_by?, created_at}. Metadata only — the client secret is returned once at creation and never stored. This backs the credentials UI (list and revoke); the credential itself is a Cognito M2M app client. created_by is the owner, stamped from the caller's verified JWT identity (email, falling back to sub), not the request body. Revoking (DELETE /credentials/{client_id}) requires a matching CRED# row — so an arbitrary app client, such as the public SPA login client, can't be deleted — and when a caller identity is present it must equal created_by.

Registry entity index. Mapping rows, declared-domain META rows, and XREF# rows carry two extra attributes — entity ("dataset" | "domain" | "xref") and pair ("<domain>/<dataset>", or "<domain>" for META) — keying the by-entity GSI (projection ALL). Listings Query it instead of Scan-with-filter: a Scan reads the whole table (harvest status + REPORT# rows included), so its cost grows with usage rather than with the dataset count. Every writer stamps the attributes (upsert_domain_mapping, declare_domain, reindex's XREF upsert — all via the okf_aws.registry_entity vocabulary); rows written before the index existed need scripts/backfill_registry_entity.py (deploy.sh runs it at the END of the compute stage — after the writers that stamp the attributes are deployed, so no row can land unstamped once the marker exists — idempotent), whose LAST step writes the readiness marker row pk = "REGISTRY", sk = "ENTITY_INDEX_READY". Readers Query the index ONLY once that marker exists (okf_aws.registry_entity — the one shared read protocol) and use the legacy filtered Scan until then: a GSI only contains rows stamped with its keys, so on a partially-stamped registry any result-shape heuristic would return the fresh rows and silently hide every pre-index dataset. The only OTHER condition that may fall back to the Scan is "the index does not exist" (marker stamped before the terraform apply); mid-pagination errors must surface instead — a silent Scan there would re-return pages the caller already consumed. Consumers: the Control API's list_domains, the consumption MCP list_domains, and the chat policy check's _policy_glue_map.

Listing: the consumption MCP list_domains(domain?, query?, cursor?, limit?) is PAGINATED — it returns {"datasets": [...], "next_cursor": ...} with a soft limit (default 100, max 500; DDB requests are Limit-bounded so tiny rows can't defeat the cap), an opaque base64 cursor, a domain partition filter (base-table Query) and a case-insensitive query substring filter over "<domain>/<dataset>"; the legacy-scan fallback returns everything with next_cursor = null. list_declared_domains scans with sk = "META"; list_credentials scans pk begins_with "CRED#".

okf-chat — conversation index (+ the policy checks' rolling context)

Partition key pk (S), sort key sk (S). Isolation is structural — the caller's Cognito sub is baked into the pk, so a user's Query can only ever return their own rows (the same argument as okf-annotations). Key shapes live in okf_core.chat_threads.

Conversation. pk = "CHAT#<user_sub>", sk = "THREAD#<thread_id>", attrs {title, model, effort, data_domain?, dataset?, created_at, updated_at, expires_at?}. thread_id is the CLIENT-FACING id the browser sends, not the <sub>:<thread_id> checkpoint-namespaced form. Written best-effort by the chat runtime on each turn (chat.threads.touch_thread — created_at/title via if_not_exists so a UI rename survives); read/renamed/deleted by the Control API. expires_at (epoch seconds, DELETED_TTL_SECONDS = 1 day) is set ONLY on delete, so an active conversation never expires; the list also skips any row that already carries one, because TTL is eventually consistent.

The THREAD row also carries the policy checks' rolling context (see "Policy checks" above): policy_curated_question (the latest turn's curated standalone question, written when the rewrite lands and overwritten inline on an ask_human fold), policy_last_answer (that turn's final answer, ~2k chars, written at stream end), policy_question_history (a DynamoDB List of the last 3 curated questions, most recent last, rolled forward with each landing — the entries before the previous question feed the next rewrite as earlier-questions resolution context), and policy_history_last_raw (the RAW question whose curated form is the history's last entry — the durable same-turn signal an ask_human fold's re-curation uses to REPLACE that entry instead of appending; cleared to "" by the stream-end answer write so it never outlives its turn, which is what lets an IDENTICAL question re-asked on a later turn append normally). All optional, best-effort, no TTL (thread rows have none) — reloading a chat days later still chains context; absent attrs mean raw-question semantics with no background (chat.threads.read_policy_state / write_policy_state).

Memory pause blob. THREAD rows also carry an optional memory_pending attr (S, JSON {obs, qa}): the long-term-memory context of a turn paused on ask_human — the harness observation so far plus the clarification rounds already folded. Written at every pause (each pause overwrites), read by the resume (chat.threads.write_memory_pending / read_memory_pending), never cleared — the next pause's overwrite is the freshness contract. Best-effort like all memory touchpoints.

Memory datasets ledger. THREAD rows also carry an optional memory_datasets attr (S, JSON list, capped at 32): every dataset the harness has observed a tool touch in this thread, merged at each turn's memory-event write (chat.threads.read_memory_datasets / merge_memory_datasets). It is the citation-validation set: a later no-tool turn's answer citation counts toward the annotation's datasets-cited line only if its dataset appears here, was observed this turn, or is the conversation pin. Advisory and best-effort.

Memory settings. pk = "CHAT#<user_sub>", sk = "SETTINGS#memory" (okf_core.chat_threads.MEMORY_SETTINGS_SK), attr memory_enabled (BOOL). The per-user switch for long-term memory: missing row (or attr) = the deploy default (OKF_CHAT_MEMORY_DEFAULT_ON, from var.chat_memory_default_on — true means opt-out/default-on, false means opt-in/default-off; flipping the var never touches rows users already set). Written by the Control API (PUT /memory/settings, the Memory page's master switch); read by the chat runtime at turn start (chat.memory.ChatMemory.user_enabled, which also treats a FAILED read as the deploy default — an unreadable row must not silently flip a user's memory in either direction). Off = the runtime neither recalls nor writes events; existing memory records are untouched — though the server still STRIPS the previous turn's recall injection from ongoing threads, so switching off silences already-injected context too. Because this row shares the CHAT#<sub> partition, the conversation list Query constrains begins_with(sk, "THREAD#") (an unconstrained Query would surface it as a phantom conversation).

Legacy: policy-check reports. v2's post-turn panel persisted one sk = "POLICY#<thread_id>#<turn_key>" row per checked turn. v3 removed the panel — these rows are no longer written or read; existing ones are inert (invisible to the list, which constrains begins_with(sk, "THREAD#")). The POLICY# key helpers stay in okf_core.chat_threads so any cleanup tooling can still address them.

okf-chat-checkpoints is a fifth table owned entirely by langgraph-checkpoint-aws's DynamoDBSaver: PK (S, HASH) + SK (S, RANGE) — UPPERCASE — TTL attribute ttl (lowercase, written only when the saver is constructed with ttl_seconds), no GSI, with checkpoints and pending writes distinguished by PK prefix (CHECKPOINT_<thread> vs WRITES_<thread>#<ns>#<ckpt>). The chat runtime namespaces the thread id with the caller's sub before it reaches the saver.

okf-freshness — reindex and incremental dedup state

Partition key pk (S), sort key sk (S). Item shapes:

Reindex dedup. pk = "VEC#<vector_key>", sk = "SEQ", attrs {last_sequencer, updated_at}. S3 object.sequencer values compare lexicographically per key, so an event at or below last_sequencer is a duplicate or replay and is ignored. last_sequencer is advanced (conditional PutItem) only after the embed and PutVectors/DeleteVectors succeed, never before — otherwise a transient failure would leave the marker ahead of the work, and the SQS retry would skip the record as a duplicate and silently drop the vector.

Table version. pk = "TABLE#<data_domain>#<dataset>#<table>", sk = "VERSION", attrs {version_id, update_time, last_seen_at}. The incremental path uses this to confirm a real change before re-harvesting. Iceberg data commits (empty column diff, +1 version) are absorbed — the new version is recorded without a re-harvest — so this row advances with every commit while re-harvests fire only on schema changes; see OKF_INCREMENTAL_ICEBERG_COMMITS.

okf-annotations — user feedback on the wiki

Partition key pk (S), sort key sk (S). A separate table (not registry/ freshness) so its DynamoDB TTL sweep — on expires_at — can never reap a durable row; the worst a stray expires_at can do is delete an annotation.

Annotation. pk = "ANNO#<data_domain>#<dataset>#<user_sub>", sk = "<concept_id>#<annotation_id>", attrs {data_domain, dataset, concept_id, annotation_id, author?, quote, prefix?, suffix?, block_line?, note, status, outcome?, resolution?, report_id?, created_at, updated_at, expires_at?} (report_id only on submitted_via: "benchmark" items — the report the note was applied from, provenance only).

Isolation is structural. The author's immutable Cognito sub is baked into the partition key, so a user's Query can only ever return their OWN annotations — there is no cross-user read path (readers pass user_sub from the verified JWT, never the body). sub (not email) is used because it never changes and is #-delimiter-safe. author is the human-facing label (email) for display only.

Anchoring is a quote, not a coordinate. quote is the selected passage; the UI grows prefix/suffix (see okf_core.annotations.normalize_text / the UI's minimalUniqueContext) only until the (prefix+quote+suffix) window is unique in the doc, so two identical quotes on a page are distinguishable. block_line is a body-relative source-line HINT (from react-markdown's node.position, stamped as data-sl) the agent can jump near — never the source of truth. A re-harvest rewrites the doc, so any coordinate would go stale; the quote is what survives.

Lifecycle. status ∈ open | in_review | resolved; outcome (set with resolved) ∈ applied | rejected | orphaned. expires_at (epoch seconds, 7-day TTL — okf_core.annotations.HISTORY_TTL_SECONDS) is set ONLY at resolution, so an open/in_review annotation never expires. The Control API's run pre-flight (POST /harvest/{domain}/{dataset}/annotations/run) takes the per-dataset lease, then for each of the caller's open annotations loads the target doc from S3 and re-anchors the quote (is_orphaned): a note whose passage is gone is auto-resolved orphaned (with ORPHAN_RESOLUTION_MESSAGE) and the agent never sees it. If EVERY open note orphans (or none are open), the run is skipped — the status row is set complete and the runtime is NOT invoked. Otherwise the survivors are flipped in_review and sent in the annotated payload; on invoke failure the Control API reverts them to open so no feedback is stranded. After the run, the harvest RUNNER (not the agent — it has no DynamoDB tools) reconciles the agent's on-mount verdict file to resolved with outcome+resolution, and reverts any survivor the agent didn't rule on back to open.

CRUD: GET|POST /annotations/{domain}/{dataset} and DELETE /annotations/{domain}/{dataset}/{annotation_id}?concept=<id> (the concept id has slashes, so it rides in the query string, not a path segment).

Annotation scope (cross-dataset docs). The run endpoint accepts an optional body {"scope": "dataset" | "cross"}: cross applies only notes whose concept_id is under external/ (the cross-dataset docs), dataset only the rest; absent = everything. _dataset-WIDE notes are general feedback and pass BOTH filters — whether one rides a given run is the annotation_ids selection's call (the UI offers them in every scope, preselected only in the dataset one). Out-of-scope OPEN notes stay untouched for a later run of the other scope; an out-of-scope in_review STRAGGLER (from a dead prior run) is reverted to open rather than dropped, preserving the reclaim invariant even for users who always pick one scope. A cross-scoped run also IGNORES dataset guidance AND the saved recursive_improvement settings (both operate on the dataset's own docs, which the scope excludes). When surviving notes reference external/<d>/<ds>/… docs, the Control API derives the counterpart datasets from the concept ids and sends their Glue database names as extra_glue_databases in the payload — the runtime widens the run's scoped session policy to them so the agent can actually verify cross claims with qualified SQL (without it, every check would be AccessDenied and the notes would be falsely rejected). The UI's picker modal offers one cross scope PER TARGET PAIR (Cross-dataset · <domain>/<dataset>, from the bundle's external/ listing plus any pending note that targets one) — never a generic "all external" bucket — so a run names exactly which target it verifies against. The pair choice rides in annotation_ids plus an optional cross_target: "<domain>/<dataset>" body field (the wire scope stays "cross"; cross_target is refused with any other scope): the selected notes' concept ids widen the session policy, and cross_target guarantees the target's Glue database is granted even when the selection carries only _dataset-wide general notes (whose ids name no pair).

Partial selection. The run endpoint also accepts an optional body annotation_ids: [<id>, …] (the UI's annotation picker): only the listed notes ride the run. Unselected OPEN notes stay open for a later run; an unselected in_review straggler reverts to open — the same stranding argument as the scope filter. An empty list is valid: with a dirty guidance the run still fires guidance-only, otherwise it short-circuits as "nothing to apply". Composes with scope (the id filter applies within the scope).

Harvest invocation payload

InvokeAgentRuntime(agentRuntimeArn=<harvest arn>, runtimeSessionId=<per-dataset id>, payload=json.dumps({...}).encode()), where the payload is either:

{ "data_domain": "sales", "dataset": "orders", "mode": "full",
  "source": { "type": "glue", "glue_database": "orders" },
  "model": "openai.gpt-5.6-sol", "effort": "xhigh",
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded." }

source (optional, all modes) is the first-class source descriptor (okf_core.sources, {type, ...config}) the Control API resolves from the mapping row and threads through so the runtime dispatches on the source type (harvest.clients.build_source) instead of assuming a Glue database named by the dataset. A glue source carries glue_database; a redshift source carries redshift_database plus its cluster/workgroup + secret_arn connection routing (self-describing — the harvest connects from the descriptor, no deploy-time env). Absent → the runtime defaults to a glue source named by dataset (back-compat: older payloads and the provision/write-domain-doc modes carry no source). The incremental path is Glue-only (it fires on aws.glue catalog events) and always sends a glue source.

(model/effort optional — see below.) Or, for an incremental run:

{ "data_domain": "sales", "dataset": "orders", "mode": "incremental",
  "changed_table": "customers",
  "diff": { "added": [], "removed": [], "retyped": [] },
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded." }

or, for a cross-dataset run (Roadmap §5 — author external/ pair docs):

{ "data_domain": "sales", "dataset": "orders", "mode": "cross",
  "source": { "type": "glue", "glue_database": "orders" },
  "target": { "data_domain": "crm", "dataset": "customers",
              "source": { "type": "glue", "glue_database": "customers" },
              "domain_description": "Customer master data",
              "domain_context": "…" },
  "domain_description": "Revenue & order pipelines" }

The Control API resolves + validates target from the UI's flat target_data_domain/target_dataset body fields (resolve_cross_target): registered mapping (404), glue-backed on BOTH sides (400 — v1 verification is qualified Athena SQL, so cross-source pairs have no common engine), distinct from the dataset itself AND resolving to a DIFFERENT Glue database (400 — the same dataset name under two domains is the same physical data), and BOTH bundles published (409). A cross payload deliberately carries no dataset_guidance and no recursive_improvement — guidance is dataset-scoped steering and the pair docs are shared with another dataset's readers. The runtime validates the target components as path segments (okf_core.paths.external_pair_prefix — they become destructive paths and the XREF key), widens the run's scoped session policy to the pair's two Glue databases (never more), snapshots the target's catalog + published docs (minus the target's own external/ subtree — another run's pair docs are not the target's own verified facts) into .metadata/external/<d>/<ds>/, and confines writes to external/<d>/<ds>/ (guard-enforced, INCLUDING the cross-mode reviewer's middleware). Ordering is load-bearing: the target-readiness re-checks (the trigger-time check only covered trigger time) and the required target snapshot all run BEFORE the first destructive step, so a failure there leaves the bundle untouched and READY — prior pair docs intact. Uses a fresh session id per trigger, like full.

or, for an annotation run (apply a user's wiki feedback in place):

{ "data_domain": "sales", "dataset": "orders", "mode": "annotated",
  "user_sub": "<cognito sub>",
  "annotations": [
    { "annotation_id": "…", "concept_id": "tables/orders",
      "quote": "one row per order", "prefix": "", "suffix": "",
      "block_line": 12, "note": "grain is per line-item, not per order" }
  ],
  "model": "openai.gpt-5.6-sol", "effort": "high",
  "subagent_model": "…", "subagent_effort": "…",
  "reviewer_model": "…", "reviewer_effort": "…",
  "domain_description": "Revenue & order pipelines",
  "domain_context": "Covers all B2C sales; refunds excluded.",
  "dataset_guidance": "Ignore the staging_* tables; status is decoded in the dictionary.",
  "dataset_guidance_version": "2026-07-17T09:00:00+00:00" }

Applying annotations is a harvest like any other, so model/effort (+ the subagent_*/reviewer_* pairs) are the SAME optional per-harvest override triple mode: "full" accepts — same three scopes (supervisor / sub-agents / reviewer), same catalog validation at the Control API trust boundary, same fallback when omitted (the runtime's deploy-time OKF_HARVEST_MODEL/ OKF_HARVEST_EFFORT). The UI's harvest picker sends its current selection on an annotation run, so applying annotations honors whatever model an operator had chosen for full harvests of this dataset, rather than silently reverting to the deploy-time default.

dataset_guidance (optional, on every mode) is the dataset's shared authoring guidance — persistent, editable operator instructions (registry DATASET# row: guidance, guidance_updated_at, guidance_applied_version). It steers the harvest prompt; on a SUCCESSFUL run the runner stamps guidance_applied_version = dataset_guidance_version so the guidance clears its DIRTY state (okf_core.guidance.is_dirty). An annotated run is invoked when there are live annotations or the guidance is dirty — so editing guidance and re-running applies it even with zero annotations (a guidance-only re-harvest, annotations: []).

Benchmark Studio invocation (mode: "benchmark"). A standalone, human-triggered evaluation on the harvest runtime — NOT a harvest: it takes no lease, doesn't use the S3-Files mount (the wiki snapshot is GET straight from S3, live or pinned to a bundle version), and writes nothing to the bundle. The harvester itself can no longer benchmark — the in-run recursive-improvement loop, its run_benchmark tool, and the recursive_improvement payload block are retired end to end. The payload (okf_core.benchmark_report field names):

{ "data_domain": "sales", "dataset": "orders", "mode": "benchmark",
  "report_id": "r20260729t101500-1a2b3c4d",
  "checks": ["sql", "behavior"],
  "runs": 3,
  "version_id": "",
  "questions_key": "benchmark/sales/orders/questions.csv",
  "questions_version_id": "3sL4kqQJlcpXroDTDmJ+rmSpXd3dIbrHY+MTRCxf3vjVBH40Nr8X8gdRQBpUMLUo",
  "solver_model": "global.anthropic.claude-sonnet-5", "solver_effort": "high",
  "judge_model": "global.anthropic.claude-opus-5", "judge_effort": "xhigh",
  "behavior_live_sql": false,
  "source": {"type": "glue", "glue_database": "orders"} }

questions_version_id (optional) is the CSV's S3 VersionId as the Control API validated/counted it at start — the bundle bucket is versioned, so the runtime GETs exactly that version and a re-upload between start and the fetch can't swap the graded set. Absent (older payloads, or a CSV written while the bucket was unversioned) → the runtime reads the latest object.

behavior_live_sql (optional, default false; also a BOOL on the REPORT# row's config summary when true) hands the BEHAVIOR solver read-only run_sql against the live dataset — a truer consumer simulation (real agents can query); its prompt flips from "you CANNOT query" to wiki-leads-SQL-verifies (harvest/benchmark/checks.py solver_protocol). It never applies to the SQL EX check, whose solver stays data-blind by design. Reports carry the flag — scores are not comparable across different settings of it.

questions_key is the uploaded CSV — one gold column per check (question,gold_sql,expected_behavior; a question participates in a check iff its gold cell is non-blank; unrecognized columns are ignored — the retired gold_answer no longer resolves). It lives under the off-mount benchmark/<domain>/<dataset>/ prefix — deliberately NOT under okf/ — so the gold is invisible to every LLM role; the runtime GETs it into process memory (needs s3:GetObject on <bundle-bucket>/benchmark/*). checks ⊆ {sql, behavior} (≥ 1); runs is clamped to 1–5; models are validated against the harvest catalog by the Control API; version_id (optional) pins the wiki to a published bundle version — the WIKI, not the DATA: grading always executes against live Athena. Question count is hard-capped at 100. sql (shown as "Accuracy") grades deterministically — BIRD-style result-set equality: rows compared as POSITIONAL tuples (column order matters, row order doesn't), with numeric-looking cells normalized to Decimal so Athena's stringified 3 vs 3.0 compare equal; behavior is judge-graded: expected_behavior is free-form prose (refusals, policy adherence, "should say it isn't tracked"), the solver answers in free-form text, and the judge rules on EVERY (question, run) attempt independently — so behavior has NO judge-adjusted score and its failed pairs never enter the overturn review (the grader already was the judge; its score block carries adjusted: null, and the REPORT# row omits behavior_adjusted). Each failed behavior pair instead gets ONE question-level SYNTHESIS review (all graded runs together): it supplies the pair's judge block — comment + one consolidated annotation (the annotation candidate) — and never changes outcomes. Failures are LOUD: a run that can't fetch/parse its questions or materialize its snapshot fails the REPORT# row with the error (no silent degradation). The judge phase is always on; there is no stop target and no loop. Every benchmark ReAct role (solver, the judge's hats, the annotation aggregator) is built via harvest/benchmark/react.py — LangChain create_agent with the chat agent's BedrockPromptCachingMiddleware — so on a Converse Claude model the per-turn re-sent conversation bills as cache READS (a Mantle GPT caches implicitly server-side, where the middleware no-ops). The judge hats DELIVER their ruling through a tool call (submit_verdict / the reviewer's submit_review — args are the output, structured by construction, no fence parsing), and a SubmitToolNudgeMiddleware steers a hat that tries to finish without submitting — at most twice, then the unparseable-output path rules the case a fail with judge_error set. The judge hats see everything the solver can't: their file tools are rooted at the JUDGE tree (wiki docs + .metadata/ + .context/ + the .traces/ solve traces), and when that tree carries .context/ files the run also opens a Code Interpreter sandbox session (harvest/code_interpreter.py sandbox_session — the same OKF_CODE_INTERPRETER_ID interpreter the harvester uses) and hands the hats run_code, so binary context uploads (PDF/DOCX/PPTX/XLSX) that read_file only base64-encodes stay readable evidence. Best-effort like the harvest's: no interpreter configured, or a start/upload failure, just means the judge reads text context only.

Report artifacts (off-mount, human-facing). The run persists benchmark/<d>/<ds>/reports/<report_id>/report.json — config recap, per-check scores (raw + judge-adjusted, per-run + mean ± spread), per-question stability, per-question detail (gold, every attempt's outcome/reason/prediction, the judge's {verdict: pass|fail, comment, annotation}), telemetry (per-tool call distribution, tokens by role, wall time), and the judge's annotation candidates (+ the aggregator's final set once generated) — and a companion traces.json (EVERY attempt's bounded solver trace, passing and failing, keyed {q_id, check, run}; shape per harvest/benchmark/trace.py). Both carry gold, so they are served ONLY via the Cognito-authed Control API: POST/GET /benchmark/{d}/{ds}/runs, GET/DELETE .../runs/{report_id}, GET .../runs/{report_id}/traces, POST .../runs/{report_id}/aggregate. An artifact past the 4 MiB inline cap (a Lambda response tops out at 6 MB; multi-run traces.json routinely exceeds it) is answered as a short-lived presigned S3 GET instead of the document — report_url on the report response, {report_id, traces_url} on traces — which the UI api client follows transparently. DELETE .../runs/{report_id} is refused (409) while the run or an aggregation is genuinely active, but a row whose updated_at heartbeat predates the harvest-lease stale cutoff (8 h) is deletable — a killed runtime must not leave an immortal zombie — and the runtime's row writes are conditional on the row existing, so a late finish can't resurrect a deleted report (the runtime also re-checks the row before persisting the S3 artifacts, so a delete mid-run doesn't leave orphaned gold behind). Because the bundle bucket is VERSIONED, deletion purges every object version and delete marker under the report prefix (list_object_versions + per-version deletes) — a plain delete would leave the gold readable as noncurrent versions. Deleting the DATASET purges the whole benchmark/<d>/<ds>/ prefix the same versioned way (questions.csv + all report artifacts) and every REPORT# row along with the bundle. POST .../runs/{report_id}/aggregate kicks mode: "aggregate_annotations" — the ReAct aggregator dedupes the candidates into the final set on the report — and POST .../runs/{report_id}/annotations batch-creates the human-selected set as normal annotations with submitted_via: "benchmark" (validated whole-batch before anything is written — no partial commits; the path's report_id must name an existing report row and is stamped on each created annotation for provenance); an unscoped annotation harvest then applies them. On both start routes (run + aggregate) the Control API reads the invoke's synchronous ack: a {"status": "rejected"} from the runtime flips the row to failed (run detail / agg_detail) and answers 502 — an accepted invoke API call is not an accepted payload. The judge reads each solver's trace — what it searched, which docs it opened — which is what separates "the wiki never says this" from "the wiki says it and the solver never found it"; beyond the per-case inline summaries, ALL traces are laid into the judge's file tree as .traces/<check>/q<id>-run<n>.md so it can grep across solvers for systemic patterns.

Synthetic question banks (mode: "generate_questions" — okf_core.qbank

• harvest/benchmark/qgen.py) share the Benchmark Studio posture exactly: no lease, no mount, nothing written to the bundle, loud failures on the row. The index row is sk = "QBANK#<qbank_id>" on the same HARVEST#<d>#<ds> partition (ids qb<timestamp>-<token>, time-prefixed and chronologically sortable like report ids), same queued → running → complete/failed lifecycle, heartbeat, and stale-delete escape; the artifact — benchmark/<d>/<ds>/qbank/<id>.json, gold-carrying, off-mount — holds {config, questions[], dropped[], counts, telemetry}. The author agents see ONLY the dataset's ground truth (.metadata/ + .context/, physically materialized without the wiki docs — the wiki is the system under test) plus the live source tools; the config (count 20–100, checks + sql_share, dimensions) is validated by okf_core.qbank.validate_config at the Control API AND in the runtime, and the deterministic allocator turns it into explicit (dimension, tier, check) slots. Questions are delivered through the submit_question tool, which validates at submit time (shape, cross-author dedup, business-language leakage lint, and a live gold execution under the grading caps — an applied bank can never produce DISCARDED questions); a quota middleware refuses a silent finish (every slot filled or explicitly forfeited), one backfill round — chunked into round-1-sized author batches so a large leftover set doesn't blow one agent's step budget — retries the leftovers, and the rest are dropped WITH reasons in the artifact. Routes: POST/GET /benchmark/{d}/{ds}/qbanks, GET/DELETE .../qbanks/{qbank_id}, POST .../qbanks/{qbank_id}/apply, POST .../qbanks/{qbank_id}/cancel. Cancel mirrors the harvest cancel (best-effort StopRuntimeSession on the row's session, then a CONDITIONAL flip to cancelled that never clobbers a terminal state) with one stronger guarantee: partial work does not survive — the cancel purges every artifact version under the qbank key, the runtime discards instead of persisting when it finds the row cancelled, EVERY runtime row write (the initial running flip included — a cancel landing while the row is still queued cannot be undone by the cold start, which also exits without generating) is conditional on NOT-cancelled (a blocked write is dropped, closing the resurrection race), the start path's failure flip is conditional the same way (a hung invoke can neither overwrite a cancel nor upsert a ghost of a deleted row), and GET/apply serve the artifact for COMPLETE rows only, so even a race-surviving orphan is unreachable. Dataset deletion purges QBANK# rows with the REPORT# rows. A bank past the inline response cap ships as a presigned bank_url (the canonical CSV stays inline). Apply renders the canonical CSV (question,gold_sql,expected_behavior,tier,dimension; the extra columns are ignored by load_questions per its documented contract) and REPLACES benchmark/<d>/<ds>/questions.csv (reversible: versioned bucket, runs pin their CSV version).

or, for writing/refreshing a domain's concept doc through the mount:

{ "data_domain": "sales", "mode": "write_domain_doc",
  "description": "Revenue & order pipelines",
  "context": "Covers all B2C sales; refunds excluded." }

The annotated payload carries only the LIVE annotations (the Control API's pre-flight already resolved any orphans) plus the user_sub needed to reconstruct each annotation's DynamoDB key for the runner's write-back. The agent assesses each note against live data, edits the doc when it holds up (augmentation guard applies), and writes a per-annotation {outcome, comment} verdict to .harvest/annotation_results.json on the mount; the runner reconciles that to the okf-annotations table. It reuses the incremental path's scoped, in-place approach (no clean_authored_output) and the deterministic per-dataset session id.

domain_description and domain_context are optional enrichment keys added by the Control API (and the incremental orchestrator) from the DOMAIN#/META row. They are threaded into the harvest prompt so authoring is domain-aware. The write_domain_doc mode writes <mount>/<domain>/_domain/overview.md through the mount (uid 1000 safe) and returns synchronously.

model and effort are optional per-harvest overrides for the LLM (chosen in the UI's harvest-settings picker; full/incremental only). When present the runtime uses them; when absent it falls back to the deploy-time OKF_HARVEST_MODEL / OKF_HARVEST_EFFORT env. subagent_model and subagent_effort are the same kind of override for the run's SUB-AGENTS — the table/reference authors, reviewers, and context-extractors; when absent the sub-agents run on the supervisor's config. The Control API validates each pair against the model catalog (OKF_HARVEST_MODEL_CATALOG, from var.harvest_model_catalog) before invoking — an unknown model or an effort not offered for that model is a 400, and an effort without its model key is a 400. This is the trust boundary: both model values reach bedrock:InvokeModel, and the runtime deliberately does not allow-list effort itself. The catalog (a JSON array of {model, label, efforts, default_effort}) is the single source of truth, shared by the Control API (validation, raw JSON env) and the UI (VITE_HARVEST_MODEL_CATALOG, base64 — see below) and defined in okf_core.harvest_models.

reviewer_model and reviewer_effort are a third override for the adversarial reviewer sub-agent ONLY — cross-model review improves coverage (a fresh model family doesn't share the authoring model's blind spots). Absent ⇒ the reviewer runs on the sub-agents' config (which itself falls back to the supervisor's).

The runtime always builds three model instances — the supervisor's, the sub-agents' (authors/extractors), and the reviewer's (identical configs when no overrides were sent) — each carrying its own scope-tagged usage callback. That is what makes the step feed's usage snapshot splittable: it carries the cumulative run totals plus a by object ({supervisor: {...}, subagents: {...}, reviewer: {...}}, same counter names) the UI renders as the per-agent token drill-down. The resolved supervisor pair is stamped on the status row as model/effort at the running transition; subagent_model/subagent_effort and reviewer_model/reviewer_effort are stamped only when the respective override was chosen (absence means "same as the tier above").

Build runtimeSessionId with okf_core.runtime_session_id(...), not a bare "<domain>__<dataset>" — AgentCore requires 33–256 characters, so the helper appends a sha256 suffix to a readable okf-<domain>-<dataset>- prefix.

• Incremental uses a deterministic id (runtime_session_id(domain, dataset)) for one session per dataset and microVM affinity. It re-authors the changed table and its backlinks in place and leaves the rest of the bundle alone.
• Full uses a fresh id per trigger (unique_token=uuid4().hex), because a one-shot batch job wants a new microVM with a clean S3 Files mount rather than reattaching to a warm one (AgentCore reuses a microVM per session id until it stops). A full harvest is a clean rebuild: run_full_harvest marks the bundle in-progress, then fsutil.clean_authored_output deletes all prior authored output (datasets/, tables/, references/, index.md, log.md) before the agent re-authors. A table dropped from Glue leaves no stale doc, and its vector is pruned through the S3 write-through → ObjectRemoved → reindex DeleteVectors. .context/ (user input) and .harvest/ (the commit marker) are preserved. The rule is: delete every top-level entry whose name does not start with ..

Reports (chat-authored)

create_report (a chat tool; see chat/reports.py) lets the MAIN chat agent compose an immutable HTML report from evidence gathered in the conversation — there is no delegated runtime and no job row; the chat loop itself is the research loop. The pure contract (block schema, composer, id/key helpers) lives in okf_core/reports.py.

• Blocks are config, not code: markdown | chart | table | kpi; chart blocks carry the same declarative renderChart spec as chat charts. Figure blocks carry provenance — {kind: "computation", slug, params?} (VERIFIED badge) or {kind: "adhoc_sql", sql} (EXPLORATORY badge, SQL disclosed in the report).
• The save is atomic and render-verified: lint → every chart rasterized through the baked harness page in headless Chromium (a chart that fails refuses the save) → one self-contained HTML (inline CSS, data-URI PNGs, print stylesheet) → PDF printed from that same HTML → S3 puts.
• No database row. The public report id is composite — rep~<domain>~<dataset>~<YYYYMMDDTHHMMSSZ>~<hex8> — so serving resolves S3 keys from the id alone (okf_core.reports.parse_report_id).
• S3 layout (bundle bucket, OUTSIDE the mounted okf/ prefix — never meets the harvest lease or reindex): reports/<domain>/<dataset>/<stamp>-<suffix>/{blocks.json, report.html, report.pdf}. blocks.json is the self-describing source (title, request, coordinates, created_by, blocks) — future re-rendering never needs the agent.
• Serving: GET /report/{report_id} (control API) returns presigned GET urls {html_url, pdf_url, blocks_url} — presigned because a composed report can exceed the Lambda response cap; pdf_url is "" on a deployment without Chromium. The UI lifts create_report / present_report tool calls into inline report cards; the viewer renders the HTML in an iframe with NO allow-scripts.
• Env: OKF_CHAT_REPORT_HARNESS_PATH (the harness page in the chat image; empty = chart blocks refused, text/table/kpi reports still work), OKF_CHAT_REPORT_MAX_BYTES (composed-HTML cap, default 8000000). IAM: the chat role holds s3:PutObject on reports/* only — the wiki stays read-only to chat.

Long-term chat memory

Per-user memory on Bedrock AgentCore Memory (infra/durable/agent_memory.tf — one memory resource + one CUSTOM user-preference strategy whose extraction/ consolidation prompts are the design's rulebook). Memory stores facts about the USER — never facts about the data (tables/joins/metrics belong to the wiki):

• Three record kinds, one namespace per user: stated preferences (presentation/workflow/language, plus meanings the user assigns to terms), personal context (name/role/team — only what the user states about themselves), and binding records (how this user's recurring question maps to a governed artifact — a verified computation or metric, with the parameter SHAPE, never literal values). The namespace is wiki/<sanitized-sub> — derived via okf_core.memory_records .memory_namespace in BOTH services (the strategy resolves {actorId} from CreateEvent's actorId, which is sanitized to [a-zA-Z0-9][a-zA-Z0-9-_]*; deriving from the raw sub anywhere would split write and read).
• Structured record fields are REAL metadata (type / dataset / expires_at, LLM-extracted per the strategy's metadata_schema — the ONE awscc-managed resource in the repo, because hashicorp/aws doesn't expose the schema yet). type+dataset are indexed → server-side retrieval filters; expires_at is client-checked (unset-OR-future isn't expressible in AND-composed filters). A legacy content-header line ([type:...] [dataset:...] [expires:...]) is parsed as FALLBACK for drifted extractions. Parser shared via okf_core.memory_records (chat runtime + control API both use it — never parse ad hoc).
• Runtime touchpoints (chat.memory): at turn START semantic retrieval (query = raw prompt + previous turn's curated question; maxResults explicit — the API's default page is 20), then client-side lazy-TTL (expired records are DELETED on recall) and dataset scoping (pinned chat → TWO filtered calls: generic dataset NOT_EXISTS + the pin EQUALS_TO, INTERLEAVED dataset-first so neither pool starves the other; unpinned → one unfiltered call), injected as a marker-carrying HumanMessage (okf_memory — skipped by history rebuild and steering's turn accounting). The per-turn injection (marker value recall) is REPLACED each turn — the server strips the previous one from checkpointed state via RemoveMessage before injecting fresh, so an edit/delete on the Memory page (or the switch going off) actually changes ongoing threads. personal records are the exception: fetched once (recall_personal, server-side type filter), injected on the thread's FIRST turn (marker value personal), carried by history. The whole turn-start prep runs in a worker thread (asyncio.to_thread) — a memory-API brownout must never stall the shared event loop. At turn END one create_event carrying the turn text (every piece excerpt-capped) + a [[okf-harness]] annotation of observed facts so extraction judges only acceptance and phrasing, never the factual core. The annotation's dataset resolution is two-level and either/or: level 1 parses the answer's <c src="dd/ds/…"> citations and keeps the pairs the harness can corroborate (observed this turn, in the thread's cumulative memory_datasets ledger, or the conversation pin) — a datasets-cited line that REPLACES the touched list (attribution beats exploration, and it survives no-tool follow-ups); level 2 falls back to the observed datasets-touched list when no citation validates (an unbacked citation is a model claim, never laundered into the trusted block). resolved-by lines (which governed tool resolved the turn) ride regardless — bindings stay observation-only evidence. The annotation also carries a curated-question: line — the policy machinery's context-resolved rewrite of the turn's question — whenever it differs from the raw text: extraction is asynchronous, so an elliptical turn ("and last month?") must be self-contained in its own event; the raw wording stays the USER message (meanings extract from the user's words, never the rewrite). Paused (ask_human) turns persist their observation + folded clarification rounds in a memory_pending JSON blob on the THREAD row (chat.threads.write_memory_pending; each pause overwrites, the resume reads) so multi-round clarifications and pre-pause governed calls survive the invocation boundary; cancelled and ERRORED turns write no event.
• A recalled binding is a HINT: the injection instructs the model to re-verify the artifact (describe it; only VERIFIED may answer) — memory degrades to normal exploration, never to a wrong answer.
• Management (control API + the Memory page): GET /memory (list, with parsed header fields + expired flag), GET/PUT /memory/settings (the per-user switch — see the okf-chat settings row above), PATCH /memory/{id} (edit text; the record's real metadata is re-supplied verbatim and the header re-emitted, and a failedRecords entry in the 200 response surfaces as a 409 — never a fabricated success), DELETE /memory/{id}. Namespace is always derived from the caller's JWT sub, never a parameter.
• Env: OKF_CHAT_MEMORY_ID on the chat runtime AND the control API (empty = feature off: no client, no recall, routes 404). Deploy gate var.enable_chat_memory; extraction model var.chat_memory_model (durable stack).

Environment variables

Variable	Meaning
AWS_REGION	region for all clients
OKF_ACCOUNT_ID	account id (for building Glue ARNs)
OKF_BUNDLE_BUCKET	S3 bundle bucket name
OKF_VECTOR_BUCKET	S3 Vectors bucket name
OKF_VECTOR_INDEX	S3 Vectors index name
OKF_REGISTRY_TABLE	DynamoDB registry table (default okf-registry)
OKF_FRESHNESS_TABLE	DynamoDB freshness table (default okf-freshness)
OKF_ANNOTATIONS_TABLE	DynamoDB annotations table (default okf-annotations) — user-scoped wiki feedback + the harvest runner's resolution write-back
OKF_HARVEST_RUNTIME_ARN	AgentCore harvest runtime ARN
OKF_ATHENA_OUTPUT / OKF_ATHENA_WORKGROUP	Athena results (glue source)
OKF_GLUE_CATALOG_ID	Glue catalog id override (glue source; default the runtime account's catalog)
OKF_MOUNT_PATH	S3 Files mount (default /mnt/data)
OKF_CODE_INTERPRETER_ID	AgentCore Code Interpreter id backing the harvest agent's run_code tool (extracts text from binary .context/ docs). A network-isolated SANDBOX-mode interpreter. Unset → harvest runs without run_code (text-only .context reading)
OKF_ENABLE_LAKEFORMATION	Set ("true") when the harvested Glue catalog is Lake Formation-governed → adds lakeformation:GetDataAccess to the harvest data role's per-invocation session policy so LF can vend S3 creds for governed table data. Set by var.enable_lakeformation; requires adopter-side LF grants + data-location registration (see docs/LAKE_FORMATION.md). Unset → plain IAM catalog access
OKF_HARVEST_MODEL	harvest model id — the fallback default used when a harvest request omits model (default us.anthropic.claude-opus-4-8). An anthropic.* id runs on the Bedrock Converse API (ChatBedrockConverse); an openai.* / gpt-* id (e.g. openai.gpt-5.6-sol) runs on the Bedrock Mantle OpenAI-compatible endpoint (ChatOpenAI, bearer-token auth via aws_bedrock_token_generator). The prefix selects the provider; see agent._build_model
OKF_HARVEST_MODEL_CATALOG	(Control API) JSON array of {model, label, efforts, default_effort} — the models + efforts the UI picker offers and the Control API validates a per-harvest model/effort against. From var.harvest_model_catalog; unset → okf_core.harvest_models.DEFAULT_CATALOG. The UI receives the same catalog base64-encoded as VITE_HARVEST_MODEL_CATALOG (base64 so it survives deploy.sh's eval "export k=v")
OKF_HARVEST_MANTLE_REGION	AWS region for the Bedrock Mantle endpoint when OKF_HARVEST_MODEL is a GPT id (default us-east-2). Independent of AWS_REGION — GPT-5.x on Mantle is only in us-east-2/us-west-2, while the harvest runtime may deploy elsewhere. Drives both the Mantle base URL and the region the bearer token is minted for. Ignored on the Converse path
OKF_HARVEST_MANTLE_USE_RESPONSES_API	selects the Mantle API surface (default true → OpenAI Responses API on the /openai/v1 path, which is what GPT-5.x requires). Set false for a gpt-oss model (Chat Completions on /v1). GPT path only
OKF_HARVEST_MANTLE_BASE_URL	override for the Mantle base URL (default https://bedrock-mantle.<region>.api.aws/openai/v1 for Responses, .../v1 for Chat Completions; region from OKF_HARVEST_MANTLE_REGION). GPT path only
OKF_HARVEST_MANTLE_READ_TIMEOUT / OKF_HARVEST_MANTLE_MAX_ATTEMPTS	httpx read timeout (s) and retry budget for the ChatOpenAI Mantle client (defaults 600 / 5, mirroring the Converse knobs). The botocore OKF_HARVEST_BEDROCK_* knobs do NOT apply to the GPT path
OKF_HARVEST_EFFORT	reasoning effort. On Converse, passed verbatim to Bedrock output_config.effort (default xhigh; valid values are model-specific). On the GPT path it maps onto OpenAI's reasoning_effort scale — verbatim on GPT-5.6 (which added max above xhigh), so low/medium/high/xhigh/max all pass through unchanged. Which efforts a given model accepts is model-specific (an older GPT id rejects max); the model catalog is the trust boundary that only offers a level a model supports
OKF_HARVEST_MAX_TOKENS	harvest model max output tokens. Default is provider-aware when unset: 128000 for Converse (Opus 4.8), 32000 for GPT. An explicit value always wins
OKF_HARVEST_MAX_SUBAGENT_CONCURRENCY	how many dynamic subagents run at once on a task() fan-out (default 5). This lowers langchain_quickjs's per-REPL task() semaphore, so a Promise.all keeps at most this many crawls in flight and queues the rest. It is not config.max_concurrency — the fan-out is a QuickJS Promise.all, not a LangGraph batch, so only the semaphore bounds it. The same value bounds the run_review tool's in-flight cluster pipelines.
OKF_HARVEST_REVIEW_DISPATCH_TIMEOUT_S	wall-clock cap per run_review dispatch (one reviewer or one fixer; default 1800). On timeout the dispatch is cancelled and its cluster is recorded as failed — retryable via run_review(cluster_ids=[...]).
OKF_HARVEST_REVIEW_CLUSTER_SIZE	docs per run_review review cluster (default 7). The supervisor-owned hubs — datasets/* overview docs and references/usage_guardrails — are excluded from clustering entirely (they'd hub-steal unrelated spokes, and only the supervisor may edit them; corrections reach it as propagation notes).
OKF_HARVEST_BEDROCK_READ_TIMEOUT	botocore read timeout in seconds for the harvest bedrock-runtime client (default 600). Botocore's 60s default is too low: one xhigh Opus 4.8 turn can generate for minutes, and a slow Converse response would otherwise raise ReadTimeoutError and fail the harvest.
OKF_HARVEST_SQL_MAX_ROWS	soft row cap on the agent-facing run_sql tool's result (default 200). Collection stops at the cap and the tool reports truncated: true — a hint to the agent to add a LIMIT or aggregate — instead of buffering an unbounded result into its context. Distinct from the benchmark grader's hard OKF_BENCHMARK_GRADER_MAX_ROWS (which raises: a truncated set can't be equality-graded)
OKF_HARVEST_PROFILE_ENABLED	"0" disables the snapshot-time column profiles (.metadata/profile/<table>.md — null share, ~distinct, min/max, top-K values; see harvest/profile.py). Default on. Profiles are best-effort: any failure downgrades to a manifest note, never fails the snapshot
OKF_HARVEST_PROFILE_SAMPLE_ABOVE_BYTES	byte-size threshold above which a table is profiled from a sample instead of a full scan (default 1 GiB). The size comes from the catalog's size hint; a hint-less ICEBERG table is sized exactly from its $files metadata sum (the source's iceberg_data_bytes capability — Iceberg Glue Parameters never carry Hive stats); any other table with NO hint is treated as large. Sampled sheets are stamped INDICATIVE — value lists from a sample are never treated as closed enums
OKF_HARVEST_PROFILE_TARGET_SAMPLE_BYTES	the scan budget one sampled profile aims for (default 256 MiB) — the sample percent is target/size, clamped to 0.01–100
OKF_HARVEST_REL_QUERY_TIMEOUT_S	per-query timeout in seconds for the relationship pass's SKETCH scans (default 60). Doubles as the cost bound on each unknown-size last-resort sketch attempt — a cancelled attempt bills at most this long a scan and produces nothing rather than wrong evidence. The join/grain probe queries keep the source default
OKF_HARVEST_REL_SMALL_TABLE_BYTES	the feasibility-FREE zone of the full-scan policy (default 10 GiB — the pre-band gate value): tables measured at or under it are probed/sketched with no rate requirement; between it and OKF_HARVEST_REL_MAX_TABLE_BYTES a proven rate is required (see that row). Clamped to the ceiling when set above it
OKF_HARVEST_PROFILE_ENUM_MAX_DISTINCT / OKF_HARVEST_PROFILE_TOPK / OKF_HARVEST_PROFILE_MAX_ENUM_QUERIES	value-list bounds: only columns with ~distinct ≤ the first (default 50) get a value list, capped at TOPK values (default 20), at most MAX_ENUM_QUERIES per table (default 15, most enum-like first). Higher-cardinality columns report the count only ("not enumerated")
OKF_HARVEST_PROFILE_MAX_COLUMNS / OKF_HARVEST_PROFILE_BUDGET_S / OKF_HARVEST_PROFILE_QUERY_TIMEOUT_S	remaining cost caps: columns profiled per table (default 100), overall wall-clock budget for the whole profiling pass (default 1800s — tables past it are skipped-budget in the manifest), and per-query timeout (default 60s — reliably tables to ~50 GB at Athena's dependable scan rates; a cancelled query bills its partial scan and yields neither a sheet nor the size measurement, so RAISE it per deployment — ~300s buys the 100–500 GB band — when such tables are worth profiling to completion; every query's timeout is CLAMPED to the budget remaining, so no setting lets one query overrun the pass). The pass-1 scan doubles as a size AND throughput measurement: its data_scanned_bytes and engine_ms (bytes/time = the table's layout-aware scan rate, powering the relationship pass's proven-band feasibility check) plus per-column {distinct, null_pct} persist as sibling keys on the table's domains.json entry (cache-carried) and feed the relationship pass — observed bytes as a sizing rung, column stats to rank sketch columns. Reuse makes re-runs cheap: profiles persist on the mount and are fingerprint-keyed (catalog update time + version + column set), so incremental runs re-profile only the changed table and cross runs only mismatches; a full harvest always re-profiles
OKF_HARVEST_BEDROCK_CONNECT_TIMEOUT	botocore connect timeout in seconds (default 10)
OKF_HARVEST_REL_ENABLED	"0" disables the snapshot-time RELATIONSHIP EVIDENCE pass (.metadata/relationships/ — joins/<a>__<b>--<key>.md with match rates both ways / cardinality / orphan samples, and grain/<table>.md with key uniqueness; see harvest/relationships.py). Candidates are enumerated mechanically (shared key-like column names — _id/_key/_nbr/_sk suffixes and table-naming prefixes, holder→home-table pairing) and probed by the SAME SQL cores as validate_join/check_grain (harvest/probes.py) — a plain loop with no model, so authors read verified evidence instead of spending agent turns re-probing. Best-effort: failures are manifest rows, never a failed snapshot. Default on
OKF_HARVEST_REL_BUDGET_S / OKF_HARVEST_REL_MAX_PAIRS / OKF_HARVEST_REL_MAX_TABLE_BYTES / OKF_HARVEST_REL_MAX_TABLES_PER_KEY / OKF_HARVEST_REL_MAX_GRAIN_PER_TABLE	the evidence pass's cost caps: wall-clock budget (default 1800s — subjects past it are skipped-budget; GRAIN probes run FIRST — one cheap aggregate per table — so a candidate-flooded join loop can never starve them, and the sketch scan is hard-capped at HALF the budget so collection can't eat the probes' time — both seen live on MusicBrainz: 3,469 skipped, 0 grain), join-pair cap (default 100; it counts PROBED pairs only — cache reuses, TYPE MISMATCH sheets, and size-skips are free — and when it binds the probe budget is SPREAD, least-covered table first with sketch nominations winning ties (a skipped renamed-key pair is unrecoverable live); pairs beyond it are skipped-cap without even being sized), per-side FULL-scan policy in three tiers — at or under OKF_HARVEST_REL_SMALL_TABLE_BYTES (default 10 GiB) unconditionally; over OKF_HARVEST_REL_MAX_TABLE_BYTES (default 50 GiB) never; in the PROVEN band between them only when a profile-measured scan rate (the table's own scanned_bytes/scan_ms, else the catalog's conservative p25) predicts completion within 90% of the probe's query timeout — no measurement in the band means it does not run (skipped-slow on grain rows, distinct from the hard skipped-size), so the raised ceiling bounds worst-case SPEND while the proof bounds worst-case TIME; the measured rates and implied byte ceiling are logged as operator guidance (job logs only, no feed line); when exactly ONE side is refused a full scan — the enterprise fact→dim shape (or a proven-band side without proof: sampling makes it feasible by construction) — the pair is probed with that side SAMPLED toward OKF_HARVEST_REL_SAMPLE_TARGET_BYTES (default 256 MiB, Athena TABLESAMPLE SYSTEM so billed bytes actually shrink) against the FULL small side, reporting ONLY the unbiased sampled→full direction on an INDICATIVE sheet (ok-sampled in the manifest) with every sampled-side number from ONE scan (each TABLESAMPLE reference draws an independent sample, so the matched count rides a LEFT JOIN inside the same query); both sides over, an unmeasurable size, or a source without sql_sampled_ref still skipped-size — and grain probes are never sampled (sample-uniqueness proves nothing); a table with NO byte-size hint is MEASURED — first for free from the profile pass's observed data_scanned_bytes (a COMPLETE-scan measurement of the profilable columns, never an early-exit lower bound, and the honest number for what a probe would bill), then via the source's estimate_table_bytes — for ICEBERG tables the exact $files metadata sum first (a manifests-only query through the engine's own permissions; an S3 listing would overcount every retained snapshot until VACUUM), then an S3 listing of the table's location, no query, early-exit at the gate — because DDL-registered tables carry no totalSize Parameter (only crawlers/ETL write it) and pure assume-large skipped every probe on such catalogs; only when even the measurement can't tell (a view, listing denied, a partitioned table whose root lists empty) does the table count as oversized. The SAMPLE PERCENT is never sized from the gate measurement (it early-exits, so it is only a lower bound that would over-sample a huge table by orders of magnitude) — the pass re-lists to completion for the sampled side and skips the pair when even that can't tell. The workgroup has no scan cutoff, so this gate is the only bound), fan-out cap for a key shared by many tables (default 6; beyond it only holder→home pairs are probed, or none when no home table exists), and grain candidates per table (default 2). Reuse mirrors the profile cache: sheets are fingerprint-keyed on BOTH tables' catalog fingerprints, incremental runs re-probe only pairs touching the changed table, a full harvest re-probes everything
OKF_HARVEST_REL_SKETCH_ENABLED / OKF_HARVEST_REL_SKETCH_K / OKF_HARVEST_REL_SKETCH_MIN_CONTAINMENT / OKF_HARVEST_REL_SKETCH_MAX_COLUMNS	the NAME-BLIND value-sketch nominator inside the relationship pass (full harvests only; default on): per key-ish/text column a KMV bottom-k sketch (k default 256, error ~1/√k) is computed during the snapshot — Athena in ONE columnar scan per table via min(DISTINCT from_big_endian_64(xxhash64(...)), k) (both tokens matter: min(x, n) is a row-level order statistic so a multiset sketch collapses on fact-side FKs, and Trino's xxhash64 returns VARBINARY so without the decode every cell is unparseable and the nominator silently disables itself), Redshift one cheap per-column query via FNV_HASH — and all cross-table column pairs are compared in memory; pairs whose estimated value containment ≥ the threshold (default 0.5) are nominated into the SAME probe/verdict pipeline as name/role candidates (via="sketch" note on the sheet). This is what catches renamed keys (cds = cdscode). Guardrails: the CONTAINED side must exceed the enum-domain size (a tiny domain is inside everything — the dense-integer trap) and must NOT be its own table's PK (bare id or a self-naming key — FKs point AT PKs, so a PK contained elsewhere is the dense-surrogate coincidence: 301 junk id↔id nominations on MusicBrainz), containment INTO a dense INT PK additionally requires the contained column to NAME the containing table (artist → artist.id, begin_area → area.id — a 1..N surrogate numerically contains every smaller int column, so values alone prove nothing there; hash/UUID PKs and non-PK containers like cdscode stay name-free since sparse domains don't collide by chance), same-named holder pairs collapse toward the column's HOME table exactly like the name source (tag in fifteen *_tag tables pairs each holder with tag, never pairwise; widely-shared same-named columns with no home are refused), max 12 columns sketched per table (eligibility is TYPE-based — int/text families only, name-blind; under the cap columns RANK by name tier first, then by the profile pass's per-column stats — high-distinct, low-null first, with columns whose OBSERVED distinct count is at or under the enum-domain floor excluded outright: the guardrails refuse their nominations anyway, so a slot spent on them buys nothing), dedup against name/role nominations, and the shared budget/size gates apply — tables MEASURED over the gate are never sketched, while UNKNOWN-size tables get bounded LAST-RESORT attempts (visited after every measured table, each capped by OKF_HARVEST_REL_QUERY_TIMEOUT_S, abandoned after two consecutive failures; a timed-out sketch yields NOTHING, never wrong evidence, and a nomination whose probe the size gate still refuses persists as a no-verdict NOMINATED sheet routing the author to a live validate_join), and a sketch is always its OWN full scan of just the sketched columns, never the profile pass's row sample (per-VALUE facts don't survive row sampling; a sampled sketch would publish deflated containment that reads as "unrelated"). Sketch nominations are the one candidate source that does not re-enumerate from the catalog, so full runs persist them in relationships/candidates.json and incremental/cross runs (which never sketch) revalidate + merge them — without it the first non-full run would silently drop every sketch-discovered sheet. Known blind spot: key pairs whose distinct counts differ by more than ~k/8 are beyond the sketch's resolving power — counted in the harvest log, falls back to live probing
OKF_HARVEST_BEDROCK_MAX_ATTEMPTS	botocore retries.max_attempts in adaptive mode (default 5); retries transient throttles and timeouts instead of failing the run
OKF_BENCHMARK_MAX_CONCURRENCY	how many benchmark solver ReAct loops (and judge reviews) run at once in a Benchmark Studio run (default 10). Its own asyncio.Semaphore — each solver is one in-flight model request at a time, so this is the peak concurrent Bedrock requests from the benchmark. Raise on generous quota, lower on ThrottlingException. mode: "benchmark" runs only.
OKF_QGEN_MAX_CONCURRENCY	peak concurrent question-author agents in a mode: "generate_questions" run (default 4 — authors are long-lived explorers, not one-question solvers). Gold validation reuses the grader's OKF_BENCHMARK_GRADER_TIMEOUT_S/_MAX_ROWS caps so a generated gold can never be DISCARDED at grading time.
OKF_BENCHMARK_ATHENA_CONCURRENCY	how many benchmark grading queries (gold/predicted SQL EX executions) run against Athena at once (default 15); size under the Athena workgroup's concurrent-DML limit. mode: "benchmark" runs only
OKF_BENCHMARK_GRADER_TIMEOUT_S	per-query timeout in seconds for the SQL EX grader's Athena executions (default 60). A timed-out query is best-effort cancelled (stop_query_execution) so it doesn't keep holding a workgroup slot; the timeout classifies as TRANSIENT (retried with backoff, never memoized). Grading only — the harvest's own sample_rows/run_sql keep their own timeout. mode: "benchmark" runs only
OKF_BENCHMARK_GRADER_MAX_ROWS	row cap on one grading query's result set (default 50000). Past the cap collection stops and the outcome is classified — gold → DISCARDED ("gold result exceeds N rows"), prediction → FAIL — instead of buffering an unbounded result. Grading only. mode: "benchmark" runs only
OKF_INCREMENTAL_ICEBERG_COMMITS	(incremental) how the change handler treats an Iceberg data commit — a Glue version bump with an EMPTY column diff, no changed partitions, and a version delta of exactly one (every Iceberg write bumps the version by swapping metadata_location, so per-commit re-harvests on a busy table would run continuously). Default skip: record the new version (so the nightly reconcile doesn't re-detect it) and absorb the event — Iceberg table docs then refresh on schema changes and full harvests, not on data writes. review restores the old always-invoke behavior. A wider version gap, a non-empty diff, or a never-seen table always invokes (a dropped event could hide a schema change the two-latest diff can't see); non-Iceberg version bumps are untouched — a Hive property-only bump still re-reviews
OKF_POLICY_BUILD_ENABLED	(harvest, incremental) "true" → author and refresh policy documents from the wiki (default false). The runner's post-complete follow-on build and the rebuild authority both no-op when unset — an ALREADY-authored document keeps being usable, since usability is decided by the row's ar_build_status + fingerprint, not by this flag. Set from var.enable_policy_build
OKF_COMPUTATIONS_ENABLED	(consumption MCP + Control API) "true" → run_computation EXECUTES (Athena / Redshift Data API); unset/false → it returns the rendered SQL with a note, and no engine client is built (the IAM grant is absent too). From var.enable_attested_computations, default false — machine-vended MCP creds must not gain source-data read from a routine apply. The CHAT surface deliberately ignores this flag: its run_computation is ALWAYS bound (the sanctioned path never costs the raw-SQL opt-in) and EXECUTES only when var.enable_chat_sql granted the clients — without them it returns the rendered SQL un-executed
OKF_COMPUTATION_MAX_ROWS / OKF_COMPUTATION_TIMEOUT_S	run_computation result-row cap (default 200; the rest truncate with truncated: true) and per-execution timeout in seconds (default 120, best-effort engine cancel on expiry). Row cap set from var.computation_max_rows; the timeout is env-read only
OKF_USER_POOL_ID	Cognito user pool id (the Control API vends and revokes M2M app clients in this pool)
OKF_MCP_SCOPE	the custom scope (okf-mcp/invoke) granted to vended M2M clients; must match the consumption authorizer's allowed_scopes
OKF_HARVEST_LOG_GROUP	the harvest runtime's CloudWatch log group the Control API reads to serve the live step feed (GET /harvest/{domain}/{dataset}/events). Derived by Terraform as /aws/bedrock-agentcore/runtimes/<runtime-id>-DEFAULT (overridable via var.harvest_log_group). Unset/incorrect → the feed returns an empty batch; status polling is unaffected
OKF_WEB_SEARCH_ENABLED	(chat runtime) "true" → offer the agent the public web_search tool. Set from var.enable_web_search. Requires OKF_WEB_SEARCH_GATEWAY_URL too: with either missing the tool is simply not wired (and the role carries no bedrock-agentcore:InvokeGateway grant anyway)
OKF_WEB_SEARCH_GATEWAY_URL	the AgentCore Gateway MCP endpoint fronting the built-in web-search connector (https://<gateway-id>.gateway.bedrock-agentcore.us-east-1.amazonaws.com/mcp). /mcp is appended if absent. The runtime speaks MCP JSON-RPC to it (initialize → tools/call) signed with SigV4
OKF_WEB_SEARCH_REGION	region the gateway lives in, i.e. the region web_search's SigV4 is signed for (default us-east-1). Independent of AWS_REGION — the web-search connector is offered in us-east-1/eu-west-1/ap-northeast-1 only (var.web_search_region picks one), so a query may leave the deployment's region (it never leaves AWS)
OKF_WEB_SEARCH_TOOL_NAME	the gateway-side tool name, <target-name>___WebSearch (AgentCore prefixes every tool with its target's name, joined by THREE underscores). Set by Terraform to save a round trip; empty → the runtime discovers it via tools/list and caches it
OKF_WEB_SEARCH_MAX_RESULTS	default results per search when the agent doesn't pick a count via the tool's max_results arg (default 10; 1–25). The connector ranks by relevance — the agent steers time through the query text (and, with the filters flag below, the date bounds), reading each result's publishedDate
OKF_WEB_SEARCH_FILTERS_ENABLED	"true" → web_search also offers the connector-1.2.0 request-level filter args (published_after/published_before, include_domains/exclude_domains). Set by Terraform through the target's version-pin step (web_search.tf), never by hand: a pre-1.2.0 target does not reject a filters argument, it silently ignores it, so an ungated runtime would hand the model constraints that quietly don't apply. Unset → the tool keeps the plain query+max_results surface
OKF_CHAT_MEMORY_ID	(chat runtime + Control API) the AgentCore Memory resource id for long-term chat memory (see "Long-term chat memory"). Set from var.enable_chat_memory ? chat_memory_id : "" — empty disables the feature end to end: the runtime builds no memory client (no recall, no event writes) and the Control API's /memory* routes return 404. The extraction/consolidation prompts + model live on the DURABLE stack's strategy resource (var.chat_memory_model), not in env
OKF_CHAT_MEMORY_DEFAULT_ON	(chat runtime + Control API) what a MISSING per-user memory switch row means: true (default) = opt-out — memory on until the user switches it off; false = opt-in — off until explicitly enabled on the Memory page. Set from var.chat_memory_default_on on BOTH services so the switch the page shows agrees with what the runtime does; rows users already set are never affected
OKF_CHAT_GUARDRAILS_GATE_ENABLED	(chat runtime, env-read — not Terraform-plumbed) default true → read_page on any page of a dataset is DENIED at the tool boundary until that dataset's references/usage_guardrails has been read in the thread (chat.guardrails_gate.GuardrailsGateMiddleware). Per-dataset marks live in CHECKPOINTED agent state (guardrails_read channel, dict-merge reducer), so resumes remember; the guardrails read itself always passes and marks on any completed attempt (a guardrails-less legacy bundle can't lock out). Browse/search tools are never gated. "false"/"0"/"no"/"off" disables
OKF_CHAT_POLICY_CHECK_ENABLED	(chat runtime) "true" → the mid-turn policy checks may be armed per run via features: ["sql", "policy:*"] (default false). Unset → no checker is ever constructed, whatever the client sends — the master gate above the per-run opt-in, set from var.enable_policy_checks
OKF_CHAT_POLICY_CHECK_MODEL	(chat runtime) the policy checks' model id, serving BOTH the curated-question rewrite (no reasoning pass — extraction) and the JUDGE fleets. Default global.anthropic.claude-sonnet-5 — judges run classifier-style on every family (thinking off + temperature 0 on Anthropic, reasoning "none" on openai.* ids like openai.gpt-5.6-terra, forced report_violations either way); from var.chat_policy_check_model, which also drives the chat role's Mantle grants, so an openai.* value needs no extra wiring. Code-level companions (env-read, not Terraform-plumbed): OKF_CHAT_POLICY_SHARD_SIZE (default 10 — policies per mini-judge), OKF_CHAT_POLICY_QUERY_TIMEOUT_S (default 60 — residual wait after the query returns) and OKF_CHAT_POLICY_QUERY_MAX_PER_TURN (default 3 — judged analytical queries per turn). Deploy-time only — deliberately NOT validated against OKF_CHAT_MODEL_CATALOG, which is the trust boundary for CLIENT-supplied models
OKF_POLICY_PREPROCESS_MODEL	(harvest runtime) model id for the policies.yaml AUTHORING AGENT (harvest.ar_author — full reasoning; policy distillation is judgment work). Default global.anthropic.claude-sonnet-5, from var.policy_preprocess_model, which also drives the harvest role's Mantle grants. Code-level companions (env-read, not Terraform-plumbed): OKF_POLICY_AUTHOR_EFFORT (default high), OKF_POLICY_AUTHOR_THINKING_BUDGET (pre-adaptive models like Haiku 4.5 — e.g. 48000), and OKF_POLICY_MAX_RULES (default 60 — the author gate's policy-count BACKSTOP against enumeration pathology; the prompt's proportionality guidance, not this cap, is what sizes a document to its dataset). The incremental Lambda runs NO models: authoring dispatches to the runtime
VITE_CHAT_POLICY_CHECK	(UI, from ui_env) shows the composer's Policy feature (the "+" menu field with Computational / Behavioural / Strict) and the Reasoning sidebar page. Defaults ON — only the literal "false" hides them (an unset var must not silently drop the affordance). A DISPLAY gate only, same pattern as VITE_CHAT_SQL_ENABLED; the server-side OKF_CHAT_POLICY_CHECK_ENABLED + feature normalization are the real boundary

HTTP and auth

• Control API and MCP requests carry Authorization: Bearer <Cognito token>.
• The API Gateway HTTP API JWT authorizer uses audience = app client id, issuer = https://cognito-idp.<region>.amazonaws.com/<poolId>.
• The consumption MCP AgentCore authorizer uses discoveryUrl = <issuer>/.well-known/openid-configuration. Inbound trust is scope-based (allowedScopes = ["okf-mcp/invoke"]), not a client allowlist, so a newly vended machine client is accepted with no infra change. allowedAudience is unusable here because Cognito M2M client_credentials access tokens carry no aud.

MCP machine credentials (apps and agents)

• An okf-mcp resource server defines the invoke scope, giving the full scope string okf-mcp/invoke. The web SPA also carries this scope, so human sessions pass the same authorizer check.
• The Control API vends credentials as Cognito M2M app clients (client_credentials grant, GenerateSecret=true, scope okf-mcp/invoke): POST /credentials {name} returns {client_id, client_secret} once; GET /credentials returns metadata from the registry; DELETE /credentials/{client_id} deletes the app client and revokes it immediately. This needs IAM cognito-idp:{Create,Delete,Describe}UserPoolClient on the pool.
• To get a token, an app POSTs to the Cognito token endpoint with HTTP basic auth client_id:client_secret and body grant_type=client_credentials&scope=okf-mcp/invoke, then sends the resulting access token as Authorization: Bearer <token> to the MCP server. Tokens are short-lived (60 minutes) and meant to be cached; the token endpoint is capped at 150 RPS per account and Region.