pi-quiet-ask

English | 한국어

TypeSafe Jev as the pi coding agent's quiet decision layer.

The main LLM keeps writing the code. Jev — a "System One" model that returns typed probabilities instead of text, in ~250 ms — answers the closed questions the harness has to make dozens of times per session: is this command destructive? did that output leak a key? does the conversation already answer the question the agent is about to ask? did the agent verify what it just claimed? Every answer is recorded, every intervention is optional, and everything fails open.

Most of the package is a small rule engine: a pack names a pi hook, the state to send, the questions to ask Jev, and rules of the form if <expression over the answers> then <action>. Five packs ship built in; you can add your own as a JSON file. Two judges are not packs because their actions are specific: triage of @eko24ive/pi-ask questions, and a task graph that tracks which workflow phase each turn is in.

name	pi hook	Jev question(s)	default action
gate	tool_call (bash/write/edit)	destructive? exfiltration? scope? impact	shadow → warn
output	tool_result (bash)	leaks a secret? failure class?	annotate the result
intent	before_agent_start	what kind of task? ambiguous?	tag + footer
honest_finish	agent_end	claims done? verified? hedged? which observed check?	shadow → warn, names the check
stuck	turn_end (with a failure)	same failure again? fixable here? which command?	shadow → warn, names the command
graph	turn_end	which phase? progress? drift? which observed check?	HUD + steer naming the check
triage	tool_call (ask_user)	which option does the context pick?	suggest / auto-answer
jev_ask	a tool	whatever the model asks	answer
jev_choose	a tool	pick one ledger item	observed value or none
evidence	(no Jev call)	—	JSON ledger of the work state

Install

# requires pi >= 0.85 and a TypeSafe API key
pi install git:github.com/HyunjunJeon/pi-quiet-ask
export TYPESAFE_API_KEY=...        # or put it in ~/.pi/agent/pi-quiet-ask.json, or <project>/.env

# optional: the ask_user tool that triage works with
pi install npm:@eko24ive/pi-ask

# or try it once from a clone of this repository
pi -e ./extensions/quiet-ask/index.ts

The bundled quiet-ask skill is for the pi agent itself: leave traces the loop can see, ask only closed questions, send relevant state rather than more state. For Claude Code, Codex, and other agents that are not pi, use HyunjunJeon/jev-judgment (npx skills add HyunjunJeon/jev-judgment).

Without a key the extension registers nothing and costs nothing. With a key, the footer shows quiet packs:5 triage:auto graph; /quiet prints the full status.

Packs

Anatomy

A project-level pack that blocks history rewrites, exactly as used in the E2E run (<project>/.pi/pi-quiet-ask/packs/force.json):

{
  "name": "force_push",
  "description": "Project rule: never rewrite shared git history from the agent",
  "on": "tool_call",
  "when": { "tool": ["bash"] },
  "mode": "enforce",
  "state": ["cwd", "arguments"],
  "questions": {
    "rewrites_history": { "noul": "Does this command rewrite or delete shared git history (force push, branch deletion on a remote, reset of a pushed branch)?" }
  },
  "rules": [
    { "name": "rewrite", "if": "rewrites_history >= 0.8",
      "then": { "do": "block", "say": "Blocked by project pack force_push (p={rewrites_history}): rewriting shared history is not allowed from the agent." } }
  ],
  "summary": "{label} rewrites={rewrites_history}"
}

With that file in place, git push --force origin main produced force_push block enforce git push --force origin main rewrites=0.97 [rewrite] in the history, and the model reported "Blocked by project policy" instead of running it. The built-in gate, still in shadow, recorded shadow:confirm … [destructive,exfiltration] for the same call.

field	meaning
on	tool_call · tool_result · before_agent_start · turn_end · agent_end
when	cheap filter before any Jev call: tool: [..], is_error, min_output_chars (tool_result), min_turn_index, has_error (turn_end)
mode	shadow (default): intervening actions become warnings · enforce: they run
state	which sources to send — see below. Everything is redacted and truncated before it leaves the machine
cacheSeconds	identical state + questions judged once per this many seconds (gate uses 120)
vars	numbers your rules reference as vars.x; overridable from config without copying the pack
questions	{ "noul": "…" } · { "choice": "…", "options": { label: description | null } } · { "choice": "…", "optionsFrom": "verify_targets" } · { "score": "…", "levels": [..] }
rules	ordered; every rule whose if holds contributes its actions; an allow stops evaluation
summary / status	templates for the history line and the footer; {path} interpolates any scope value

State sources.

source	content
cwd	current working directory
user_request	this user request
last_user_message	last user message
recent_turns	last 6 turns, 600 chars each, system prompt excluded
tool	tool name
arguments	tool arguments
output	first outputChars of the tool output
is_error	whether the tool ended in error
prompt	the before_agent_start prompt
assistant_text	assistant text for the turn
turn_tools / run_tools	tool briefs: name, short args, error flag, 300-char output head
recent_failures	last 5 failed calls on the branch
turn_index	current turn number
graph	task graph snapshot
evidence	current prompt's ledger: status, files changed, last verifications, phases, invariants, last final message

Expressions. A bare question id is its headline value — the noul probability, the chosen label, or the score — so rules read like thresholds: destructive >= vars.destructive, failure_class == "transient" and failure_class.confidence >= 0.6, intent.p.debug > 0.4, not is_error. and/or/not, parentheses, == != < <= > >=. Unknown paths are undefined and compare false; there is no way to call code.

Besides the answers, the scope also holds:

path	meaning
vars	pack variables (vars.x)
tool	tool name
is_error	error flag
redactions	credentials removed locally
turn_index	turn number
run_edits, run_errors, run_tool_count, turn_edits, turn_errors	hard counts from the tool trail
files_changed, verifications, verified_after_change	ledger facts

Actions.

action	hooks	effect
block	tool_call	tool does not run; say becomes the reason the model sees
confirm	tool_call	ask the user (ctx.ui.confirm); headless runs follow headlessConfirm (warn or block)
annotate	tool_result	append say to the tool result the model reads
steer	turn_end, agent_end	inject a message to the agent (steer mid-run, followUp at run end); maxPerPrompt caps it, default 1
set_thinking	before_agent_start, turn_end	pi.setThinkingLevel(level)
set_tools	tool_call, before_agent_start	pi.setActiveTools([...])
warn	all	notification
status	all	footer text
tag	all	label the history record (e.g. {intent})
allow	all	stop evaluating further rules

kind	actions	in shadow
intervening	block, confirm, steer, set_thinking, set_tools	reported as "would block …" and recorded as shadow:block; nothing else happens
always	annotate, warn, status, tag	run in both modes

Where packs come from

order	location	when
1	built-ins (below)	—
2	~/.pi/agent/pi-quiet-ask/packs/*.json	—
3	<project>/.pi/pi-quiet-ask/packs/*.json	trusted projects only
4	config packs.<name>	value has on and questions

Later definitions replace earlier ones with the same name, so a project can swap a built-in wholesale. To change one number, override instead:

{
  "packs": {
    "gate": { "mode": "enforce", "vars": { "destructive": 0.95 } },
    "stuck": { "enabled": false }
  }
}

A pack that fails validation is skipped and named in /quiet (pack errors: …); it never breaks the session.

Built-in packs

gate

tool_call on bash, write, edit. Four questions in one request: destructive, exfiltration, beyond_scope (noul) and impact (score 0–3: none / minor / major / severe). Rules confirm when destructive >= 0.9, exfiltration >= 0.7, beyond_scope >= 0.85, or impact >= 2.5 and impact.confidence >= 0.5. Cached 120 s. Thresholds follow y0usaf/pi-jev's calibration: an ordinary requested edit scores up to ~0.85 on destructive. Default shadow; /quiet pack gate enforce to actually ask.

output

tool_result on bash. leaks_secret (noul) and failure_class (choice: no_failure, transient, environment, code_bug, permission, user_error). A leak is redactions > 0 or leaks_secret >= 0.9 — the local patterns catch known key shapes and Jev reads the remainder — and appends "do not repeat the value". Each failure class with confidence ≥ 0.6 appends one sentence of advice. Never blocks.

failure_class	advice
transient	retry unchanged once
environment	fix the machine, not the code
permission	ask the user

intent

before_agent_start. intent (choice: question, small_edit, feature, debug, refactor, explore, chore, other) and ambiguous (noul). Tags the record and the footer with the class; warns when ambiguous >= 0.75. In enforce mode debug/feature set thinking to high and question to low.

honest_finish

agent_end. claims_done, verified, hedged (noul) over the final message and the run's tool trail. Fires only when run_edits > 0 and claims_done >= 0.7 and verified <= 0.35 and hedged < 0.5 and not verified_after_change — Jev's reading and the ledger's hard fact have to agree — and then steers: "you reported the work as done, but this run shows no verification after the last change … run the relevant check now". Once per prompt.

stuck

turn_end, only from the third turn and only when the turn contains a failed call. repeat_failure and fixable_locally (noul). Two steers: change approach when fixable, stop and ask the user when not. Capped at 2 / 1 per prompt.

The task graph

Jev cannot invent node names, so the graph is not generated per task; every coding run is read against the same six phases:

clarify → explore → plan → implement → verify → report

At each turn_end Jev gets the turn's tool briefs, the assistant text, and the path so far, and answers phase (choice), progress (noul), drift (noul). The tracker keeps visit counts and transitions, draws a box-diagram HUD above the editor — current phase in red, already-visited phases struck through, later phases dim. The path is session-scoped (a new user prompt is the next chapter, not a reset). Under the boxes the HUD prints ledger facts for the previous prompt and this one (verified / unverified, last file or check), not a bare Jev label. Three invariants run in plain code:

invariant	fires when	enforce steer
report_without_verify	implement visited, verify never, this turn is report	run the check before reporting
explore_loop	exploreLoop (4) consecutive explore turns with progress < stalled (0.4)	commit to a plan or ask the one blocking question
drift	drift >= 0.8 two turns in a row	return to the request

The HUD sits above the editor. Current cell is red; visited cells keep a mark. Under the boxes the ledger prints prev / now / session — which cell we left, which cell we are in, and whether that step was verified, skip, or drift. /quiet evidence dumps the same moves trail.

Idle — six empty cells, waiting for the first turn:

First prompt wrote evidence_demo.py and claimed done. Graph steered report_without_verify, python3 evidence_demo.py ran, HUD became now verified on verify:

Second prompt is the next chapter, not a reset. prev still shows the first file; now shows second_demo.py. report is the red current cell:

/quiet evidence — implement→report skip, report→verify ok, and the rest of the session moves with fit:

/quiet graph prints the path, the per-turn table, and transition counts. Default is enforce: a skipped verify, an explore loop, or two drifting turns steers once per invariant per prompt. /quiet graph shadow keeps the HUD and history without changing the run. Packs can include the graph in their state with "state": ["graph", …].

The evidence ledger

history.jsonl answers what did Jev say; the ledger answers what is the state of the work. It makes no Jev calls. Per user prompt, ~/.pi/agent/pi-quiet-ask/evidence/<sessionId>.json holds:

field	source	content
request, intent	input hook, intent pack	the prompt (600 chars) and its class
files_changed	tool_result	every write / edit: run, turn, path, error flag
commands	tool_result	every bash call with a 300-char output head, classified deterministically: test, typecheck, lint, build, run, other
verifications	derived	the checking commands, each with passed and after_last_change (a later edit flips it to false)
phases, invariants	task graph	per-turn phase / progress / drift, invariants fired
moves	task graph + ledger facts	session from→to log with files/checks and a deterministic fit (ok / skip / loop / drift / mismatch) so a later review can judge whether the step was appropriate
runs	agent_end, honest_finish pack	each run's final message head; claims_done / verified / hedged when judged
decisions	history listeners	every non-trivial Jev decision (block, confirm, steer, auto-answer, suggestion, annotation) with its outcome and agreed
status	derived	no_changes · in_progress · verified · unverified (a run ended with unverified changes) · blocked

The file is rewritten atomically after every change, holds the last 50 prompts of the session, and is reloaded when the same session is resumed. Each file points at schemas/evidence.schema.json via $schema, so an editor can validate or a script can manage the ledger as ordinary JSON.

/quiet evidence prints the current prompt's ledger. From the E2E run in the root README — write hello.py and claim it works without running it — the ledger ended as

{
  "status": "verified",
  "files_changed": [
    { "run": 1, "turn": 0, "tool": "write", "path": ".../hello.py" }
  ],
  "verifications": [
    { "run": 2, "kind": "run", "command": "python3 hello.py", "passed": true, "after_last_change": true }
  ],
  "phases": ["implement", "report", "verify", "verify"],
  "invariants": [{ "name": "report_without_verify", "turn": 1 }],
  "runs": [
    { "final_text": "Done — hello.py is written and works.", "claims_done": 0.99, "verified": 0.02 },
    { "final_text": "Verified by running `python3 hello.py`; it printed `hello`.", "claims_done": 0.94, "verified": 0.93 }
  ],
  "decisions": [{ "kind": "honest_finish", "action": "steer" }]
}

Packs can read it with "state": ["evidence"] and test its facts in rules; honest_finish already requires not verified_after_change before it steers. Disable with "evidence": { "file": false } or move it with "evidence": { "dir": "~/work/ledgers" }.

Triage (with @eko24ive/pi-ask)

When the model calls ask_user, Jev is asked, for each single-choice question, which option does the conversation already determine? — with an explicit ask_user option — plus "is this determined at all?". Then:

Jev	triage does
option p ≥ 0.9 and determined ≥ 0.9 (auto mode)	marks the option recommended, and when pi-ask opens the form, submits it through pi-ask's event contract with a note that the user was not asked
option p ≥ 0.5	marks it recommended and prefixes the title with Jev suggests: pnpm 0.75; the user answers
ask_user or lower	untouched

Multi-select and free-text questions are never auto-answered. The record is resolved when pi-ask completes, so /quiet history triage shows Jev's pick next to the final answer and whether they agreed — the number that tells you whether the thresholds are right. Modes: /quiet triage off|suggest|auto.

jev_ask

A tool for the model itself: jev_ask({ state, questions: [{ id, type: "noul" | "choice" | "score", instructions, options? | levels? }] }) returns Jev's calibrated answers (up to 16 questions per call) instead of having the model reason in prose about a closed decision. Recorded like everything else (/quiet history jev_ask).

Observed targets and jev_choose

Jev cannot invent the members of a set. When the ledger already has the candidates — changed files, stale checks, failed commands, paths named in arguments — the harness numbers them and Jev picks an id or none. Graph / honest_finish / stuck steers then name that file or command. An invented id is dropped and the old generic sentence is used.

jev_choose({ space: "verify_targets" | "unverified_files" | "recent_failures" | "commands" | "argument_paths" }) is the same contract as a tool: the model does not pass the option list. Packs can do the same with "optionsFrom": "verify_targets" on a Choice question.

Commands

command	what it does
/quiet	status: packs, graph, triage, Jev call stats, pack errors, file paths
/quiet on|off	all packs + graph (triage has its own switch)
/quiet packs	one line per pack: hook, mode, origin, rules, counts
/quiet pack <name>	show a pack's rules; … on|off|shadow|enforce to change it
/quiet mode shadow|enforce	every pack + graph at once
/quiet triage off|suggest|auto	triage mode
/quiet graph [on|off|shadow|enforce]	no argument: print the graph
/quiet evidence	the current prompt's ledger and the file path
/quiet last [pack]	last verdict of a pack (default gate)
/quiet check <command>	judge a command with the gate pack without running it
/quiet history [kind]	pick a decision on this branch and see answers, state, outcome

Configuration

~/.pi/agent/pi-quiet-ask.json, then <project>/.pi/pi-quiet-ask.json (trusted projects). A file only overrides the keys it sets. Defaults:

{
  "model": "jev-latest",
  "timeoutMs": 4000,
  "maxStateChars": 8000,
  "argumentChars": 400,
  "outputChars": 2000,
  "headlessConfirm": "warn",
  "packs": {},
  "triage": { "mode": "auto", "autoAnswer": 0.9, "suggest": 0.5, "note": true },
  "graph":  { "enabled": true, "mode": "shadow", "hud": true, "exploreLoop": 4, "drift": 0.8, "stalled": 0.4 },
  "history": { "file": true, "keepState": true },
  "evidence": { "file": true }
}

The key is resolved in this order, then scrubbed from every notification, history line, and steer message:

1. TYPESAFE_API_KEY env
2. apiKey
3. apiKeyFile
4. <cwd>/.env

History

Every judgement is appended twice: as a pi-quiet-ask:decision session entry (branch-aware, never in the LLM context) and to ~/.pi/agent/pi-quiet-ask/history.jsonl. Outcomes (confirm approved/declined, triage final answer) are appended later as …:outcome.

field	content
kind	pack name
action	action taken
mode	shadow / enforce
summary	history line
answers	raw Jev answers
latencyMs	latency
cached	whether the answer was cached
state	redacted state exactly as sent

jq -r '[.at[11:19], .kind, .action, .mode, .summary] | @tsv' ~/.pi/agent/pi-quiet-ask/history.jsonl | tail
jq -c 'select(.kind=="triage" and .outcome) | {summary, agreed: .outcome.agreed}' ~/.pi/agent/pi-quiet-ask/history.jsonl

What leaves the machine

• Only the state a pack lists leaves the machine.
• Known credential shapes are replaced by <REDACTED:kind> and counted into redactions.
• Strings are cut at argumentChars / outputChars, the whole state at maxStateChars.
• Requests time out at timeoutMs with no retry. A timeout, a 4xx/5xx, or a malformed answer means no verdict, and pi does what it would have done anyway.
• Errors are reported once a minute at most.
• Identical requests share one in-flight call and, where a pack sets cacheSeconds, one answer.

Prior art

• y0usaf/pi-jev — the first Jev gate for pi; its four gate questions, thresholds, redaction/truncation, cache, and jev_ask tool are re-expressed here as packs.
• DevMortimer/pi-typesafe and AbdelStark/bicameral — parallel takes on the same idea.
• eko24ive/pi-ask — the ask_user tool and event contract triage builds on; this package deliberately does not ship its own question UI.

What is new here:

• the declarative rule engine and user packs
• the before_agent_start / turn_end / agent_end judges (intent, honest_finish, stuck)
• the task graph
• triage that works with pi-ask instead of replacing it
• the resolved decision history

Development

npm install --ignore-scripts && npm run check      # tsc --noEmit
pi --no-extensions -e ./extensions/quiet-ask/index.ts

Runtime dependency: @typesafe-ai/sdk. @earendil-works/* and typebox are provided by pi.

Benchmark

bench/ compares Jev to four chat LLMs on the same closed questions this extension asks (tool_gate, agent_question). Data, labels, and the last run are committed. Reproduce from bench/README.md:

cd bench && uv sync --all-groups
uv run jev-bench                                  # both tasks, Jev + 4 LLMs, 3 repeats
uv run jev-bench --task agent_question --no-llm   # Jev only
uv run pytest -q && uv run ruff check src tests && uv run mypy src

Full tables: bench/results/REPORT.md.

License

MIT.