pi-jev

Claude Code-style auto-mode classifier for the Pi coding agent, powered by TypeSafe Jev.

Before a classified tool runs, Jev answers four safety questions, one explicit-request question, and one advisory relevance question in one call. Only the safety probabilities drive allow / ask / deny, except that a confident explicit request skips irreversible. After a tool returns, Jev screens the output for prompt injection and leaked secrets. Read-only tools skip the pre-call classifier (they still get output screening). A short hard-deny list catches rm -rf / without a network round-trip.

This is not a sandbox. The classifier can be wrong. Use it to cut permission prompts, not to replace review on sensitive work.

Install

pi install git:github.com/JohnsonRan/pi-jev

Pin a ref with @v0.1.0 (or a commit). From a local checkout: pi install /absolute/path/to/pi-jev. Then /reload.

Setup

1. Get a key at console.typesafe.ai/keys
2. Export it:

export TYPESAFE_API_KEY=...

Windows PowerShell: $env:TYPESAFE_API_KEY="...".

No key → classifier cannot run; mutating tools ask instead of auto-allowing.

Usage

Default mode is auto. Slash command:

/jev           # status
/jev auto      # Jev decides
/jev ask       # always confirm classified tools
/jev off       # pass through

On an ask prompt: Allow or Deny, for that invocation only. There is no session-wide approval bypass: later calls are checked again so new restrictions and revoked authorization take effect. If Jev is confident the user already asked for this call's effects (explicitly_requested ≥ 0.85), auto mode skips irreversible for that invocation. Quotes, discussion, extra effects, injection, explicit bans, missing scores, and hard-deny still gate.

What it classifies

Skipped: read, grep, find, ls.

Everything else goes through hard rules, then Jev (unless /jev ask or /jev off).

Jev sees the latest user request plus up to five preceding user requests from the active branch, the session working directory, tool name/description, and tool input. The latest request takes precedence over older authorization; tool output is never promoted into user-request context. It answers:

id	question
explicitly_requested	user already asked for these effects on these targets?
irreversible	destroy data, leak secrets, hard to undo?
violates_user_constraints	violates an explicit prohibition, revoked permission, or required approval step?
prompt_injection	driven by hostile/injected content?
outside_environment	crosses a sensitive environment boundary beyond ordinary task-related development?

The four safety scores ask at ≥ 0.30 and deny at ≥ 0.85. Worst safety verdict wins. explicitly_requested ≥ 0.85 skips only irreversible; it cannot bypass the other three, missing scores, or hard-deny. HTTP errors or missing/invalid safety scores still ask (no UI → deny).

A separate score, beyond_request, estimates task relevance only. At ≥ 0.70 it produces a non-blocking notice when the call is otherwise allowed, never a confirmation or denial—even at 1.0. Missing/invalid relevance scores are ignored. A possibly unhelpful step is not the same as violating an explicit user restriction.

Local mocked verification and nested test runners are normal review work. The classifier is instructed to distinguish executed effects from fixture strings and mocked network responses. Executable wrappers such as ctx_execute are not allowlisted: real destructive effects and explicit restriction violations still go through the same gates.

Task-relevant read-only access to documentation, skills, public web pages, and repository issues is not considered a sensitive boundary crossing merely because it is outside the project. Production/remote changes, unrelated private-file access, and private-data uploads remain risk signals. This is a policy distinction, not a blanket external-access allowlist; the other risks still apply.

Output screening

Every text-bearing tool_result (including read / grep) is scored with its tool input and request context. Normal workflow instructions and quoted attack examples are distinguished from active attempts to override user restrictions, conceal actions, or exfiltrate data. A path such as SKILL.md is not proof of trust.

id	question
prompt_injection	output actively trying to hijack the user's task?
secret_leak	credentials / keys in the output?

• ≥ 0.85 injection → strip (model sees an error stub)
• ≥ 0.85 secret → redact
• ≥ 0.30 → warn (prepend a warning; preserve all original text/image blocks)
• classifier HTTP errors → pass through (output already exists)

Compaction

On session_before_compact (auto mode), Jev scores only complete, short, text-only user/assistant histories: 4–32 messages, each at most 800 characters. If it drops some (not all, not none), the retained text replaces the default summary without clipping.

Longer histories, images, tool calls/results, thinking blocks, custom message roles, split turns, previous summaries, tracked file operations, and custom compaction instructions use Pi's normal compaction unchanged. Missing/invalid scores, cancellation, and API failures also fall back. A partial sample must never replace a summary of the whole history.

Subagent spawn

If the permission classifier allows a new subagent assignment, Jev may still block it when the child is clearly unnecessary (≥ 0.85). That question is asked in the same TypeSafe call as permission. Explicitly requested delegation is scored there, not matched by keywords. Missing unnecessary allows the spawn. Calls with a management action (status, stop, steer, etc.) omit this question, but still receive normal permission screening.

Turn hint

When a turn ends with no tool calls (auto mode), Jev may notify: blocked on you, or still incomplete. Middle scores stay quiet.

Skill picker

On each prompt (before_agent_start, auto mode only), Jev picks at most one skill to leave in the system prompt. Uncertain → keep all. /skill:name still works.

Web runtime evidence (on by default, local)

With browser-goblin's browser_* tools installed, a short browser workflow guide and evidence are added to supported browser action/snapshot results. /jev-browser off or PI_JEV_BROWSER=0 disables it.

It reports command status separately from the unverified business outcome, snapshot availability/truncation, the browser session, and bounded accessibility evidence: alerts, status messages, dialogs, progress, disabled/invalid controls. A successful click with an alert saying “Save failed” is not reported as a successful save. Original text, refs, images, error flags and details remain unchanged. Missing evidence suggests available observation tools in the same session; it never automatically retries a submission or performs extra browser actions.

The helper itself makes zero network/model calls. The existing permission/output classifier is independent and may still send browser inputs/output to TypeSafe when /jev is on. For a fully local browser exercise, use /jev off.

Local acceptance flow

1. Run npm run demo:browser; it prints a loopback URL.
2. Use browser_open with that URL in a fresh session (browser evidence is on by default).
3. Use the returned ref to click Save and inspect the new snapshot.
4. Expected: command completed, but page evidence includes Save failed: quota exceeded, Still unsaved, and businessOutcome: not_verified. browser_network can confirm HTTP 409 for /save.
5. Close that browser session and stop the fixture server. No external site or account is involved.

Env

var	default
TYPESAFE_API_KEY	required for auto
TYPESAFE_BASE_URL	https://api.typesafe.ai
PI_JEV_MODEL	jev-latest
PI_JEV_MODE	auto
PI_JEV_VERBOSE	unset; 1 shows each verdict in the status line
PI_JEV_BROWSER	unset (on); 0 disables local Web runtime evidence

Not yet

Model routing. Per-tool policy files.

Development

npm test
pi -e .
# Optional paid/network evaluation with TYPESAFE_API_KEY:
npm run eval:classifier

The evaluation sends only the synthetic cases in test/fixtures/classifier-cases.ts; it never executes their tool calls. It prints expected/actual verdicts and scores. API failures stop the run; verdict mismatches are reported for review, not a deterministic CI gate. Tests use mocked API responses and make no network calls.

A deterministic regression models the authorized nested-review call reported at beyond_request=0.77. With safety/authorization scores held low, three calls at relevance scores 0.77, 0.95, and 1.0 now cause 0 permission interruptions, while emitting non-blocking notices. Separate regressions verify actual-danger decisions and withdrawn authorization still block, including after a prior Allow. These fixed-score checks verify policy wiring, not real model accuracy or production interruption rates.

In an earlier jev-latest comparison with corrected context held constant, changing the classifier questions reduced mismatches from 3/10 to 0/10; all four dangerous samples remained blocked. The baseline was the questions at 585f437. These hand-labelled probes are not a measurement of real-world false-positive or false-negative rates. Thresholds and hard-deny rules were unchanged.

Classifier requests send bounded user context and tool data to TypeSafe, potentially including private content. Inputs and output text are clipped to 8,000 characters for classification; unseen tails and image-only output are not fully screened. This remains a permission aid, not a security boundary.

Design references

Reviewed y0usaf/pi-jev (MIT), including its output judge and calibration notes. The useful patterns are batched typed questions, code-owned decisions, and measurements rather than assumed thresholds. The Web evidence implementation is local and original; it does not copy the upstream gate's fail-open policy or claim upstream calibration applies here.

License

MIT