Reference

Detail that is useful when something needs explaining, kept out of the README so the README stays readable: run statuses and error classification, the boundary between the decision loop and the surface it drives, and what changed from the Cline plugin this project was ported from.

• What the browser side provides
• Run results and statuses
• What the loop remembers, and what it refuses to act on
• Working with the manual tools
• Profiles
• Where the loop ends and the surface begins
• Differences from the Cline plugin

What the browser side provides

Beyond the decision loop, the browser tools come with:

• screenshots returned as image tool results, so the agent can verify outcomes
• batched click, double_click, scroll, type, wait, keypress, drag, move, navigate, back, forward, reload and screenshot actions
• browser console, page-error, failed-request, navigation, download and security logs
• per-run PNG artifacts and optional WebM video recording with a visible agent cursor and animated click pulses
• a tokenized live screenshot/log viewer bound to 127.0.0.1
• session isolation, blocked downloads, blocked service workers, no inherited host environment, disabled extensions and browser file-system access
• bounded browser launch, navigation, video-finalization and cleanup timeouts, so an unavailable app fails with its underlying error instead of hanging
• automatic visible-browser fallback on macOS when headless Chromium cannot start
• a live footer status line while a browser run streams decisions
• prompt guidelines covering prompt injection, sensitive data and consequential actions

Run results and statuses

A jev_run result carries status, steps, elapsedMs and a JSONL tracePath. Statuses are done_unverified, blocked, needs_review, uncertain, step_limit, evaluation_limit or interrupted. A failed run also reports failure (stage, category, detail) and an errorsLogPath.

failure.detail is a bounded summary (error name, HTTP status when known), and errors.log in the run directory holds the full error including a stack. Provider errors can quote the outgoing request body, so the full text is written only to that local file and never returned to the model. An attempted action in a failed run might already have taken effect: inspect before continuing.

failure.category

failure.category	Meaning
configuration	Missing or unreadable configuration; the message names the file and the variable to set. Reported before Chromium starts.
text_helper_invalid_output	The active pi model did not return a usable {"text": ...} value. Nothing was typed.
cancelled	The run was aborted or hit its deadline.
navigation_context	The document changed while it was being read.
timeout	A bounded browser or model timeout elapsed.
document_not_ready	The page never produced a readable document.
unexpected_error	Anything else, usually a provider error.

stopReason

stopReason explains why the loop stopped, independent of the model's status.

stopReason	Meaning
model_done	Jev reported DONE; the status is done_unverified because a claim is not proof.
model_blocked	Jev reported BLOCKED: no supported action can progress.
model_review	Jev reported REVIEW: the next step needs sensitive data, submits something, or crosses a safety barrier such as a CAPTCHA.
submit_review	The loop itself refused to press Return in a field that has no confirm control of its own (a chat composer, a command line): that sends or executes what was typed. Status is needs_review; the user sends.
verification_gate	The loop itself refused a human-verification gate before asking Jev: the page text announced a challenge and a control offered to pass it. Status is needs_review; hand the step to the user.
min_probability	The selected choice fell below the requested minProbability.
step_limit / evaluation_limit	The action or evaluation budget ran out.
repeated_action	An identical action stopped producing new state (a control cycling between states it already produced), or ran 12 times in a row as a backstop. Repeated presses that keep producing new state are allowed, because entering 111 is legitimate input.
scroll_oscillation	SCROLL_UP and SCROLL_DOWN alternated repeatedly, a two-cycle that is not exploration.
stale_observations	Four consecutive observations were invalidated before an action could run.
no_progress	Three actions produced no observable change. A control pressed twice without any change is withdrawn from the next question first, so the count starts again when the question changes; the run only stops when Jev keeps choosing fresh controls that change nothing.
text_unavailable	Neither the goal nor the pi text helper supplied a value for the field, so nothing was typed.
cancelled / error	The run was aborted, or it failed; see failure.

Returned page evidence

A jev_run result carries the page state captured when the loop stopped: the JSON summary (status, stopReason, failure, steps, tracePath, errorsLogPath, finalPageUrl, finalPageTitle), a readable block with the final URL, title and visible page text, and the before/after screenshots. The text block is what a model without image input uses to verify the outcome, so the loop returns it instead of forcing a screenshot read. The full record of every run also lands in the trace's final result entry.

Traces record decisions (including terminal and rejected decisions), stale observations, action attempts, completion and the final result. Provider confidence is recorded separately when supplied. Traces omit generated field text but may contain page labels.

What the loop remembers, and what it refuses to act on

The loop retains the last ten actions, entered text, and observed progress across runs of the same goal in one browser session. Text stays in memory and is not written to traces. Stale decisions are re-evaluated within a budget of twice maxSteps; executed mutations are never retried.

It retains observed DOM nodes and checks page semantics, node identity and occlusion before acting. Open shadow roots are walked, so a web component's controls and text are observed and acted on like light DOM: hit-testing descends into the shadow tree, and a shadow-rendered overlay that covers a target is named as the cover. Closed shadow roots, frames, canvas controls, nested scrolling, uploads and arbitrary keyboard widgets are outside this DOM loop; use jev_actions where appropriate.

Model context is capped at 200 action targets and 6,000 visible text characters, plus 50 selected options and up to 50 offscreen control labels in each direction. This can omit controls on dense pages.

Page text and visible field values are sent to TypeSafe api.typesafe.ai, and the same content is sent to your pi model provider when a field value has to be generated because the goal does not contain it. Password and file fields are excluded, but other sensitive content is not automatically redacted. Jev is instructed to return REVIEW before consequential actions; that is model guidance, not a deterministic security boundary.

Working with the manual tools

Element targets instead of coordinates

A manual click, fill or select can address an element instead of a pixel, so the same call keeps working after a layout change:

[
  { "type": "click", "target": { "role": "link", "name": "部落格" } },
  { "type": "fill", "target": { "role": "textbox", "name": "Email" }, "value": "a@b.c" },
  { "type": "select", "target": { "role": "combobox", "name": "Plan" }, "value": "pro" },
  { "type": "click", "target": { "selector": "#submit" } },
  { "type": "click", "target": { "role": "button", "name": "Buy", "nth": 1 } }
]

A target accepts role, name (needs role), text, selector and nth. Matching waits up to five seconds for the element to exist, so a batch can click a link and act on the page it navigates to. A target that matches nothing fails with No element matched … instead of clicking whatever happens to be under the cursor. x/y coordinates remain available for canvas-like surfaces.

A coordinate click is a raw event, so the browser will route it into an iframe even though the automatic loop never sees frame content. When a click or drag lands on a frame, the result carries a warnings entry and a matching security entry appears in jev_logs, naming the frame origin and flagging known anti-bot providers (reCAPTCHA, hCaptcha, Cloudflare challenge):

A click at (160, 165) landed inside a frame from https://www.google.com [reCAPTCHA].
The automatic Jev loop never interacts with frame content; this was a raw coordinate click.

This is a record, not a block: it makes coordinate clicks auditable and tells the agent to stop clicking there. Keep it in mind when deciding whether to auto-approve browser tools.

Tabs

Sites open new tabs constantly (target="_blank" on login and outbound links). Pi Jev Browser never switches the observed page silently:

• popups: "stay" (default) keeps the run on the original page, records a tab entry in jev_logs, and returns a warning naming the new URL.
• popups: "follow" adopts the new tab as the observed page, and says so.
• jev_actions can then move deliberately with activate_tab / close_tab, and jev_state.activePageIndex reports which tab is being observed.

If the observed tab closes, the run falls back to another open tab instead of keeping a reference to a closed page.

Profiles

profile decides how much browser state survives between runs:

• "session" (default) — one profile per pi session, under ~/.pi/agent/pi-jev-browser/profiles/<session id>. Log in once in the visible window and later runs in the same pi session reuse the cookies. Two pi sessions never fight over one Chrome profile.
• "shared" — one profile for every session, at profileDir. Convenient, but two concurrent pi sessions cannot use it at the same time.
• "off" — a clean browser every run, the pre-profile behavior.

Only cookies with an expiry persist; Chromium drops session cookies on exit by design. A persistent profile also keeps localStorage and IndexedDB, so point profileDir at something you are comfortable reusing.

Where the loop ends and the surface begins

The decision loop implements the part that turned out to be hard: bounded steps, stale-observation handling, an oscillation guard, a no-progress guard, traces, and handing control back to the human before a consequential action. None of that is web-specific, so src/loop.ts does not import Playwright. It drives a Driver (src/driver.ts):

interface Driver {
  /** Identity of the surface; a change means the run moved and must not act. */
  id(): unknown;
  observe(signal?: AbortSignal): Promise<ObservationSnapshot>;
  /** The driver's own vocabulary for read failures: a navigation context, a closed window. */
  readFailureCategory?(
    error: unknown,
  ): "navigation_context" | "document_not_ready" | "window_unavailable" | undefined;
}

Observation is the contract the decision layer reads: text plus addressable targets (role, label, value, href, and state), never pixels. A snapshot is bound to the state it was read from, so assertFresh and execute both refuse to act once that state has moved — which is what keeps an action from landing on whatever replaced the target in the meantime.

test/driver.test.ts proves the boundary is real: it drives the loop with an in-memory driver that has no Playwright, no DOM and no browser, and still exercises done_unverified, model_review, model_blocked, text_unavailable, repeated_action, no_progress, scroll_oscillation, stale_observations and step_limit. Driving a different surface means writing another driver, not rewriting the loop.

The browser driver

browserDriver(getPage) in src/observe.ts — everything Playwright-specific lives there, including the error vocabulary: a destroyed execution context is a navigation in progress, and a document that never became readable is a browser condition, not a generic failure.

The desktop driver

src/drivers/desktop.ts drives a real macOS application through its accessibility tree, with desktop/ax-helper.swift as the resident helper that walks the tree and performs accessibility actions. It maps onto the same Observation, so the loop is untouched. Hard-won details are encoded there:

• Applications are addressed by bundle id because display names are localised, and the accessible name lives in AXDescription while the stable handle is AXIdentifier. Calculator publishes nothing in AXTitle, and its multiply button is labelled in the system language but identified as Multiply.
• Titlebar close/minimize/zoom buttons are excluded by subrole, since pressing close terminates an application that quits with its last window.
• The surface identity is the driven application's process, not the frontmost application: accessibility actions land without focus, so a person switching windows mid-run changes nothing the loop acts on, while a quit or relaunch still stops it.
• Editable text (AXTextArea, AXTextField) carries no press action and is offered as a TYPE_TEXT target driven by setting its value; its content joins the window text, because for an editor the document is the state.
• An application that draws its own document (Word's page is one AXLayoutArea with no value, no actions and no children) is offered as a text target, read with text recognition, and typed into with keyboard events after a click places the caret.
• AXError -25204 (cannot complete) and -25205 (invalid element) are the application not answering in time and the control dying as a result of the press, not a refusal, when the window has changed since the observation.

Desktop driving is covered by the desktop benchmark tier on two applications: Calculator, verified by an arithmetic result computed outside the application, and TextEdit, verified by reading the document back from the application.

Planning

createJevPolicy({ planning: true }) adds a planning phase that runs once, against the initial observation: the same choice model picks the next step of a plan (or PLAN_COMPLETE) until the plan is finished, and the run then executes the user's goal plus the enumerated steps. This closed a measured gap on the desktop tier, where a short goal ("compute 1234 times 5678") reached one correct press without a plan and ten of ten with one. It is opt-in because a plan made from one observation only covers what that observation shows, which suits an application window and misleads on a multi-page web task. The plan is written to the trace as a plan step and returned in the run result.

Differences from the Cline plugin

The observation pipeline, the Jev prompt, the run statuses and the safety contract were carried over from the Cline plugin; the host plumbing, the decision transport and the text helper changed, and everything from the Driver interface onward (the desktop driver, the planning phase, the verification gate, shadow DOM, the benchmark suite) was added here.

Area	Cline plugin	Pi Jev Browser
Host API	plugin.setup(api) with JSON Schema inputSchema	export default (pi) with TypeBox parameters
Safety rules	api.registerRule()	promptSnippet + promptGuidelines (verified in the built system prompt)
Decision transport	@ai-sdk/gateway + experimental_evaluate	direct POST https://api.typesafe.ai/v1/systemone
Decision model	typesafe-ai/jev via AI Gateway	jev-latest (configurable) direct
Text helper	google/gemini-2.5-flash-lite via Gateway	active pi model via ctx.modelRegistry.complete()
Credential	AI_GATEWAY_API_KEY	TYPESAFE_API_KEY
Config	~/.cline/plugins/cline-jev-browser.config.json	~/.pi/agent/pi-jev-browser.config.json
Artifacts	~/.cline/data/jev-browser/	~/.pi/agent/pi-jev-browser/
Session identity	context.sessionId over JSON IPC	ctx.sessionManager.getSessionId()
Cancellation	local controller only (signals were not transported)	pi AbortSignal + local controller
Failure reporting	one opaque interrupted message	classified failure + errors.log with the full provider error
Setup	kicked off eagerly when the plugin loaded	lazily on first browser tool use
Tool names	jev_run, jev_actions, jev_state, jev_logs, jev_stream, jev_stop	the same names, plus jev_extract for deterministic reads
Tool results	{ result: [...] } interpreted by Cline	{ content, details, usage }; nested model usage is reported to pi
Schemas	JSON Schema with additionalProperties: false	TypeBox with additionalProperties: false, plus per-action field validation
Concurrency	host-defined	executionMode: "sequential" for all browser tools
Cleanup	plugin teardown	session_shutdown handler calls stopAll()