computer-use-jev

Drive macOS applications from Go — with Jev (TypeSafe's System One decision model) as the decision maker.

Extracted from herbie's computer_use tool and extended with a typesafe decision layer.

How it works

• computeruse — operates native macOS applications through the Accessibility API via a persistent Swift worker (computeruse/swift/). Actions: apps, windows, activate, snapshot, click, fill, type, press, screenshot. Tokens (a1, w2, e5) reference apps, windows, and elements discovered at runtime.
• typesafe — a minimal client for the TypeSafe System One API (POST /v1/systemone, model jev-latest).
• decide — the Jev decision loop. Given a natural-language goal, each step gathers real state (app list, windows, last snapshot) and asks Jev one batch of typed questions: which action next, which target token, whether the goal is satisfied, whether text is needed. Jev's answers are typed — code, not the model, maps them to a tool call. If the action confidence drops below a threshold, the run stops and reports the distribution rather than acting blindly. Closed sets stay closed: targets are chosen only from tokens actually present in the current snapshot.

Install

go build ./cmd/computer-use-jev

macOS 14+. Requires the Accessibility permission (and Screen Recording for screenshots) granted to the running terminal, plus Xcode Command Line Tools to compile the Swift worker (built once, then cached).

Use

Let Jev drive toward a goal (needs TYPESAFE_API_KEY):

computer-use-jev -goal 'in TextEdit, type "hello world"'
computer-use-jev -goal 'open Safari and take a screenshot' -json
computer-use-jev -goal '...' -dry-run        # decide, don't act
computer-use-jev -goal '...' -max-steps 24

Or call the tool directly, no Jev, for scripting:

computer-use-jev apps
computer-use-jev windows -app a1
computer-use-jev snapshot -window w1
computer-use-jev fill -target e3 -text "hi"

Development

make check    # build, vet, test
make smoke    # live run against Jev + real Accessibility

Tests are hermetic: a fake worker binary stands in for the Swift worker, and an httptest.Server stands in for the TypeSafe API.

License

MIT

Compound goals

Before desktop execution, Jev judges whether a goal contains multiple requested instructions. A single instruction passes through unchanged and needs no LLM configuration. Compound goals are split once by an LLM into an ordered JSON sequence; the LLM has no desktop tools. Commands then run through Jev in the same worker session, with fresh observations and separate completion state.

export COMPUTER_USE_JEV_PROVIDER=openai
export COMPUTER_USE_JEV_MODEL="your-model"
# Set OPENAI_API_KEY and TYPESAFE_API_KEY in your environment.
make build
./build/computer-use-jev -goal "open textedit select the text and make it bold"

The splitting provider supports:

COMPUTER_USE_JEV_PROVIDER	Credential	Protocol
openai	OPENAI_API_KEY	OpenAI Responses
anthropic	ANTHROPIC_API_KEY	Anthropic Messages
openrouter	OPENROUTER_API_KEY	OpenAI Chat Completions

COMPUTER_USE_JEV_MODEL must name a model available through that provider. No provider/model is silently chosen, and credentials are never read from Herbie's personal configuration. For openai, set OPENAI_BASE_URL to override https://api.openai.com/v1; an empty value uses the default. The endpoint must support the OpenAI Responses protocol. This override does not affect other providers. The provider-neutral interface and wire implementations are copied from Herbie under internal/herbie/; see its README for provenance.

• -dry-run classifies and prints the plan without inspecting or changing the desktop. It may call the splitting LLM and therefore incur API charges.
• -json emits a plan record followed by step records with an instruction number. Screenshot diagnostics go to stderr.
• -max-steps is a total budget across all instructions. Later instructions do not run after an earlier instruction fails or exhausts that budget.
• Jev can choose select-all, bold, italic, underline, copy, paste, save, undo, or redo from a closed shortcut set. Explicit shortcuts (e.g. cmd+b) take precedence; unknown or low-confidence semantic choices are not sent.
• Text remains literal, not generated by Jev. Prefer double quotes around the exact text within the goal, using single quotes for the shell argument.
• Uncertain classification and malformed/oversized LLM splits stop before any desktop actions. A valid split is not proof that its semantics are correct; inspect the plan with -dry-run before using consequential goals.

"Open" currently means activate an already running application; this does not add an application-launch action. TextEdit must already be running.