Computer-use architecture

August 25, 2026 · View on GitHub

dsh-computer-use is the policy and safety layer around several computer-use backends. It should expose one predictable control loop to an agent even when the active model can inspect images and when it cannot:

AX / DOM observation
        │ target unique?
        ├── yes ────────────────► semantic target
        │                              │
        └── no / incomplete ───► measured vision route
                                  ├── native model vision
                                  └── dsh-vision-router sidecar
                                                │
                                      one primary actuator
                                                │
                                      AX verify → pixel verify

The arrows are an ordering guarantee. A model capability result selects the perception route only; it must not cause the executor to switch between several competing mouse/keyboard controllers during one task.

1. Observe AX/DOM first

Observation begins with the target app's Accessibility tree or DOM. This path is preferred because it avoids a screenshot, is usually cheaper, and gives a stable semantic locator and state that can be checked after an action.

An observation records an opaque observation_id and a monotonically useful version (or equivalent freshness token), the target app/window, the AX/DOM completeness result, and whether the requested target is unique. An incomplete tree is not a successful target match. An action must reject an observation whose version is stale instead of guessing from its old coordinates.

When AX/DOM uniquely identifies the target, the router does not ask the model to interpret an image. It returns the semantic target to the single actuator adapter.

2. Select vision from measured capability

If AX/DOM is missing or ambiguous, the router checks a capability record for the exact active deployment identity (provider, endpoint, model/deployment name, and relevant image-input mode). A model-name allow-list is not sufficient: providers can change an endpoint's behavior without changing its marketing name.

The capability probe is small, explicit, and cached with expiry:

  • native-vision-grounding: the active model accepted an image and produced a usable target/grounding result;
  • native-vision-describe-only: the model accepted an image but did not meet the grounding contract;
  • image-rejected: the endpoint rejected image input;
  • unknown: no authorized probe has run or the cache is expired.

Background probes are off by default. A probe involving a paid endpoint needs explicit authorization. When the result is unknown, the safe route is the sidecar; the router must not infer support from a model string.

Native measured vision

For native-vision-grounding, pass the current screenshot (or a minimally cropped image) to the active model and require a structured target with a confidence/provenance result. The image is tied to the same observation token; coordinates from an older screenshot cannot be acted on silently.

Sidecar vision

For rejected, unknown, or describe-only native vision, call the dsh-vision-router sidecar. The sidecar owns pixel grounding and optional crop/description/diff operations, with bounded timeout, rate limit, and a safe unavailable result. It is a perception adapter, not a second actuator.

No route should proceed on a guessed coordinate after a vision timeout or an ambiguous result. The caller receives an actionable failure and can re-observe or ask for clarification.

3. One primary actuator

All click, type, and drag requests go through one primary actuator, currently the open-computer-use soft-cursor path in the intended design. Its independent cursor gives the user visibility without moving the user's real pointer. The router is responsible for preserving the selected observation token and for holding the GUI serial lock while the action is in flight.

Other GUI MCPs may remain installed as explicit adapters for compatibility or a user-requested physical-pointer handoff. They are not equivalent defaults and must not be exposed in a way that lets an agent bypass stale-observation checks, confirmation policy, locking, or verification.

The lock is runtime state, not a shell convention. It has an owner, an expiry, and a stale-owner recovery path. A process crash must not leave a permanent lock, and a live owner must not be stolen merely because another agent is waiting.

4. Verify after every mutation

The loop is observe → route → act → verify:

  1. Read the current AX/DOM or screenshot state.
  2. Select native measured vision or sidecar only if AX/DOM cannot identify the target.
  3. Dispatch exactly one action through the primary actuator.
  4. Re-read AX/DOM and confirm the expected state transition.
  5. Use a before/after screenshot diff only when semantic verification is not available. If there is no expected delta, stop and report failure; do not assume that a click landed.

Destructive, payment, credential, or externally visible actions also pass the upstream confirmation policy. On-screen text is untrusted content and cannot override the agent's task, confirmation requirement, or lock policy.

Normalized behavior-level contract

The intended runtime surface has three operations. They are described by behavior here so that the DSH host can implement them as native tools or as a thin adapter over its MCP tools; the names are not evidence that every host profile already advertises them in tools/list.

computer_observe

Returns the latest AX/DOM summary, an observation freshness token, the selected vision route (semantically semantic, native, sidecar, or none; an implementation may label these ax, native-vision, and sidecar-vision), and the reason for that route. A screenshot is returned or referenced only when the selected perception route needs it. The result must not expose raw secrets by default.

computer_act

Accepts an observation token and one click/type/drag/keyboard action. It rejects stale or ambiguous observations before dispatch, obtains the GUI lock, applies confirmation policy, calls the one primary actuator, and returns verification evidence. It releases or renews the lock according to the runtime lease.

computer_health

Reports actuator readiness, AX and screen-capture permission state, native vision probe evidence, sidecar availability/latency, lock state, and the current degradation route. It is diagnostic and must not perform a desktop mutation or a background paid-model probe.

Failure and privacy boundaries

  • A missing AX tree selects measured vision; it never authorizes a blind click.
  • Unknown native image capability selects the sidecar or a clear blocked result; it never selects a model by name alone.
  • A sidecar timeout, stale observation, lost lock, or failed verification is a hard failure for that action.
  • Screenshots are minimized, cropped, redacted where possible, and not logged or uploaded by default. Vision providers receive only the image needed for the current observation.
  • Permission errors are surfaced as health/degradation evidence. No backend bypasses macOS Accessibility or Screen Recording prompts.

Implementation status

The bundle now mounts the runtime and Vision Router explicitly. The runtime registers the three normalized operations and keeps the actuator's raw MCP surface private. Static preflight executes plugin registration against an isolated registry; live preflight additionally installs an isolated DSH profile, expands its configuration, and checks the real actuator tools/list without observing or mutating the desktop.