Foreground-safe input policy

Requirement

DSH Computer Use must let an Agent operate a native macOS application while the user continues working elsewhere. The default pointer path must not move the system cursor, post pointer events to the global HID stream, or activate the target application merely to make a pointer action work. Keyboard input is the one deliberate exception: the Bundle enables keyboardPolicy: activate so typing works reliably by bringing the target app to the foreground first, matching Codex Computer Use behavior. Pointer actions may show a separate Agent cursor, but that overlay must never become an input source or a window manager participant.

The default Bundle configuration is:

interaction:
  focusPolicy: preserve
  keyboardPolicy: activate
  pointerInputPolicy: targeted
  cursorVisualization: visible
  cursorSpeedPxPerSecond: 1600
  cursorAccelerationPxPerSecondSquared: 6000
  cursorClickDelayMs: 90
  cursorAutoHideMs: 0

preserve means pointer actions do not request foreground activation. keyboardPolicy: activate (the Bundle default) requests foreground activation before type-text keyboard fallback and press-key; keyboardPolicy: preserve keeps typing routed to the selected pid without activation and is reliable only for applications that accept background keyboard events. targeted means mouse, drag, and wheel events may be sent only to the exact observed process and window. visible enables the separate Agent cursor only when that target application is frontmost. Cursor speed, acceleration, post-arrival click delay, and auto-hide timing are host-owned. The deprecated 0.2.x cursorMotionMs field is accepted during upgrade but ignored. The model cannot change any interaction policy through Tool arguments.

This is an input-routing property, not a consequence of Accessibility permission alone. Accessibility grants semantic UI access; foreground preservation comes from choosing semantic Accessibility operations first and using process/window-targeted fallback instead of the system cursor.

Overall design

The action path has five ordered layers:

1. The DSH Service binds the request to one unexpired observation, exact bundle id, pid, window, and element target handle, compatibility index, window-relative point, or screen-global point.
2. For a handle action, the Service obtains fresh Accessibility state and resolves the target through the original locator, one unique provider-native identifier, or one unique semantic role/name/actions/ancestor match. Application, process, and selected-window identity remain exact; ambiguity and low confidence fail closed.
3. The native helper observes the resolved target again and rejects any race that changed its locator, properties, process, or window before input.
4. The helper prefers AXPress, Accessibility value assignment, selected-text assignment, or an action advertised by the element. When a click target advertises AXPress but macOS rejects the press, the helper retries AXPress on the element's pressable descendants within a bounded depth before considering the element-frame or coordinate pointer route.
5. When semantic input is unavailable, keyboard events are posted to the selected pid and pointer events are posted to the selected pid and window. No pointer fallback uses a global event tap.

Pointer delivery uses the observed window when it contains the point; otherwise it resolves the topmost on-screen window of the selected app that contains the screen point and posts through SLEventPostToPid with the pid/window fields and window-local coordinates. This is the same target-process shape Codex Computer Use uses (SynthesizedEvent.send(to: pid) with CGWindow.window(at:)), so coordinateSpace: screen supports arbitrary-coordinate clicks without a global HID event stream. Unavailable SkyLight symbols or a point outside every window of the selected app fails closed.

Visual feedback uses a 28x28 NSPanel owned by a dedicated, persistent cursor process. The panel is borderless, nonactivating, click-through, and excluded from normal window cycling; it is not made visible on every Space or over a background target application. It draws the embedded transparent whole-image cursor (assets/cursor.png, Cursor arrow plus DeepSeek whale) with its top-left corner at the target point. The first visible move starts at the system cursor's current location without moving that cursor. Later moves start at the prior Agent-cursor point. Distance, requested maximum speed, and acceleration determine a travel time bounded to 48 through 2000 ms; the safety bound takes priority over exact physical units at extreme combinations. Symmetric acceleration/deceleration and a small deterministic Bezier bend produce a legible path. The native move command replies only after arrival, then the provider applies cursorClickDelayMs before press feedback. Click input follows that sequence. Drag reaches and presses at the start point first. The native helper then validates the target and waits at a start barrier until the endpoint cursor command has been written; Agent-cursor travel and targeted native drag follow the same bounded Quartz path. Once native mouse-down is possible, caller cancellation waits for the bounded mouse-up instead of terminating the helper mid-gesture. Release validation follows both. The overlay never posts input and never changes the system cursor position.

Policy is enforced twice on the supported DSH Tool path. The Service rejects a known pointer or foreground requirement before obtaining a control lease or consuming a sensitive-action confirmation. The helper validates the same resolved policy immediately before input, including fallback decisions that can only be made at runtime. The helper also requires an isolated process group plus three standard pipe or Unix-socket transports whose peer endpoints belong to its direct parent process; ordinary shell redirection fails closed before any command is parsed. This transport check is defense in depth, not authentication against arbitrary code running as the same macOS user: a deliberately constructed detached parent can reproduce the topology, especially under danger-full-access. The registered Tool path remains the only supported route because it applies leases, confirmations, and host policy before invoking the helper.

Every action result reports the route actually used:

activation: 'not-requested' | 'already-frontmost' | 'activated'
pointerInput: boolean
pointerRouting: 'none' | 'target-process'
resolution?: {
  mode: 'exact-locator' | 'native-identifier' | 'semantic-rebind'
  confidence: number
  candidateCount: number
  targetChanged: boolean
}

These fields do not claim that a target application can never change focus as its own side effect. They report only what the helper requested and emitted.

Action matrix

Action path	Default activation	Pointer route	Default result
click through AXPress	None	None	Allowed
set-value through Accessibility	None	None	Allowed
non-foreground perform-action advertised by the element	None	None	Allowed
AXRaise	Denied	None	Requires explicit focusPolicy: activate, then re-observation/revalidation
type-text through selected-text assignment	None	None	Allowed when the focused element accepts it
type-text keyboard fallback	None with keyboardPolicy: preserve; activated with keyboardPolicy: activate	Target pid	Allowed; activation makes it reliable
press-key	None with keyboardPolicy: preserve; activated with keyboardPolicy: activate	Target pid	Allowed; activation makes it reliable
coordinate click or element-frame fallback	None	Target pid + window	Allowed when pointerInputPolicy: targeted
scroll	None	Target pid + window	Allowed when pointerInputPolicy: targeted
drag	None	Target pid + window	Allowed when pointerInputPolicy: targeted

pointerInputPolicy: deny disables coordinate click/fallback, scroll, and drag while leaving semantic Accessibility and process-targeted keyboard paths available. coordinateSpace: window (default) resolves coordinates inside the observed window frame; coordinateSpace: screen accepts Quartz screen-global points.

Critical decisions

Host policy is not a Tool argument

The deployment owns focusPolicy, keyboardPolicy, and pointerInputPolicy. allowCoordinateFallback says only that computer_click may try the host-authorized pointer route after AXPress is unavailable. It cannot enable pointer delivery or foreground activation. computer_perform_action also treats AXRaise as foreground-affecting and rejects it under preserve.

Accessibility remains the primary route

Semantic Accessibility operations are more stable than pixels and need no cursor emulation. They also work against many background applications. The helper revalidates the exact target before invoking them and reports activation: not-requested, pointerInput: false, and pointerRouting: none.

Stable handles rebind only with independent identity evidence

The public targetHandle is opaque and observation-local; it does not expose an AXUIElement, locator, or AXIdentifier. The Service stores the normalized descriptor in Agent-owned memory and re-observes immediately before a handle action. Exact locator identity is preferred. Rebinding is opt-in through allowRebind and requires either one provider-native identifier whose stable semantics still match or one semantic candidate with the same role, normalized accessible name, advertised actions, and stable ancestor fingerprint. The resolver preserves the exact bundle id, pid, and selected-window identity. A truncated tree cannot authorize rebinding. Duplicate candidates return COMPUTER_TARGET_AMBIGUOUS; missing or insufficient evidence returns COMPUTER_TARGET_LOW_CONFIDENCE.

Sensitive confirmation is bound to the exact handle action. Any native-identifier or semantic fallback marks the target changed, invalidates the prior token, and returns COMPUTER_TARGET_REBIND_REQUIRES_CONFIRMATION before input. Coordinates and screenshot-derived boxes cannot become handles, identity evidence, or authorization for a sensitive target. The caller must select the current handle and obtain a new one-use confirmation. Provider-native visual hit-testing remains a separate follow-up because the current visual workflow cannot independently validate a screenshot coordinate.

The default pointer route is virtual and target-specific

The helper never moves the system cursor and then tries to restore it. That design would still interrupt the user, race with real input, and risk delivering an event to the wrong application.

Instead, pointer fallback creates an event at the target screen point, binds it to the exact pid and CGWindowID (the observed window when it contains the point, otherwise the topmost app window under the point), supplies the window-local point expected by AppKit, and sends it through the per-process SkyLight route. Click, scroll, and drag share this route. The committed helper contains no CGWarpMouseCursorPosition, global CGEventPost, or .post(tap: .cghidEventTap) path.

Cursor visualization is presentation-only

The visible Agent cursor is deliberately not the input source. It is a separate process with a strict JSON-lines protocol and a startup-ready handshake. Every show, press, and release is bound to the observed pid, CGWindowID, and expected frame; a closed, moved, resized, minimized, or off-screen window hides the overlay. Because presentation is separate from routing, disabling the cursor cannot change action semantics and overlay failure cannot redirect or globally emit input.

Activation is an explicit compatibility mode

Some applications accept input only while active. A deployment may set focusPolicy: activate (all actions) or keyboardPolicy: activate (keyboard actions only), accepting that the target application can take the foreground. Before emitting input, the helper activates the exact process and observes it again. Element and window targets are revalidated; for keyboard actions the refreshed focused element is the target, because activation may move focus to the app's default control. Any element/window mismatch fails with COMPUTER_STALE_OBSERVATION instead of acting on the pre-activation target.

With focusPolicy: preserve and keyboardPolicy: preserve, the helper never performs this activation step.

Pointer delivery fails closed

Target-process pointer delivery resolves the window from the point itself: the observed window is used when its frame contains the point; otherwise the helper takes the topmost on-screen window of the selected app whose frame contains the point. Observation still captures AXWindowNumber when available and falls back to a unique frame/title match, but coordinate actions no longer require that match. A point outside every on-screen window of the selected app fails closed instead of falling back to the global cursor.

Private SPI is isolated and optional at runtime

The per-process pointer route uses dynamically resolved SkyLight symbols. This keeps the failure explicit on an unsupported macOS build: semantic Accessibility and process-targeted keyboard input remain available, while pointer fallback returns COMPUTER_ACTION_BLOCKED. The helper never silently changes to global pointer injection.

Verified evidence

The release evidence covers both implementation and observed behavior:

• source and binary checks reject system-cursor warp symbols, the exact global CGEventPost symbol, and unknown dynamically resolved native symbols;
• overlay checks require a nonactivating panel, click-through hit testing, prohibited application activation policy, and no cursor-warp primitive;
• the overlay runtime rejects missing or malformed target identity, oversized or invalid JSON-lines commands, unsupported timing, and direct helpers that do not own a managed parent transport;
• a real overlay process must emit its ready frame before commands, reuse one process across commands, and stop cleanly on disposal;
• a native monitor requires the overlay to be the only 28x28 window owned by its process, not become frontmost, and produce no global pointer events from that process pid;
• the helper must contain SLEventPostToPid and CGEventSetWindowLocation;
• a coordinate click without an observed window id resolves the app window under the Quartz screen point and still reports pointerRouting: target-process;
• keyboardPolicy: activate makes a background fixture become active and receive the key event, and the fixture transcript records the activation;
• the fixture is started through open -g with --background, so LaunchServices does not request foreground activation;
• the fixture records every applicationDidBecomeActive callback and the default path must not increase activationCount;
• an independent native monitor samples cursor position and the frontmost pid every millisecond throughout click, scroll, and drag; every sample must remain unchanged;
• background AXPress, Accessibility value/action, selected-text input, and pid-targeted key input change the fixture without activating it;
• the native fixture inserts a harmless sibling and recreates a uniquely identified checkbox, proving the raw locator becomes stale while AXIdentifier resolution still finds exactly one target;
• target-process click and scroll are each observed exactly once; drag has exactly one down/up gesture; the target remains non-frontmost;
• pointerInputPolicy: deny rejects click fallback, scroll, and drag before any target pointer event is delivered;
• clean Profile and real-model validation require the model-visible action result and fixture transcript to agree.

Known limitations

• Target-process pointer delivery is less universal than semantic Accessibility. Custom canvases, games, hardened input surfaces, or future macOS changes may reject it.
• The clicked point must fall inside an on-screen window of the selected app; minimized, fully hidden, or windowless targets fail closed.
• focusPolicy: activate and keyboardPolicy: activate are intentionally disruptive and exist only as operator-selected compatibility modes.
• A target application may change its own activation or focus as a side effect of an accepted action; the helper does not claim control over application-internal behavior.
• The Agent cursor is scoped to the current frontmost application, one active Space, and the exact observed window. cursorAutoHideMs: 0 keeps it visible only while all three remain valid; a foreground change hides it within the periodic target check.
• Stable handles currently use exact locators, provider-native identifiers, and strict semantic identity. Semantic-spatial rebinding and provider-native visual hit-testing remain follow-up work; a vision-derived coordinate alone is never a verified target.