API Contract

The initial API namespace is peekaboox.v1.

Core RPCs:

• CaptureScreen
• CaptureDelta
• CaptureBackends
• MoveMouse
• Click
• Drag
• TypeText
• Hotkey
• FindElement
• ListWindows with optional id, app, title, title_regex, focused, limit, sort, backend, and diagnose fields plus backend metadata in the response
• GetDesktopState
• OcrScreen
• CompareImages
• DetectUiState
• DetectUiElements

Rust and Python implementations must treat the protobuf contract as the stable boundary between daemon, CLI, MCP server, and agent runtime.

gRPC API

peekabooxd exposes the protobuf contract over gRPC using the service defined in proto/peekaboox/v1/peekaboox.proto.

Incremental capture uses peekaboox-capture's frame path. Backends that can write image bytes to stdout are decoded directly in memory; file-only backends remain available as an internal fallback. The capture crate also exposes zero_copy_capture_capabilities(), select_zero_copy_backend(), and open_pipewire_screencast() for the optional Portal/PipeWire DMA-BUF path. With the pipewire-backend crate feature, capture_screen_dmabuf() consumes that PipeWire stream and returns DMA-BUF plane descriptors. DmaBufFrameDescriptor owns its duplicated plane file descriptors and closes them when dropped. prepare_dmabuf_import_descriptor() and import_dmabuf_frame() validate those descriptors and produce the backend handoff contract for EGL, Vulkan, or compute importers. With the optional egl-backend feature, EglDmaBufImporter imports the checked descriptor into a native EGLImage, and EglTextureDmaBufImporter binds that image as a GLES GL_TEXTURE_2D. The CLI command peekaboox capture-backends prints the same diagnostics as text or JSON, can include missing backend reasons with --diagnose, and can run file, frame, region, or DMA-BUF probes with --probe so target systems can be checked before enabling the importer. peekaboox capture-dmabuf --import egl-texture runs a live descriptor/import/texture probe when the CLI is built with --features pipewire-backend,egl-backend.

Default gRPC address:

127.0.0.1:47777

Start the daemon with both gRPC and local JSON IPC enabled:

cargo run -q -p peekabooxd -- run

Use a custom gRPC address or disable gRPC:

cargo run -q -p peekabooxd -- run --grpc-addr 127.0.0.1:47778
cargo run -q -p peekabooxd -- run --no-grpc

Protect the gRPC API with a shared token:

PEEKABOOX_GRPC_TOKEN=secret cargo run -q -p peekabooxd -- run
PEEKABOOX_GRPC_TOKEN=secret python3 -m peekaboox.mcp.server

The daemon rejects non-loopback --grpc-addr values unless --grpc-token or PEEKABOOX_GRPC_TOKEN is configured. Python clients, AgentRuntime.connect(), and peekaboox-mcp forward the token as x-peekaboox-token metadata.

The daemon starts a best-effort CTRL + ALT + ESC emergency hotkey listener by default. It reads Linux /dev/input/event* devices and shuts the daemon down while releasing common modifiers when the hotkey is pressed. Disable it with:

cargo run -q -p peekabooxd -- run --no-emergency-hotkey

Vision fallback for semantic element and click lookups is disabled by default. Enable it daemon-wide with:

cargo run -q -p peekabooxd -- run --vision-fallback

The daemon caches the AT-SPI semantic tree briefly so repeated semantic queries avoid a full desktop traversal. The default TTL is 500ms:

cargo run -q -p peekabooxd -- run --accessibility-cache-ttl-ms 500

It also subscribes to AT-SPI focus, window, and object events and invalidates the cache when UI state changes. Disable that best-effort listener for debugging with:

cargo run -q -p peekabooxd -- run --no-accessibility-events

Current gRPC method coverage:

• CaptureScreen for full-screen, region, or window_id PNG capture
• CaptureDelta for persistent low-bandwidth full-screen, region, or window_id deltas with raw changed-rectangle patch bytes, plus explicit full-frame mode
• CaptureBackends for screenshot backend discovery plus optional file, frame, region, or DMA-BUF probe diagnostics
• MoveMouse for absolute, relative, region/window ratio, dry-run, smooth, and bounds-aware pointer movement
• Click for coordinate clicks and AT-SPI semantic_selector clicks, with optional vision_fallback
• Drag for absolute, current-position, region/window ratio, dry-run, smooth, bounded, backend-selected, and cursor-restoring pointer drags
• TypeText
• PasteText for clipboard-backed text insertion with dry-run detection, clipboard backend selection, paste hotkey backend selection, paste timing, and optional textual clipboard restoration
• Hotkey for keyboard shortcuts such as ctrl+s
• FindElement through AT-SPI selector queries, with optional vision_fallback
• ListWindows
• GetDesktopState with windows, active-window metadata, and AT-SPI UI elements
• OcrScreen for Tesseract-backed full-screen or region OCR
• CompareImages for image-byte visual diffs with region, ignore-region, tolerance, size-policy, and alpha options
• DetectUiState for image-sequence stable/loading/changing classification
• DetectUiElements for vision-only UI-region detection from image bytes
• ProbeDmaBuf for the optional DMA-BUF capture/import path
• ListPlugins and CallPluginTool for plugin discovery and bounded process tool execution. Python/MCP callers can require trusted plugin manifests by passing require_trusted plus an optional trust_policy path; local CLI execution exposes the same guard through plugin-call --require-trusted.
• DesktopProfiles, DesktopFocus, DesktopLocate, DesktopClick, DesktopDrag, DesktopTypeInto, and DesktopAssert for named app-target desktop helpers backed by the Rust desktop profiles. DesktopProfiles returns schema_version, count, aliases, desktop ids, full launch commands with arguments, per-target capabilities, and optional availability fields. The action helpers accept optional window_title or window_id scoping; live DesktopClick, DesktopDrag, and DesktopTypeInto requests focus the scoped app/window before resolving coordinates unless dry_run or an offline image is used. Mutating helper actions also accept verify to run a post-action guard before returning.

Supported FindElement selector forms:

• Submit matches element labels containing Submit
• label=Submit
• label-exact=Submit
• label-regex=^Sub.*
• text=Submit
• role=push button,label=Submit
• role-exact=push button
• id=button-1
• state=enabled
• not-state=disabled
• bounds=10,20,90,30
• contains=55,35
• within=0,0,400,300
• intersects=40,40,80,30
• min-width=40
• min-height=20
• confidence>=0.9

Selector parsing is strict for daemon, CLI, and Rust accessibility lookups: unknown keys, malformed geometry, invalid numbers, and invalid regexes are reported as errors. FindElementRequest also accepts app, window_title, and window_id scope fields plus optional vision fallback tuning fields: vision_region, vision_edge_threshold, vision_min_width, vision_min_height, vision_min_component_pixels, vision_max_elements, and vision_merge_distance.

UiElement responses include id, role, optional label, bounds, optional center, confidence, AT-SPI states, and hierarchy metadata when available: window_id, window_title, app_id, parent_id, and child_ids. FindElementResponse additionally reports backend name/kind, warnings, cache hit status, cache age, and whether vision fallback was used.

When FindElementRequest.vision_fallback or ClickRequest.vision_fallback is true, the daemon tries the AT-SPI path first and only captures the current screen for vision-based fallback if accessibility lookup fails or finds no matching element. The same fallback can be enabled for all daemon requests with --vision-fallback or PEEKABOOX_VISION_FALLBACK=1.

Daemon-routed MoveMouse, Click, Drag, TypeText, and Hotkey requests require the daemon to be started with --profile operator, --allow-input, or PEEKABOOX_ALLOW_INPUT=1. Daemon-routed plugin tool execution uses a separate gate: --profile operator, --allow-plugins, or PEEKABOOX_ALLOW_PLUGINS=1.

ClickRequest accepts one target at a time: coordinates, semantic_selector, or a region/window scope with ratio_x and ratio_y. It also supports button, dry_run, bounds_policy, backend, and restore for parity with the CLI.

Python Runtime Client

The Python package includes generated protobuf bindings under peekaboox.v1 and a synchronous runtime client:

from peekaboox.agent import AgentRuntime
from peekaboox.client import Rect
from peekaboox.security import (
    CapabilityProfile,
    ConfirmationPolicy,
    DangerousAction,
)

runtime = AgentRuntime.connect(
    "127.0.0.1:47777",
    capability_profile=CapabilityProfile.ASSIST,
    confirmation_policy=ConfirmationPolicy.require_for([DangerousAction.CLICK]),
    audit_log_path="peekaboox-runtime-audit.jsonl",
)
windows = runtime.list_windows()
focused = runtime.list_windows(focused=True, limit=1, sort="focused")
window_result = runtime.list_windows_result(app="calculator", diagnose=True)
state = runtime.get_desktop_state()
doctor = runtime.doctor()
buttons = runtime.find_element("role=push button", vision_fallback=True)
text = runtime.ocr_region(Rect(x=10, y=20, width=400, height=120), language="eng")
delta = runtime.capture_delta(
    stream_id="agent-loop",
    region=Rect(x=10, y=20, width=400, height=240),
    per_channel_threshold=2,
    low_bandwidth=True,
)
backends = runtime.capture_backends(output="screen.png", diagnose=True, probe="frame")
window_capture = runtime.capture_screen(window_id="window-1")
app_capture = runtime.capture_screen(app="calculator", title_regex="Calculator")
window_region = runtime.capture_screen(
    app="calculator",
    region=Rect(x=10, y=10, width=220, height=160),
)
diff = runtime.compare_image_files(
    "before.png",
    "after.png",
    ignore_regions=[Rect(x=10, y=20, width=80, height=24)],
    max_changed_ratio=0.01,
    max_changed_pixels=100,
    size_policy="common-region",
)
ui_state = runtime.detect_ui_state_from_image_files(
    ["frame1.png", "frame2.png", "frame3.png"],
    ignore_regions=[Rect(x=10, y=20, width=80, height=24)],
    stable_max_changed_pixels=20,
)
target = runtime.desktop_locate("telegram", "search-input")
profiles = runtime.desktop_profiles("telegram", supports="type-into", check=True)
runtime.desktop_click("telegram", "search-input", dry_run=True)
runtime.desktop_type_into("telegram", "message-input", "PeekabooX", dry_run=True)
runtime.click_selector("role=push button,label=Submit", vision_fallback=True)
runtime.click(
    region=Rect(x=0, y=0, width=400, height=240),
    ratio_x=0.5,
    ratio_y=0.5,
    button="right",
    bounds_policy="clamp",
    backend="auto",
    dry_run=True,
)
runtime.move_mouse(100, 200)
runtime.move_mouse(relative_x=24, relative_y=0, dry_run=True)
runtime.move_mouse(
    region=Rect(x=0, y=0, width=400, height=240),
    ratio_x=0.5,
    ratio_y=0.5,
    duration_ms=180,
    steps=8,
    bounds_policy="clamp",
    backend="auto",
    dry_run=True,
)
runtime.drag(100, 200, 360, 260, button="left", duration_ms=350)
runtime.drag(
    from_current=True,
    to_ratio=(0.8, 0.5),
    region=Rect(x=0, y=0, width=400, height=240),
    steps=10,
    bounds_policy="clamp",
    backend="xdotool",
    restore=True,
    dry_run=True,
)
runtime.hotkey(
    "control+s",
    dry_run=True,
    backend="auto",
    delay_ms=25,
    key_delay_ms=30,
    repeat=2,
    interval_ms=40,
    release_before=True,
    release_after=True,
)

Use client.find_elements(...) when callers need backend, warning, cache, and vision fallback metadata; client.find_element(...) remains the backward compatible tuple-of-elements helper.

AgentRuntime accepts either a custom CapabilityPolicy or a capability_profile for granular in-process permission checks. The profile names are observe, plan, assist, and operator. The supported capabilities are observe, click, type_text, workflow_execute, workflow_record, workflow_generate, vision, memory_read, and memory_write, plugin_read, and plugin_execute. Denied calls raise CapabilityDeniedError, and all checks append an in-memory CapabilityAuditEvent retrievable through runtime.capability_audit(). Preflight checks append PreflightAuditEvent entries retrievable through runtime.preflight_audit(). This policy is layered above the daemon's input policy; daemon-routed input still requires peekabooxd run --profile operator, --allow-input, or PEEKABOOX_ALLOW_INPUT=1.

ConfirmationPolicy adds optional confirmation checks for dangerous runtime operations before execution. The dangerous action names are click, type_text, and workflow_execute; paste_text is confirmed under type_text, while pointer movement, drags, and hotkeys are confirmed under the click action gate. Required confirmations without a configured confirmer raise ConfirmationRequiredError; rejected confirmations raise ConfirmationDeniedError. Audit events are available through runtime.confirmation_audit(). When audit_log_path is set, capability, confirmation, and preflight checks are also persisted as newline-delimited JSON records.

Semantic Desktop Graph

AgentRuntime owns a MemoryStore with a SemanticDesktopGraph. Call ingest_desktop_snapshot() to sample GetDesktopState and store a graph snapshot, or pass an existing DesktopState to avoid an extra daemon call:

snapshot = runtime.ingest_desktop_snapshot()
latest = runtime.latest_desktop_snapshot()
matches = runtime.query_desktop_graph(
    kind="element",
    label_contains="submit",
    contained_by="window-1",
)
graph_json = runtime.memory.export_desktop_graph()

Event updates are handled through record_desktop_event(). Passing no fresh DesktopState invalidates the current graph and records affected node IDs when the event targets a known window or element. Passing state=... writes a new snapshot immediately. Queries can opt into automatic refresh:

runtime.record_desktop_event(
    kind="accessibility.element.changed",
    source="accessibility",
    target_id="button-1",
)
status = runtime.desktop_graph_status()
matches = runtime.query_desktop_graph(kind="element", refresh_if_stale=True)

Snapshots contain graph nodes for the snapshot, windows, and UI elements, plus edges for has_window, active_window, focused_window, has_element, and window-to-element contains relationships. Graph queries can filter by node kind, label substring, role, exact attribute values, and containment window. The runtime also uses fresh graph snapshots as a semantic lookup cache: find_element(selector) and semantic click_selector(selector) first evaluate the selector against cached graph elements. If the graph is stale, absent, or no cached element matches, the call falls back to the daemon path with the same vision_fallback behavior as before.

For persistent memory, pass a SQLite path when connecting or instantiate a store directly:

from peekaboox.memory import SQLiteMemoryStore

runtime = AgentRuntime.connect(memory_path="peekaboox-memory.sqlite3")
store = SQLiteMemoryStore("peekaboox-memory.sqlite3")

SQLiteMemoryStore persists key/value memory plus graph snapshots into normalized desktop_graph_snapshots, desktop_graph_nodes, and desktop_graph_edges tables. It also persists desktop_state_events, desktop_graph_invalidations, and stale/fresh metadata while still supporting JSON round-trips through export_desktop_graph() and import_desktop_graph().

AgentRuntime also exposes deterministic workflow execution:

• plan_workflow(goal) creates the current simple observe workflow.
• generate_workflow(goal, refresh_desktop_graph=False) creates an editable workflow draft from a goal and optional graph context.
• save_generated_workflow(goal, path, format_name=None) writes that draft as JSON or YAML.
• refine_workflow(goal, workflow=None, refresh_desktop_graph=False) sends a draft through the configured structured workflow provider.
• replan_workflow(goal, failed_workflow, failed_result, ...) asks the configured replanning provider for a validated replacement workflow after a failed execution.
• save_refined_workflow(goal, path, workflow=None, format_name=None) writes the refined draft as JSON or YAML.
• execute_goal(goal, replan_on_failure=True, max_replans=1) plans and executes that workflow, then can run one or more validated replans after a failure.
• execute_workflow(workflow) executes explicit WorkflowStep sequences.
• load_workflow_file(path) loads a JSON or YAML workflow definition.
• execute_workflow_file(path) loads and executes that workflow definition.
• list_workflow_templates(...) and workflow_template_info(template_id) expose built-in editable workflow templates.
• start_recording(name) starts capturing subsequent actions as workflow steps.
• stop_recording() ends the active recording and returns a Workflow.
• recorded_workflow() returns the active or last completed recording.
• save_recording(path, format_name=None) exports the recording as JSON/YAML.
• peekaboox-agent workflow validate|replay|bundle validates workflow files, replays them through the daemon, and writes portable replay bundles with normalized workflow files, Doctor diagnostics, and optional replay results.
• execute_step(step) retries a single step up to retries + 1 attempts.

Execution results include per-attempt messages, verification results, and structured recovery metadata. Failed workflows report failed_step, action, reason, attempts, and next_action. Selector replay can also report strategies and events when self-healing was attempted. Built-in actions are observe, find_element, click, move_mouse, drag, type_text, paste_text, hotkey, list_windows, and get_desktop_state. click, type_text, paste_text, and other input actions sample desktop state after successful execution as the current verification hook; callers can pass a custom verifier for stricter domain-specific checks. For selector-based find_element and click steps, retry attempts first refresh the semantic desktop graph and later enable vision_fallback when the original step did not request it. Each ActionAttempt exposes a recovery object for the strategy applied before that attempt. Generated workflows are drafts, not implicit execution. They start with observe, use graph-backed selectors when a target label is known, include find_element before selector clicks, and can be serialized through the same JSON/YAML workflow file helpers. Provider-backed refinement and replanning are optional and never execute provider output directly. A provider must return a Workflow, a workflow object, or JSON/YAML workflow text. The runtime validates every returned WorkflowStep against the supported action set before exposing, saving, or executing it.

Workflow files use the same fields as WorkflowStep: action, selector, value, pointer coordinates, scoped window fields, backend, dry_run, vision_fallback, and verify. type_text steps can also set typing_speed_chars_per_second, delay_ms, and key_delay_ms; paste_text steps can set preserve_clipboard, dry_run, clipboard_backend, hotkey_backend, delay_ms, restore_delay_ms, and restore_policy; hotkey steps can set backend, delay_ms, key_delay_ms, repeat, interval_ms, release_before, and release_after. JSON is parsed with the standard library; YAML support covers the repository workflow shape without an extra runtime dependency. The recorder writes the same schema, so recorded workflows can be reviewed, edited, and replayed through execute_workflow_file. For recorded coordinate clicks, the runtime attempts to resolve the point against a fresh semantic desktop graph and records a unique semantic selector when possible. If no element resolves, the step falls back to x/y.

MCP Tool Surface

peekaboox-mcp exposes a local MCP-style registry with tool descriptors, JSON-schema input metadata, and a call_tool(name, arguments) dispatcher bound to AgentRuntime. It includes stdio JSON-RPC for MCP clients plus HTTP and SSE transports for local bridge processes:

PYTHONPATH=python/src python3 -m peekaboox.mcp.server --list-tools
PYTHONPATH=python/src python3 -m peekaboox.mcp.server
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --audit-log runtime-audit.jsonl
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --capability-profile observe
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --preflight-mode strict
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --grpc-timeout 15
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --grpc-token secret
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --transport http --port 47778 --auth-token secret
PYTHONPATH=python/src python3 -m peekaboox.mcp.server --transport sse --port 47778 --auth-token secret

The stdio and HTTP transports handle initialize, ping, tools/list, tools/call, resources/list, resources/read, resources/templates/list, prompts/list, prompts/get, completion/complete, logging/setLevel, and notifications/initialized. Tool calls return structuredContent plus a serialized JSON text content block for compatibility. Image-producing tools such as capture_screen also return an MCP image content block. Desktop action tools declare an outputSchema with focus_diagnostics; MCP clients can read the same ordered focus lookup, fallback, and verification entries from structuredContent without parsing the text content block. HTTP accepts JSON-RPC POST requests on /mcp; SSE exposes /sse with endpoint and tool-list events for clients that discover HTTP JSON-RPC through server-sent events. Tool execution requires the Python runtime dependencies and a reachable PeekabooX daemon at PEEKABOOX_GRPC_TARGET or the --target address; without those dependencies the server can still list tool descriptors for inspection. MCP tools are bound to the runtime's CapabilityPolicy; denied tool execution returns a normal tools/call result with isError: true, structuredContent.error set to CapabilityDeniedError, and structured capability, operation, and next_action fields. When the runtime has a ConfirmationPolicy, missing or denied confirmations are reported the same way with ConfirmationRequiredError or ConfirmationDeniedError, including action, operation, retryable, and next_action. Doctor-backed preflight blocks return PreflightError with blocked_categories, warning_categories, next_action, and the full preflight result in structuredContent. examples/mcp/jsonrpc_preflight_error_client.sh is a CI-safe client-side sample for handling that response shape and prompting the user to run Doctor. Set --audit-log or PEEKABOOX_RUNTIME_AUDIT_LOG to persist those runtime checks for MCP sessions. Set --capability-profile, PEEKABOOX_MCP_CAPABILITY_PROFILE, or PEEKABOOX_CAPABILITY_PROFILE to apply a reusable runtime allowlist to MCP tool calls. Set --preflight-mode off|warn|strict and --preflight-timeout <seconds> to enable Doctor-backed preflight checks directly at MCP server startup. Set --grpc-timeout or PEEKABOOX_GRPC_TIMEOUT when desktop focus or launch actions need more time than the default daemon RPC deadline. HTTP/SSE transports accept --auth-token or PEEKABOOX_MCP_TOKEN; non-loopback hosts are rejected without a token. Clients can send either Authorization: Bearer <token> or X-PeekabooX-MCP-Token. Request bodies are limited by --max-request-bytes or PEEKABOOX_MCP_MAX_REQUEST_BYTES.

MCP resources expose server state, tool descriptors, workflow templates, desktop profile metadata, latest Doctor/preflight state, desktop graph status/snapshots, plugin discovery, runtime audit events, and selected repository docs under peekaboox://... URIs. MCP prompts provide reusable guidance for diagnostics, safe desktop actions, window inspection, workflow building, structured error recovery, plugin development, OCR, and semantic click planning. Completion supports common arguments such as tools, prompts, resources, app profiles, desktop targets, preflight categories, capability profiles, workflow actions, formats, and log levels.

The current tool surface includes:

• capture_screen
• capture_delta
• capture_backends
• capture_dmabuf
• doctor
• preflight
• probe_dmabuf
• click
• move_mouse
• drag
• type_text
• paste_text
• hotkey
• find_element
• find_elements
• elements
• list_windows
• list_plugins
• call_plugin_tool
• get_desktop_state
• desktop_profiles
• desktop_focus
• desktop_locate
• desktop_click
• desktop_drag
• desktop_type_into
• desktop_assert
• ingest_desktop_snapshot
• latest_desktop_snapshot
• record_desktop_event
• desktop_graph_status
• refresh_desktop_graph
• query_desktop_graph
• query_desktop_edges
• ocr
• ocr_image
• ocr_screen
• compare_images
• detect_ui_state
• detect_ui_elements
• vision_elements
• plan
• plan_workflow
• execute_goal
• generate_workflow
• save_generated_workflow
• refine_workflow
• save_refined_workflow
• replan_workflow
• load_workflow_file
• execute_workflow
• execute_workflow_file
• start_workflow_recording
• stop_workflow_recording
• get_recorded_workflow
• save_recorded_workflow
• capability_audit
• confirmation_audit
• preflight_audit

click accepts either x/y coordinates or selector/semantic_selector. find_elements and elements are CLI-compatible aliases around semantic lookup and add limit; vision_elements aliases detect_ui_elements and accepts the same image path, region, ignore-region, confidence/size/area, sorting, padding, mask-output, and overlay-output arguments; ocr/ocr_image alias the OCR surface; and capture_dmabuf aliases probe_dmabuf. list_windows supports id, app, title, title_regex, focused, limit, sort, backend, and diagnose arguments through MCP, matching the daemon CLI and Python runtime client. capture_screen accepts optional region, window_id, app, window_title, and title_regex. When a region is combined with a window filter, the region is resolved relative to the matched window before capture. capture_delta accepts stream_id, reset, optional region or window_id, per_channel_threshold, and low_bandwidth for persistent low-bandwidth capture streams. capture_backends accepts output, optional region, diagnose, and probe values none, file, frame, region, dmabuf, or all. doctor accepts optional strict and timeout_seconds arguments and returns the structured peekaboox doctor --json health checks with per-check category/severity fields and top-level category summaries. preflight accepts categories, optional operation, refresh, timeout_seconds, and require; it returns the Doctor-backed category gate used by AgentRuntime(preflight_mode="strict") before live automation. move_mouse accepts absolute x/y, relative_x/relative_y, region/window filters with ratio_x/ratio_y, dry_run, duration_ms, steps, bounds_policy, backend, and restore, matching the CLI move surface. The desktop helper tools accept supported app profile names such as telegram, paint, drawing, pinta, kolourpaint, and text-editor, plus named targets such as Telegram's search-input/message-input, Paint's canvas, or Text Editor's document. Additional profiles can be supplied through PEEKABOOX_DESKTOP_PROFILE_PATH using the desktop-profile.v1 JSON format documented in the CLI guide. desktop_profiles supports app, target, command, desktop_id, supports, check, installed, and available filters and is the discovery surface for command arguments, target capability metadata, external generic targets, and installation checks. External target metadata can advertise and use OCR/text anchors, RGB color anchors, and wait rules through sources/supports values such as ocr, text-anchor, color-anchor, and wait. Use window_id for exact-window targeting when multiple windows share an app profile; use verify: true on focus, click, drag, or type-into calls when the caller needs an immediate postcondition check. desktop_focus, desktop_click, desktop_drag, and desktop_type_into return focus_diagnostics when they focus an app or window before the requested action; the field is also present as an empty list when no focus path ran. click and find_element both accept vision_fallback: true; when a fresh graph cache hits, find_element returns cached elements directly and semantic click uses the cached element center as a coordinate click. File-oriented vision tools use expected_path/actual_path or image_path arguments so MCP callers do not need to pass raw image bytes. ingest_desktop_snapshot samples the current desktop state and appends it to the runtime's semantic desktop graph; latest_desktop_snapshot returns the most recent stored snapshot or null. record_desktop_event records a desktop change and invalidates the graph; desktop_graph_status reports stale state and the latest invalidation; refresh_desktop_graph samples a fresh graph snapshot. query_desktop_graph filters stored graph nodes by kind, label_contains, role, attribute_equals, contained_by, latest_only, and optionally refresh_if_stale. query_desktop_edges filters stored graph edges by source, target, kind, and latest_only. execute_goal accepts a goal string plus optional replan_on_failure and max_replans, then runs the runtime planner plus workflow loop. plan returns high-level planning steps, plan_workflow returns a draft workflow, replan_workflow accepts a failed workflow plus structured recovery metadata, and load_workflow_file loads JSON/YAML workflows without executing them. generate_workflow accepts goal, optional refresh_desktop_graph, and optional format of json or yaml; it returns the workflow object plus serialized text. save_generated_workflow writes the same draft to path. refine_workflow accepts goal, optional workflow, optional refresh_desktop_graph, and optional format; it returns the validated provider-refined workflow object plus serialized text. save_refined_workflow writes that validated draft to path. execute_workflow accepts name and steps; each step supports action, selector, value, x, y, from_x, from_y, to_x, to_y, button, duration_ms, vision_fallback, and verify. execute_workflow_file accepts path and loads a JSON/YAML workflow file before executing the same retry/verification loop. Workflow recording tools capture subsequent capture_screen, find_element, click, type_text, and paste_text calls as replayable steps; save_recorded_workflow accepts path and an optional format of json or yaml. list_plugins accepts optional paths, and call_plugin_tool accepts plugin_id, tool, optional arguments, optional paths, and optional timeout_seconds and max_output_bytes; execution is gated by plugin_execute. Recorded coordinate clicks are enriched from the graph cache when possible, so the saved step can replay through selector instead of fixed x/y. Workflow tool results include the same per-attempt verification and recovery metadata as the Python runtime API, including selector self-healing strategies such as refresh_desktop_graph and vision_fallback. Use capability_audit, confirmation_audit, and preflight_audit when MCP clients need in-band inspection of the runtime security decisions.

The client uses the checked-in generated modules from proto/peekaboox/v1/peekaboox.proto. Regenerate them after proto changes with:

python3 -m pip install -e "python[dev]"
python3 -m grpc_tools.protoc \
  -I proto \
  --python_out=python/src \
  --grpc_python_out=python/src \
  proto/peekaboox/v1/peekaboox.proto

Local IPC

For the Rust CLI, peekabooxd also exposes the daemon surface over a local Unix socket using newline-delimited JSON.

Default socket:

$XDG_RUNTIME_DIR/peekabooxd.sock

Start the daemon:

cargo run -q -p peekabooxd -- run
cargo run -q -p peekabooxd -- run --sandbox basic

Route CLI commands through the daemon:

cargo run -q -p peekaboox-cli -- --daemon windows
cargo run -q -p peekaboox-cli -- --daemon windows --focused --limit 1 --json

Inspect the installed Python runtime without starting an MCP client:

peekaboox-agent --version
peekaboox-agent plugins --path examples/plugins
peekaboox-agent preflight desktop capture --operation capture_screen
peekaboox-agent windows
peekaboox-agent --preflight-mode strict windows
peekaboox-agent windows --focused --limit 1 --sort focused
peekaboox-agent windows --app calculator --diagnose
peekaboox-agent desktop-state

Supported request methods:

• ping
• capture
• capture_delta
• capture_backends
• desktop_profiles
• desktop_focus
• desktop_locate
• desktop_click
• desktop_drag
• desktop_type_into
• desktop_assert
• click
• move_mouse
• drag
• type_text
• paste_text
• hotkey
• list_windows
• find_elements
• ocr
• compare_images
• detect_ui_state
• detect_ui_elements
• probe_dmabuf
• list_plugins
• call_plugin_tool

Daemon-routed capture requests accept output plus optional region, window_id, app, window_title, title_regex, format (png or xwd), no_overwrite, and include_semantic_tree. JSON responses include output path, backend, byte count, dimensions, MIME type, capture region, matched window, source, timestamp, and optional semantic tree. Daemon-routed capture_delta requests accept stream_id, reset, optional region or window_id, per_channel_threshold, and low_bandwidth. low_bandwidth=true is the default and returns changed-rectangle patches after the first frame; low_bandwidth=false forces a full-frame patch for that request. Responses carry patch bytes as patch_base64 and echo low_bandwidth. Daemon-routed capture_backends requests accept output, optional region, diagnose, and probe values none, file, frame, region, dmabuf, or all, returning the same backend diagnostics and probe results as the CLI. Daemon-routed desktop_profiles requests accept app, target, command, desktop_id, supports, check, installed, and available, returning the same registry metadata as the CLI and gRPC/Python/MCP surfaces. Daemon-routed desktop action helper requests accept the same window_id, window_title, and verify fields as the gRPC/Python/MCP surfaces. Desktop action responses on the CLI, JSON IPC, gRPC, and Python client surfaces may include focus_diagnostics, a short ordered list of focus lookup, fallback, and verification steps when an action focuses an app before interacting with it. Daemon-routed probe_dmabuf requests accept import_target values compute, egl, or egl_texture when peekabooxd is built with the matching pipewire-backend/egl-backend features. Daemon-routed list_plugins requests accept optional plugin paths; otherwise the daemon uses its configured --plugin-path values plus SDK defaults. Daemon-routed call_plugin_tool requests accept plugin_id, tool, optional JSON arguments, optional plugin paths, timeout_ms, and max_output_bytes. Arguments are validated against the tool input_schema, the process runs with a restricted environment, and stdout/stderr are capped. Daemon-routed find_elements requests accept vision_fallback: true, app, window_title, window_id, vision_region, vision_edge_threshold, vision_min_width, vision_min_height, vision_min_component_pixels, vision_max_elements, and vision_merge_distance. Daemon-routed click, type_text, paste_text, pointer movement, drags, and hotkeys require dry_run: true where supported unless the daemon was started with --profile operator, --allow-input, or PEEKABOOX_ALLOW_INPUT=1. Daemon-routed hotkey requests accept keys, dry_run, backend, delay_ms, key_delay_ms, repeat, interval_ms, release_before, and release_after. Use --sandbox basic for no_new_privileges and non-dumpable daemon process state. Use --sandbox strict only on Linux hosts where unprivileged user namespaces are available; startup fails if namespace isolation cannot be applied.

The Rust CLI accepts both coordinate and semantic click targets:

cargo run -q -p peekaboox-cli -- click --x 100 --y 200 --dry-run
cargo run -q -p peekaboox-cli -- click --text "Submit" --dry-run
cargo run -q -p peekaboox-cli -- click --selector "role=push button,label=Submit" --dry-run
cargo run -q -p peekaboox-cli -- click --selector "role=button" --vision-fallback --dry-run
cargo run -q -p peekaboox-cli -- elements --selector "role=button" --vision-fallback

OCR through the CLI:

cargo run -q -p peekaboox-cli -- ocr
cargo run -q -p peekaboox-cli -- ocr --region 10,20,400,120 --language eng
cargo run -q -p peekaboox-cli -- --daemon ocr --language eng

Visual comparison through the CLI:

cargo run -q -p peekaboox-cli -- compare before.png after.png
cargo run -q -p peekaboox-cli -- compare --expected before.png --actual after.png --region 10,20,400,120 --threshold 3 --max-changed-ratio 0.01
cargo run -q -p peekaboox-cli -- compare before.png after.png --ignore-region 10,20,80,24 --max-changed-pixels 100 --max-mae 2.5 --diff-output diff.png --report diff.json
cargo run -q -p peekaboox-cli -- --daemon compare before.png after.png

UI-state/loading detection through the CLI:

cargo run -q -p peekaboox-cli -- state frame1.png frame2.png frame3.png
cargo run -q -p peekaboox-cli -- state --image frame1.png --image frame2.png --threshold 3 --stable-max-changed-ratio 0.001 --loading-min-changed-ratio 0.02
cargo run -q -p peekaboox-cli -- state frame1.png frame2.png --ignore-region 10,20,80,24 --stable-max-changed-pixels 20 --loading-min-changed-pixels 200
cargo run -q -p peekaboox-cli -- --daemon state frame1.png frame2.png

Vision-only UI element detection through the CLI:

cargo run -q -p peekaboox-cli -- vision-elements screenshot.png
cargo run -q -p peekaboox-cli -- vision-elements --image screenshot.png --region 10,20,400,300 --ignore-region 10,20,80,24 --threshold 24 --min-width 8 --min-confidence 0.8 --sort confidence --mask-output mask.png --overlay-output overlay.png --max-elements 25
cargo run -q -p peekaboox-cli -- --daemon vision-elements screenshot.png

Rust Vision OCR

peekaboox-vision exposes the current OCR surface:

• OcrBackend for provider implementations
• TesseractOcrBackend using the tesseract CLI and TSV output
• ocr_screen() for full-screen OCR
• ocr_region(Rect) for region-filtered OCR
• ocr_image_file(path, region) for OCR over an existing image file
• daemon/gRPC, Python, MCP, and CLI bindings for full-screen, region, image-file, and window-scoped OCR
• Tesseract controls for language, page segmentation mode, engine mode, DPI, minimum confidence, character whitelist, and repeated key=value config entries
• preprocessing controls for scale, grayscale, threshold, invert, contrast, and deskew

OCR text is returned as line blocks and word blocks with UiElement metadata carrying role, label, bounds, confidence, and empty states. Region OCR crops before Tesseract runs, then maps coordinates back to the screen or source image coordinate space.

Rust Vision Comparison

peekaboox-vision also exposes a frame-based visual comparison foundation:

• VisualCompareOptions selects an optional Rect region, repeated ignored regions, per-channel pixel threshold, maximum changed-pixel ratio, absolute changed-pixel limit, MAE and max-channel gates, size policy, and alpha mode.
• compare_frames(expected, actual, options) compares CaptureFrame values in RGB space across Rgb8, Rgba8, and Bgra8 by default. Set alpha mode to compare to include the alpha channel.
• VisualDiffResult reports compared pixels, changed pixels, changed ratio, mean absolute error, maximum channel delta, changed bounds, and whether the frames match the requested tolerance.
• write_visual_diff_image_file(expected, actual, output, options) writes a transparent PNG-compatible diff mask with changed pixels marked red and returns the same VisualDiffResult.
• incremental_capture_delta(previous, current, sequence, options) emits an initial full-frame patch when no previous frame exists, otherwise emits only a densely packed patch for the changed bounds reported by compare_frames.
• IncrementalCaptureDelta carries sequence number, source frame dimensions, pixel format, full-frame marker, changed bounds, changed-pixel statistics, patch stride, and patch bytes. Unchanged frames return no bounds and empty patch bytes.
• CaptureDelta exposes this over gRPC using daemon-held stream state. The daemon keeps the previous decoded frame per stream_id; reset=true forces a fresh full-frame patch. low_bandwidth=false also suppresses previous-frame reuse for that request and returns a full-frame patch while preserving the stream sequence. Region targets use native region capture, and window_id targets resolve current window bounds through the window enumerator before capture. Local IPC and CLI expose the same stream controls, and Python/MCP clients return patch bytes as bytes/base64 respectively.
• The live daemon path calls peekaboox-capture::capture_screen_frame() so incremental capture no longer performs PNG temp-file decoding in daemon code; stdout-capable capture tools feed decoded frames directly, with file-only capture retained as a backend-level fallback.
• peekaboox-capture exposes DMA-BUF zero-copy capability probing for Portal/PipeWire Wayland sessions and can open a ScreenCast session through CreateSession, SelectSources, Start, and OpenPipeWireRemote. When the crate is built with --features pipewire-backend, capture_screen_dmabuf() consumes the opened PipeWire stream, negotiates DMA-BUF buffers, and returns DmaBufFrameDescriptor plane metadata. The capture crate also exposes DmaBufImportTarget, DmaBufFrameImportDescriptor, and ValidatingDmaBufImporter so graphics/compute backends receive a checked DMA-BUF handoff before creating native EGL/Vulkan resources. DmaBufFrameImportDescriptor owns duplicated plane file descriptors, closes them on drop, and can produce another owned copy with try_clone_owned(). With --features egl-backend, EglDmaBufImporter opens an EGL display, checks EGL_EXT_image_dma_buf_import, builds EGL_LINUX_DMA_BUF_EXT attributes, and destroys the resulting EGLImage when the imported frame is dropped. EglTextureDmaBufImporter additionally creates a GLES2 context, requires GL_OES_EGL_image, binds the image to GL_TEXTURE_2D, and deletes the texture on drop. The CLI can run the same compute/EGL/GLES texture probe through the daemon over local IPC with peekaboox --daemon capture-dmabuf --import .... The default build keeps the stable frame path on owned CPU bytes and reports that optional backend features are disabled.
• CompareImages exposes the same diff result over gRPC using image bytes, including ignore regions, absolute metric gates, size_policy, and alpha.
• The local daemon IPC and CLI compare image file paths and use the same tolerance fields. The CLI adds report-oriented conveniences such as --diff-output, --report, and --no-fail.

Small image fixtures live under tests/fixtures/vision for regression tests, including screen-like PBM fixtures for decoder-backed UI-element detection, loading-state transitions, and vision-fallback lookup behavior.

Plugin SDK

Plugin discovery is defined by docs/plugins.md. The stable manifest name is peekaboox.plugin.json, and the current schema version is peekaboox.plugin.v1. The SDK validates plugin ids, declared capabilities, process entrypoints, and JSON-schema-shaped tool input metadata. The same validated descriptors are exposed through peekaboox plugins, daemon JSON IPC and gRPC list_plugins, Python AgentRuntime.list_plugins(), and MCP list_plugins. Declared process-plugin tools can be executed through peekaboox plugin-call, daemon JSON IPC and gRPC call_plugin_tool, Python AgentRuntime.call_plugin_tool(), and MCP call_plugin_tool, gated by the plugin_execute capability where the Python runtime is in the path.

Rust UI-State Detection

peekaboox-vision includes a deterministic UI-state detection foundation built on the same frame comparison primitives:

• UiStateOptions selects an optional Rect region, repeated ignored regions, per-channel threshold, stable ratio/pixel/MAE/channel gates, loading ratio/pixel gates, required trailing stable transitions, size policy, and alpha mode.
• detect_ui_state(frames, options) compares adjacent CaptureFrame samples and classifies the sequence as Stable, Loading, or Changing.
• UiStateResult reports transition counts, trailing stability, latest diff, maximum and mean changed ratio, and aggregate changed bounds.
• detect_ui_state_from_image_files and detect_ui_state_from_image_bytes provide decoder-backed helpers for fixtures and API bindings.
• DetectUiState exposes the same result over gRPC using repeated image bytes, including ignore regions, absolute gates, size_policy, and alpha.
• The local daemon IPC and CLI compare image file sequences with the same tolerance fields.

Rust UI Element Detection

peekaboox-vision includes a first deterministic UI-element detection foundation for accessibility fallback scenarios:

• UiElementDetectionOptions selects an optional Rect region, repeated ignore regions, edge/contrast threshold, minimum and maximum component dimensions/area, minimum confidence, result padding, maximum result count, merge distance, and position, area, or confidence sorting.
• detect_ui_elements(frame, options) finds salient visual components and returns them as UiElement values with role visual-region, bounds, confidence, and visible state.
• detect_ui_elements_from_image_file and detect_ui_elements_from_image_bytes provide decoder-backed helpers for fixtures and API bindings.
• detect_ui_elements_*_with_outputs can write a saliency mask image and an overlay image with detected bounds for debugging and fixture examples.
• HeuristicVisionBackend implements VisionBackend::detect_ui_elements with this fallback detector while frame OCR remains delegated to the OCR pipeline.
• DetectUiElements exposes the same detector over gRPC using image bytes, including filter options and optional output paths.
• The local daemon IPC and CLI detect elements from image file paths and return the same UiElement shape used by accessibility queries.