NDJSON stdio protocol

September 20, 2026 ยท View on GitHub

Start one long-lived helper process:

windows2text.exe serve

Write one JSON object per stdin line. The process writes exactly one JSON response per stdout line and reserves stderr for fatal startup diagnostics.

Envelope

Request:

{"id":"request-1","method":"snapshot","params":{"scope":"foreground","maxDepth":8,"maxNodes":500}}

Successful response:

{"id":"request-1","ok":true,"result":{}}

Failed response:

{"id":"request-1","ok":false,"error":{"code":"unknown_ref","message":"..."}}

id may be a string or number and is copied to the response. Requests are executed sequentially in input order.

Observation methods

ping

Returns the executable version, protocol version, current snapshot ID, and supported method names.

{"id":1,"method":"ping"}

windows

Lists top-level windows. maxNodes limits the result count.

{"id":2,"method":"windows","params":{"maxNodes":100}}

snapshot

Captures a new UIA tree and replaces all previously cached element references.

{"id":3,"method":"snapshot","params":{"scope":"foreground","view":"control","maxDepth":8,"maxNodes":500,"maxText":2000,"visibleOnly":false}}

Server references include the snapshot ID, for example s3:e12. After another snapshot, s3:e12 expires and actions return unknown_ref instead of targeting a potentially different element.

Snapshot parameters match the CLI options. scope can be foreground, desktop, cursor, process, or hwnd.

observe

Captures a snapshot and projects actionable UIA elements into the ordered visible list expected by Jev computer-use decision loops. targets has the same ordering and maps each label back to a ref and supported actions.

{"id":4,"method":"observe","params":{"scope":"foreground","maxDepth":8,"maxNodes":500,"maxTargets":100}}

Example result shape:

{
  "snapshotId":"s1",
  "app":"Settings",
  "visible":["Button \"Save\" [ref=s1:e12; actions=invoke,click]"],
  "context":["Text \"Settings saved\""],
  "targets":[{"index":0,"ref":"s1:e12","label":"Button \"Save\" [ref=s1:e12; actions=invoke,click]","actions":["invoke","click"]}]
}

Set includeTree to true when the planner also needs the original hierarchy. It defaults to false to keep the Jev state small.

UI Automation actions

Actions require a ref from the latest snapshot. waitMs is optional, defaults to 100, and may be 0 to 5000.

Invoke a button or menu item:

{"id":4,"method":"invoke","params":{"ref":"s1:e12"}}

Set an editable value or numeric range:

{"id":5,"method":"set_value","params":{"ref":"s1:e9","value":"hello"}}
{"id":6,"method":"set_range","params":{"ref":"s1:e10","value":50}}

Other pattern-based actions:

{"id":7,"method":"focus","params":{"ref":"s1:e9"}}
{"id":8,"method":"toggle","params":{"ref":"s1:e15"}}
{"id":9,"method":"select","params":{"ref":"s1:e18"}}
{"id":10,"method":"expand","params":{"ref":"s1:e20"}}
{"id":11,"method":"collapse","params":{"ref":"s1:e20"}}
{"id":12,"method":"scroll_into_view","params":{"ref":"s1:e25"}}

Scroll a container. Each axis accepts no_amount, small_increment, small_decrement, large_increment, or large_decrement:

{"id":13,"method":"scroll","params":{"ref":"s1:e5","horizontal":"no_amount","vertical":"large_increment"}}

Input fallback actions

Physically click the UIA clickable point, or the center of its bounds as fallback:

{"id":14,"method":"click","params":{"ref":"s1:e12","button":"left","count":1}}

Focus an element and enter Unicode text. clear sends Ctrl+A, then Backspace, first:

{"id":15,"method":"type_text","params":{"ref":"s1:e9","text":"hello world","clear":true}}

Send a key or chord. Supported names include arrows, Enter, Tab, Escape, Home, End, PageUp, PageDown, Insert, Delete, Backspace, F1-F12, and modifiers CTRL, SHIFT, ALT, and WIN:

{"id":16,"method":"key","params":{"ref":"s1:e9","key":"CTRL+SHIFT+P","focus":true}}

click, type_text, and key use Win32 input injection and therefore require the target to run at the same or a lower Windows integrity level.

Lifecycle

{"id":99,"method":"shutdown"}

Closing stdin also terminates the helper process.

Error codes

  • invalid_request: malformed request envelope
  • invalid_params: missing or invalid method parameters
  • unknown_method: unsupported method
  • unknown_ref: reference not present in the latest snapshot
  • stale_element: the underlying UIA provider invalidated the element
  • unsupported_pattern: the element does not expose the requested UIA pattern
  • read_only: attempted to modify a read-only value
  • not_clickable: no clickable point or usable bounds
  • input_failed: Win32 input injection failed
  • action_failed: the UIA provider rejected the action
  • internal_error: unexpected implementation failure