windows2text

Current version: 0.5.0.

windows2text directly reads the Windows Accessibility / UI Automation tree and emits structured JSON or compact model-friendly text. It does not use screenshots or OCR.

The native reader has no package dependencies and builds with the .NET Framework compiler included in Windows, so a separate .NET SDK is not required. The Node integration uses the official MCP TypeScript SDK and Zod.

Build

powershell -NoProfile -ExecutionPolicy Bypass -File .\build.ps1

The executable is written to dist\windows2text.exe. windows2text.cmd builds it automatically on first use.

Run the desktop smoke tests with:

powershell -NoProfile -ExecutionPolicy Bypass -File .\test.ps1

Usage

Read the foreground window as JSON:

.\windows2text.cmd snapshot

Read it as compact text for an LLM or computer-use planner:

.\windows2text.cmd text --max-depth 6 --max-nodes 300

List top-level windows, then capture one by handle or process ID:

.\windows2text.cmd windows
.\windows2text.cmd snapshot --scope hwnd --hwnd 0x123456
.\windows2text.cmd snapshot --scope process --pid 1234

Inspect the element under the mouse pointer, including the raw accessibility tree:

.\windows2text.cmd snapshot --scope cursor --view raw --max-depth 4

Supported scopes are foreground, desktop, cursor, process, and hwnd. Supported UIA views are control, content, and raw.

Output contract

Each snapshot node can include:

• ref: snapshot-local ID such as e12
• path: tree path from the selected root
• role, name, value, and text
• automationId, className, and framework
• processId, hwnd, and UIA runtimeId
• bounds: physical screen rectangle for coordinate fallback
• state: enabled, focused, focusable, off-screen, selected, checked, etc.
• patterns: supported actions such as Invoke, Value, Toggle, and Scroll

ref and runtimeId are not durable across application restarts. A future action layer should resolve elements using a combination of process, window, AutomationId, role/name, tree path, and current bounds.

Jev / computer-use integration

MCP

The recommended agent integration is the stdio MCP server:

npm install
npm run build
$env:TYPESAFE_API_KEY = "your-key"
npm run mcp

It exposes windows_list, windows_observe, windows_snapshot, windows_decide, windows_act, windows_step, and windows_run. Observation and explicit actions work without a Jev key; Jev-backed tools read TYPESAFE_API_KEY from the server process environment.

Use windows_run for multi-step tasks. The complete UIA/Jev/action loop stays inside the MCP server, so the host LLM makes one tool call and receives compact step summaries rather than a full tree after every action.

windows_decide and windows_step send the goal, application name, projected accessibility labels and values, notes, previous action, and proposed input text to the configured TypeSafe endpoint. Use observation-only tools when desktop content must remain local.

See MCP.md and the ready-to-edit configs under examples.

Modular Node API

The package exports three independent layers:

• Windows2TextClient: long-lived Windows UIA observation and action process
• JevClient: direct TypeSafe System One API client
• WindowsComputerUse: observation, Jev decision, confidence gate, and execution loop

Set the API key only in the process environment:

$env:TYPESAFE_API_KEY = "your-key"

Do not put the key in source files or command arguments.

One gated decision without execution:

import { WindowsComputerUse } from "./lib/index.mjs";

const computer = new WindowsComputerUse();
try {
  const result = await computer.step("Open the settings page");
  console.log(result.decision);
} finally {
  await computer.close();
}

Execute one step only when Jev's safety gate returns execute:

const result = await computer.step("Open the settings page", { act: true });

Run a bounded loop:

const result = await computer.run("Open the settings page", {
  act: true,
  maxSteps: 8,
  getText: async ({ decision }) => {
    // Jev selects the type action and target but does not generate text.
    return generateTextFor(decision);
  },
});

WindowsComputerUse accepts injected windows and jev clients, so applications can replace either side with a mock, another transport, or a custom policy.

Combined CLI

Build first, then use the dependency-free Node CLI:

npm run build
node .\bin\windows2text-jev.mjs observe
node .\bin\windows2text-jev.mjs decide --goal "Open Settings"
node .\bin\windows2text-jev.mjs step --goal "Open Settings" --act

decide is always dry-run. step and run are also dry-run unless --act is supplied. Even with --act, confirm, escalate, and abort gates are never executed automatically.

Run the synthetic end-to-end Jev test without touching user applications:

powershell -NoProfile -ExecutionPolicy Bypass -File .\test-live.ps1
powershell -NoProfile -ExecutionPolicy Bypass -File .\test-live.ps1 -Act

The test compiles its own WinForms fixture, asks Jev to choose an action, and with -Act verifies the result by reading the fixture's UIA state again.

Raw Protocol

For observations only, execute the CLI and parse stdout:

windows2text.exe snapshot --scope foreground --max-depth 8 --max-nodes 500

For observation plus actions, keep one helper process running:

windows2text.exe serve

Send one JSON request per stdin line and read one JSON response per stdout line:

{"id":1,"method":"observe","params":{"scope":"foreground","maxDepth":8,"maxNodes":500}}
{"id":2,"method":"invoke","params":{"ref":"s1:e12"}}
{"id":3,"method":"type_text","params":{"ref":"s1:e18","text":"hello","clear":true}}

The server supports UIA actions (invoke, set_value, set_range, toggle, select, expand, collapse, scroll) and input fallbacks (click, type_text, key). See PROTOCOL.md for the complete transport and action contract.

Each server snapshot replaces the reference cache. References such as s1:e12 therefore cannot accidentally resolve to a different element after a new observation.

The observe result contains app, an ordered Jev-ready visible string array, and an equally ordered targets array. This matches the public Jevbridge computer-use shape: pass goal, app, and visible to jev_computer_use, obey its confidence gate, map the selected visible label back through targets, then execute the corresponding ref here.

examples/jev-adapter.mjs provides compatibility exports for the modular clients:

import { WindowsComputerUse } from "./examples/jev-adapter.mjs";

const computer = new WindowsComputerUse();
const result = await computer.step("Open the settings page");
await computer.close();

Jevbridge actions map as follows: click uses InvokePattern when available and coordinate click otherwise; type uses ValuePattern or Unicode input; scroll uses UIA scrolling; wait stays local; screenshot requests a fresh accessibility observation; done and abort terminate the loop. A confirm, escalate, or abort gate is never executed automatically.

Limitations

• UIA cannot read text painted on a canvas, games, video, or inaccessible custom controls. Those require OCR or vision fallback.
• A normal process cannot inspect elevated applications reliably. Run windows2text at the same integrity level as the target.
• The Windows secure desktop cannot be inspected.
• Browser and Electron accessibility trees depend on the application's accessibility support and settings.

Related projects

• FlaUI: a full-featured MIT-licensed .NET UIA2/UIA3 wrapper; recommended if the project later adopts a modern .NET SDK.
• pywinauto: a mature BSD-licensed Python Win32/UIA automation library.
• Windows-MCP: useful reference for accessibility snapshots, action IDs, and MCP computer-use design.
• Accessibility Insights for Windows: Microsoft's open-source accessibility inspection tool and a useful robustness reference.