MCP server

windows2text-mcp exposes Windows UI Automation and Jev-gated computer use through the official Model Context Protocol TypeScript SDK over stdio.

Start

npm install
npm run build
$env:TYPESAFE_API_KEY = "your-key"
npm run mcp

The API key is only required by windows_decide and windows_step. Observation and explicit UIA actions work without it.

The MCP process must remain alive between observation and action because refs such as s1:e12 are stored in that process. Taking another observation expires earlier refs.

Tools

Tool	Side effects	Description
windows_list	None	List top-level windows and PIDs.
windows_observe	None	Return compact actionable controls and snapshot refs.
windows_snapshot	None	Return the full UIA tree.
windows_decide	Network only	Observe and ask Jev for one gated action; never execute.
windows_act	Yes	Execute one explicit action against a current ref.
windows_step	Optional	Observe, decide, and execute only when act=true and gate=execute.
windows_run	Optional	Run the bounded UIA/Jev/action loop locally and return compact step summaries.

windows_run defaults to 10 steps and 60 seconds and is capped at 20 steps and 120 seconds. It pins subsequent observations to the first HWND and returns no full UI trees in its history.

Run the synthetic end-to-end MCP test with:

powershell -NoProfile -ExecutionPolicy Bypass -File .\test-mcp-live.ps1

Recommended flow

1. Call windows_observe with scope=foreground or a specific PID.
2. Call windows_decide with the user goal.
3. If its gate returns confirm, ask the user.
4. Execute with windows_act, or use windows_step with act=true for a single gated step.
5. Observe again before choosing another action.

For multi-step goals, call windows_run once instead of repeatedly routing windows_step through the host LLM:

{
  "goal": "Create a new session, select gpt-5.6-sol, and send the supplied text",
  "act": true,
  "text": "Hello",
  "maxSteps": 10,
  "maxDurationMs": 60000
}

For text entry, Jev chooses the type action and target, but the generating model must supply the actual text. When text is supplied to windows_step, it is included in the Jev safety decision before execution.

OpenCode

Copy the mcp.windows2text entry from examples/opencode.json into the relevant OpenCode config and update the absolute path. Project config lives in opencode.json; global config lives in ~/.config/opencode/opencode.json.

After changing OpenCode configuration, quit and restart OpenCode because MCP configuration is not hot-reloaded.

Security

• Keep TYPESAFE_API_KEY in the process environment, not the JSON config or repository.
• windows_act and windows_step are annotated as destructive-capable tools.
• windows_step defaults to act=false.
• A Jev confirm, escalate, or abort gate never executes automatically.
• UIA and input injection cannot cross to a higher-integrity target process.
• windows_decide and windows_step send the goal, application name, projected accessibility labels and values, notes, previous action, and proposed input text to the configured TypeSafe endpoint. Avoid using them on screens containing sensitive data unless that disclosure is acceptable.