Jev Mac Voice

English voice control for your Mac, built with OpenAI Realtime API, a native Swift desktop bridge, and optional Jev + Playwright browser automation.

Talk naturally, ask for an action, and interrupt while the assistant is speaking. Microphone input and generated audio use simultaneous WebRTC media tracks. This is speech-to-speech Realtime, not browser speech recognition followed by text-to-speech. Semantic VAD detects turns and cancels the assistant response when you interrupt; interruption also cancels pending local work.

Quick start

Requires macOS 14+, Node.js 22.9+, Xcode Command Line Tools (xcode-select --install), and an OpenAI API key with Realtime access and API billing.

git clone https://github.com/brudarko/jev-mac-voice.git
cd jev-mac-voice
npm ci
npm run build:native
cp .env.example .env
# Add OPENAI_API_KEY to .env using your editor.
npm run desktop

The Electron application opens with the React voice interface and a live activity panel. In the app, Settings lets you save API keys encrypted with macOS secure storage. For the browser-only version, run npm start and open the private localhost URL printed in your terminal. The URL contains a per-run access token: keep it private. The app removes it from the address bar and retains it in session storage. API keys stay in the local backend. In packaged builds they are encrypted with Electron safeStorage using macOS Keychain integration.

1. Grant Jev Mac Voice (or the launching terminal/native helper when running the web version) Accessibility access in System Settings → Privacy & Security. The app has a request button. Quit and reopen after granting access.
2. For screenshots, grant Screen & System Audio Recording / Screen Recording access as well. Captures are off until you enable screen sharing in the app.
3. Start the conversation and grant the app or browser microphone access.
4. Enable Allow computer control when you want actions. Review actions first is on by default; approve each proposed input in the local UI. You may turn review off for immediate execution of requested inputs.

Try: “Open Finder,” “List my running apps,” “What buttons are in this window?” or “Type hello into this field.” Use headphones if speakers cause echo.

Build a macOS app

npm run package:mac

The unpacked application is generated under dist/mac-arm64/Jev Mac Voice.app on Apple Silicon (or dist/mac/ on Intel). This local POC build has a verified ad-hoc signature and is not notarized; public distribution requires an Apple Developer signing identity. Replacing a local build can invalidate macOS permission grants: remove the old permission entry, add the installed app again, and restart it. Keys are never bundled in the application. Configure them in Settings on first launch. The desktop app runs its own loopback service on a free port and shuts it down when closed.

The native menu shortcut Command + Shift + . stops actions.

What it controls

• Native apps: launch/focus by bundle ID, inspect foreground accessibility elements, press observed buttons, type text, send keyboard shortcuts, scroll, and click primary-display coordinates.
• Screen context: on-demand primary-display screenshots, resized for the Realtime data channel. No continuous screen recording.
• Browser: optional browser_command delegates a single English instruction to Jev, then Playwright executes in a separate Chromium profile.
• Conversation: English output, marin voice by default, live transcript, microphone mute, interruption, typed input, and emergency stop.

The native executor needs a fresh foreground snapshot for every input and invalidates it after each mutation. When local review takes focus, an approved action restores the app identified by its snapshot. Tool calls are serialized and deduplicated. Errors return to the conversation; the model must inspect to verify effects.

Configuration

Variable	Default	Purpose
OPENAI_API_KEY	Required	OpenAI API credential, never sent to the UI
OPENAI_REALTIME_MODEL	gpt-realtime-2.1	Realtime model available to your project
OPENAI_REALTIME_VOICE	marin	Realtime output voice
PORT	8787	Loopback-only server port
TYPESAFE_API_KEY	Optional	Jev browser decisions
TYPESAFE_MODEL	jev-latest	Optional Jev model override

For optional browser automation:

npx playwright install chromium
# Add TYPESAFE_API_KEY to .env.

Jev requests go directly to https://api.typesafe.ai/v1/systemone. JEV_API_KEY is accepted as an alias for TYPESAFE_API_KEY. npm run start:browser runs the original browser-only UI. See legacy browser documentation.

Local control and privacy

The server binds only to 127.0.0.1. API routes require a random launch token; Host and Origin checks reject other websites and DNS rebinding attempts. There is no arbitrary shell or AppleScript execution tool. Desktop access starts disabled. Disabling access, ending the voice session, interruption, or Stop cancels queued work and pending approvals. Inputs already dispatched to macOS cannot be undone by Stop.

Audio goes to OpenAI. Requested accessibility context and opted-in screenshots also go to OpenAI. Optional browser context goes to the configured Jev provider. Secure accessibility fields are skipped, but other visible content can contain private information. Do not enable context sharing around secrets. No audio, screenshots, or conversation transcripts are saved by this application; recent action activity remains in server memory. Provider retention policies apply separately.

Review mode is a local approval boundary. Turning it off permits clicks, text and shortcuts that can have side effects, including inside terminals. Model instructions ask for confirmation on sensitive actions, but those instructions are not a security boundary. Keep review enabled for sensitive work. This is a local single-user development application; do not expose its port to a network.

Development and validation

npm run build:native
npm test
npm run test:ui # requires Playwright Chromium; uses simulated voice, not live API
npm run build:ui
npm run test:react
npm run test:electron

Tests cover input validation, control/screen-sharing gates, approval, cancellation, duplicate execution, server authentication/origin checks, and the Realtime session contract, alongside upstream browser policy tests. CI compiles the native helper on macOS.

Live audio requires a configured API key, microphone, and manual listening test. Native input testing requires macOS Accessibility permission. Default automated tests do not send live API requests or operate your desktop.

Known limits: primary-monitor pointer/screenshot support; app-dependent accessibility trees; fixed supported keyboard keys; no guaranteed automation of every app or custom canvas. Multiple-step tasks are sequences of tool calls, not transactions. Screenshot downsampling can hide small text. The app does not hook into an existing ChatGPT conversation; it creates its own Realtime API session.

Architecture: src/mac-server.js owns authentication and session setup; src/desktop/ owns tool schemas and execution; native/DesktopBridge.swift handles macOS APIs; renderer/ contains the React voice interface, electron/ owns the native window and secure credential storage, and src/public/mac/ retains the original web UI. Original Jev/Playwright modules remain available.

Attribution

Based on moritzkremb/jev-voice-browser, copyright 2026 Moritz Kremb, under the MIT license. The original copyright and permission notice are retained in LICENSE.

OpenAI integration references: Realtime conversations and WebRTC.

To run the paid, bounded WebRTC smoke test explicitly: RUN_LIVE_SMOKE=1 node --env-file=.env test/realtime-live.mjs. It uses a fake microphone, requests one short response (128 output tokens maximum), and closes the connection automatically. It never operates the desktop.

For a real desktop demonstration, build the app and configure its credentials, then run RUN_LIVE_DESKTOP_SMOKE=1 node test/electron-live.mjs. This paid, opt-in test sends a typed instruction through the React interface, approves opening Finder, receives an English audio response, and closes the session. It uses a silent fake microphone.