Inworld Conversational Assistant

May 6, 2026 · View on GitHub

An emotionally aware voice assistant. The agent watches the user's face in real time and steers its Inworld TTS v2 delivery to match — whispering when the user looks sad, getting excited when they're engaged, gently re-engaging when they look away.

Built on Vision Agents with:

  • Inworld TTS v2 with inline steering tags ([whisper], [laugh], [sigh], …)
  • Gemini 3.1 flash-lite as the conversational LLM
  • Deepgram STT
  • Stream edge for the call
  • MediaPipe Face Landmarker running locally as a VideoProcessor — emits smoothed emotion, gaze, engagement derived from blendshapes + head pose.

How it works

  1. A MediaPipeFaceProcessor consumes the user's video track at 8 fps. Each frame goes through MediaPipe Face Landmarker to get 52 blendshape scores and a 4×4 head-pose matrix. Categorical state (happy / sad / surprised …, at_camera / off_left / up …) is derived with thresholded combinations and EMA-smoothed to suppress flicker.
  2. FacialAwareAgent overrides simple_response. Before every user-turn LLM call, it prepends a brief state tag — e.g. [user state: looking down, sad] I'm fine.
  3. The agent's audio guide teaches Gemini to react to that tag with appropriate Inworld steering: [whisper] and gentle content for sad, [speak with excitement] for engaged-and-happy, etc.
  4. Inworld TTS v2 renders the steered reply.

Setup

cp .env.example .env       # fill in your keys
uv sync
uv run python scripts/download_face_model.py

Run as the Frontend Backend

uv run python main.py serve --host 127.0.0.1 --port 8000

The Next.js frontend calls this server to create, close, and inspect agent sessions.

Run Standalone

uv run python main.py run

The browser opens a Stream demo room — join, allow camera, and talk to the agent.

Why local editable paths

The pyproject.toml pins vision-agents and its plugins to local paths (../../../vision-agents/…) because this demo depends on local-only changes to the Inworld plugin (v2 model + LINEAR16 streaming) that are not yet released. Once those land upstream, switch the sources to git URLs.