Interaction Loop

February 19, 2026 · View on GitHub

✅ Implemented 🧪 Tested

Current state: The full interaction loop (observe → execute → observe) is implemented with gfxinfo-based idle detection. See the Status Glossary for chip definitions.

Interaction loop demo - setting an alarm

This interaction loop is supported by comprehensive observation of UI state and UI stability checks (Android uses dumpsys gfxinfo-based idle detection) before and after action execution. Together, that allows for accurate and precise exploration with the action tool calls.

sequenceDiagram
    participant Agent as AI Agent
    participant MCP as MCP Server
    participant Device as Device

    Agent->>MCP: 🤖 Interaction Request
    MCP->>Device: 👀 Observe
    Device-->>MCP: 📱 UI State/Data (Cached)

    MCP->>Device: ⚡ Execute Actions
    Device-->>MCP: ✅ Result

    MCP->>Device: 👀 Observe
    Device-->>MCP: 📱 UI State/Data
    MCP-->>Agent: 🔄 Interaction Response with UI State