Phone-Eye ๐ฑ๐๏ธ
August 29, 2026 ยท View on GitHub
English | ็ฎไฝไธญๆ
Your AI agent can finally see and operate a real Android phone.
phone_look("what's on screen, where is the login button?")
โ "Login button at (540, 1830) โ a green 'Sign in' โฆ"
phone_tap(540, 1830)
phone_look("did the next page load?")
Works with any MCP client โ Claude Code, Codex, Cursor, dsh, and friends.
What you need (plain words)
| You provide | One-time or every time? | How hard? |
|---|---|---|
| An Android phone with USB debugging on | one-time, ~60 seconds (tap "Build number" 7ร โ enable USB debugging) | easy, step-by-step below |
| Plug the USB cable once | one-time โ after that the tool switches the phone to Wi-Fi and you never need the cable again (unless the phone factory-resets) | trivial |
| A computer with Python 3.10+ and git (or just Docker โ see step 3) | โ | n/a |
| A vision model | one-time setup โ bring any one of these: โข an OpenAI API key (or any OpenAI-compatible: GLM, DeepSeek, local llama.cpp/Ollamaโฆ) โข an existing MCP vision server | one env var, most people already have a key |
That's everything. No app to install on the phone. No root. No extra server.
What it can and can't do (honest table)
| โ Stable | โ ๏ธ Works but with caveats | โ Not possible (any tool, not just us) |
|---|---|---|
| See the screen (screenshot + read it) | Locked screens can be read but not operated | Fully control a brand-new phone before you enable USB debugging yourself |
| Tap / swipe / type | Typing is ASCII (Chinese input needs a clipboard trick โ known adb limit) | The very first "allow USB debugging?" popup on a new computer key โ that one tap is yours |
| Survive Wi-Fi adb drops (auto-reconnect) | Some vendor ROMs restrict input on lock screens (e.g. MIUI) | iOS โ different universe |
| Run 24/7 unattended; unexpected popups get read & handled by your agent | Vision quality depends on the model you bring | |
Multiple phones (one phone-eye process per phone, set ANDROID_SERIAL for each) |
The 60-second rule: every Android requires one human moment โ enable debugging + authorize once. After that, the phone belongs to your agent, even over Wi-Fi, even after reboots of the computer.
Setup (3 steps)
1. Enable USB debugging (once per phone)
Settings โ About phone โ tap Build number 7ร (unlocks Developer options) โ Developer options โ USB debugging ON. (Got stuck? Tell us your phone model in Discussions โ we'll walk you through it. MIUI/HyperOS may also ask you to sign into a Xiaomi account first.)
2. Install adb (if you don't have it), plug USB once, then go wireless
# macOS: brew install android-platform-tools ยท Ubuntu/Debian: sudo apt install adb
# Windows: scoop install adb (or download Android platform-tools)
adb devices # phone shows up? tap "Allow" on its popup โ check "always allow"
adb shell ip route # โ note the phone's Wi-Fi IP (e.g. 192.168.1.23) while still plugged
adb tcpip 5555 # switch to Wi-Fi mode (adb restarts; the USB entry disappears โ normal)
adb connect 192.168.1.23:5555 # use the IP from above; then unplug the cable, forever
Phone IP alternatives if ip route prints nothing
Settings โ Wi-Fi โ your network โ details shows the IP; or adb shell ip addr show wlan0 | grep inet.
3. Start phone-eye
git clone https://github.com/boheastill/phone-eye && cd phone-eye
pip install -r requirements.txt
# your eyes โ pick ONE:
export PHONE_EYE_VISION_API_KEY=<key> # OpenAI / GLM / any compatible
# (optional: PHONE_EYE_VISION_BASE_URL, PHONE_EYE_VISION_MODEL)
# local & offline: ..._API_KEY=sk-noauth ..._BASE_URL=http://<host>:8080/v1 ..._MODEL=<your qwen-vl>
python server.py # stdio MCP server โ wire into your client:
Wire it into your client โ pick yours:
# Claude Code (easiest):
claude mcp add phone-eye -- python /path/to/phone-eye/server.py
# Codex:
codex mcp add phone-eye --url stdio://python /path/to/phone-eye/server.py # or see codex docs
// any MCP client (generic stdio shape):
{ "mcpServers": { "phone-eye": { "command": "python", "args": ["/path/to/phone-eye/server.py"] } } }
Docker (optional โ no Python needed on the host)
The repo ships a Dockerfile (Python 3.12 + adb):
podman build -t phone-eye . # or: docker build -t phone-eye .
# smoke: a JSON-RPC initialize reply on stdout means it boots:
printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"smoke","version":"0"}}}\n' \
| podman run --rm -i phone-eye
Use --network host so adb reaches a Wi-Fi phone and your vision endpoint:
"phone-eye": { "command": "podman", "args": ["run","--rm","-i","--network","host",
"-e","ANDROID_SERIAL=192.168.1.23:5555","-e","PHONE_EYE_VISION_API_KEY=<key>","phone-eye"] }
Recommended vision models (any vision-capable chat model works): gpt-4o-mini (default),
GLM glm-4.6v-flash (cheap), or run a local Qwen-VL via llama.cpp for fully-offline โ
screenshots then never leave your LAN.
What happens when something breaks
- "No Android device reachable" โ the tool already tried reconnecting; run
adb connect <ip>:5555, or replug USB. - "No vision server reachable" โ you haven't set a key; the error message tells you the exact two fixes.
- Phone rebooted โ Wi-Fi adb survives phone reboots on most ROMs; if not, one
adb connectagain. - Still stuck? Open a discussion โ we answer, and we'll debug your setup with you. Bug reports and "it works on my X" notes are equally welcome.
Tools
| Tool | What it does |
|---|---|
phone_look(question?) | Ask a vision model about the live screen; fuses a UI-tree dump for exact text/button bounds |
phone_tap(x, y) | Tap |
phone_swipe(x1, y1, x2, y2, ms?) | Swipe |
phone_type(text) | Type ASCII text |
phone_key(key) | Press a hardware key โ wake revives a sleeping phone (the unattended essential), back/home/recents navigate |
phone_intent(action, uri?, component?) | Open any screen by Android intent (deep settings pages, app pages) without coordinates |
phone_screenshot() | Save screenshot to disk, return path |
Examples
- Mobile web QA loop โ the agent verifies its own work on a real screen
- Surviving an OEM setup wizard โ vision handles whatever pops up
- Form regression check
- Unattended sentinel โ your agent on night watch: wake โ unlock โ intent โ look โ screenshot
Curious how it works โ or want to modify it? Read the whitepaper (architecture, failure-classification decision tree, security model, how to add a verb). Running it as an always-on HTTP service behind your own fleet? docs/fleet.md.
Why "see", isn't this just adb?
UI-tree-only tools are blind to game canvases, images, and anything the accessibility tree can't show. Vision-only tools drift on coordinates. phone_look fuses both: the model answers what is this, the UI tree supplies exactly where. The day we dogfooded it, it discovered a USB-debugging popup on its own screen, read the buttons, and tapped "Allow" by itself โ the story.
License
MIT. Verified on Redmi K40 Gaming / Android 13 โ add your device to the table via PR.