PhoneWorld: Scaling Phone-Use Agent Environments

Paper: arXiv 2605.29486

An automated pipeline that generates scalable, reproducible mobile agent evaluation environments from real-world app GUI trajectories.

Note on APK distribution: Gated APK packages are provided as an external research distribution. To make the research-only nature clear and avoid confusion with real apps, app launcher labels may be prefixed with "Mock" (e.g., "Mock京东" instead of "京东"). App package names, task definitions, task verification, and core interactions are preserved; the runner resolves original names, Mock/PW labels, aliases, and package-name variants automatically.

Overview

PhoneWorld automatically constructs mock Android apps with:

• Realistic UI (Kotlin/Jetpack Compose)
• Pre-filled seed data (products, users, reviews, etc.)
• Local SQLite database for verifiable write operations
• Offline search engine (BM25 with Chinese tokenization)
• Rule-based automatic task verification

Scale: 34 apps, 16 fine-grained domains (grouped below into 10 app categories), 120 benchmark tasks, and 300 public verified training tasks.

Mocked APK Resource: EthanLeoLYX/PhoneWorld-APKs

Downstream Model: PhoneBuddy

PhoneBuddy is an open phone-use model line trained with real-app RL and PhoneWorld mock-app RL. PhoneWorld provides scalable, resettable, automatically verified interaction signals that complement real-app reinforcement learning.

• Project: https://phonebuddyai.github.io/
• Paper: https://arxiv.org/abs/2606.23049
• Code: https://github.com/PhoneBuddyAI/phonebuddy
• Models: PhoneBuddy-4B Real+Mock, PhoneBuddy-4B-RealApp, PhoneBuddy-0.8B Real+Mock

Getting Started

Option A: One-command quickstart

This is the fastest path once an Android emulator/device is running. It installs Python dependencies, downloads the gated APKs from Hugging Face, installs the apps, and runs one smoke-test task if an API key is available.

# 1) Enter the PhoneWorld repo
cd PhoneWorld

# 2) Login once for the Hugging Face gated APK dataset
pip install -U huggingface_hub
hf auth login

# 3) Start your Android emulator in another terminal, then run:
PHONEWORLD_HF_REPO=EthanLeoLYX/PhoneWorld-APKs \
OPENAI_API_KEY=$OPENAI_API_KEY \
bash scripts/quickstart.sh

If you only want to install APKs and skip model inference, omit OPENAI_API_KEY.

Option B: Manual setup

Step 1: Setup emulator

# Create Android emulator (API 33, Pixel 6 — same as AndroidWorld)
avdmanager create avd -n PhoneWorldAvd -k "system-images;android-33;google_apis;arm64-v8a" -d pixel_6
emulator -avd PhoneWorldAvd -no-snapshot

Step 2: Install dependencies and download APKs

The 34 mock app APKs are hosted as a Hugging Face gated dataset. Open the dataset page, accept the research terms, then download them into apps/:

pip install -r requirements.txt
hf auth login
PHONEWORLD_HF_REPO=EthanLeoLYX/PhoneWorld-APKs python scripts/download_apks.py

If you already have the APK package, place the 34 *.apk files in apps/.

APK access is gated rather than email-based: users accept the Hugging Face dataset terms once, then download programmatically. The APK terms require non-commercial academic use, citation of the PhoneWorld paper, no APK redistribution, and acknowledgement that the apps are research mock environments not affiliated with any real app company.

Step 3: Install apps

# Install all 34 mock apps + ADBKeyboard (required for Chinese input)
bash scripts/install_all.sh

Step 4: Run evaluation

# Single task
python run_task.py \
    --task tasks/mjd/mjd_v3_001.json \
    --model gpt-4o \
    --base-url https://api.openai.com/v1 \
    --api-key $OPENAI_API_KEY

# Full benchmark (120 tasks)
python run_benchmark.py --task-dir tasks/ --output-dir results/

# Multi-device parallel
python run_benchmark.py --task-dir tasks/ --output-dir results/ --devices 6

Two Task Sets

Set	Path	Count	Purpose
Benchmark	tasks/	120	Official evaluation — results comparable to paper
Training	tasks_train/	300	Rollout / SFT / RL — all verified passable by an agent

• Report results on tasks/ (120 tasks) for fair comparison with published baselines
• Use tasks_train/ (300 public verified tasks) to collect trajectories and train your agent; the paper also studies larger internally generated rollout pools from the same environment pipeline

Task Format

{
    "task_id": "mjd_v3_001",
    "app": "mjd",
    "difficulty": "easy",
    "max_steps": 30,
    "goal": "Search for 'Samsung Galaxy S24 Ultra' in JD app and tell me its price.",
    "verification": {
        "type": "answer_contains",
        "keywords": ["8999"],
        "min_length": 3
    }
}

Verification Types

• answer_contains: Agent's final answer must contain specified keywords
• sqlite: App's SQLite database must contain a matching record

Benchmark Statistics

Metric	Value
Mock Apps	34 (16 fine-grained domains; grouped below into 10 app categories)
Benchmark Tasks	120
Public Training Tasks	300 verified tasks in tasks_train/
Cross-App Tasks	18
Difficulty	Easy 37% / Medium 34% / Hard 29%
Verification	41% answer_contains / 59% sqlite

App Domains

Domain	Apps
E-commerce	Taobao, JD, Dewu, Xianyu
Food Delivery	Meituan Waimai, Eleme, Hema, KFC, McDonalds
Social/Video	Douyin, Bilibili, Xiaohongshu, Weibo, QQ
Travel	Didi, Gaode Maps, 12306, Ctrip
Music	NetEase Music, QQ Music, Ximalaya
Video Streaming	iQiyi, Tencent Video
Reading	Fanqie Novel, WeRead, Douban
Life Services	Meituan, Dianping, Beike
Finance	Alipay, Tonghuashun
Fitness/Knowledge	Zhihu, Toutiao, Keep

Baseline Results

Model	Easy	Medium	Hard	Overall
Seed 2.0 Pro	90.9%	80.5%	65.7%	80.0%
GPT-4o	—	—	—	—
Gemini 2.5 Pro	—	—	—	—

We welcome community contributions of baseline results.

Source Edition (App Source Code)

The apps_source/ directory contains the PhoneWorld Research Source Edition — full Kotlin/Jetpack Compose source code for all 34 mock apps. Use it to:

• View and modify synthetic seed data (seed.json)
• Extend SQLite database schemas
• Add new mutable state and write operations
• Create new tasks with custom verifiers
• Compile your own research APKs
• Collect custom rollouts for SFT/RL

The Source Edition is not the bit-for-bit source of the official gated APK release. It may differ in launcher labels, research notices, and non-essential presentation. Where possible, it preserves app identifiers, package names, database schemas, and task/verifier compatibility.

For paper-comparable benchmark results, use the versioned gated APK release. APKs built from apps_source/ (especially after modification) are custom research environments and must not be reported as official PhoneWorld benchmark scores.

See apps_source/README.md for build instructions and customization guide.

Project Structure

PhoneWorld/
├── README.md                # This file
├── LICENSE                  # Research-only license
├── TRADEMARKS.md            # Third-party trademark notice
├── TAKEDOWN.md              # Takedown request process
├── requirements.txt
├── run_task.py              # Single task runner
├── run_benchmark.py         # Batch benchmark runner
├── model_adapters.py        # Model API adapters
├── apps/                    # 34 APK files (created by scripts/download_apks.py)
├── apps_source/             # 34 app source code (Research Source Edition)
├── tasks/                   # 120 benchmark tasks (official test set)
│   ├── {app_id}/            # Per-app tasks (3 each)
│   └── cross_app/           # 18 cross-app tasks
├── tasks_train/             # 300 training tasks (verified passable)
└── scripts/
    ├── download_apks.py     # Download gated APKs from Hugging Face
    ├── quickstart.sh        # One-command install + smoke test
    └── install_all.sh       # APK installer

Custom Model Integration

Implement a model adapter in model_adapters.py:

MY_ADAPTER = {
    "system_prompt": "...",
    "tool_prompt": "...",
    "parse_action": my_parse_function,
    "use_tool_role": False,
    "default_params": {"temperature": 0.7},
}

The runner uses OpenAI-compatible chat API. Any model with a /v1/chat/completions endpoint works.

Citation

@misc{tang2026phoneworldscalingphoneuseagent,
  title={PhoneWorld: Scaling Phone-Use Agent Environments},
  author={Yuxuan Liu and Xin Lai and Junyi Li and Pengyuan Lyu and Jason and Yiduo Guo and Zhengyao Fang and Yang Ding and Yi Zhang and Weinong Wang and Huawen Shen and Xingran Zhou and Liang Wu and Fei Tang and Sunqi Fan and Shangpin Peng and Zheng Ruan and Anran Zhang and Chengquan Zhang and Han Hu and Benyou Wang and Ji-Rong Wen and Rui Yan and Zhengyang Tang},
  year={2026},
  eprint={2605.29486},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2605.29486}
}

APK Artifact Provenance

The PhoneWorld APK artifacts are academic research resources independently developed and maintained by authors Yuxuan Liu, using personal research resources. They are shared for non-commercial academic research as mock Android environments.

For the public release, we keep this provenance explicit while still making the benchmark easy to run: code and tasks are public in this repo, and APKs are provided through a Hugging Face gated dataset so researchers can accept the terms and download them programmatically.

Disclaimer

PhoneWorld is an academic research project for evaluating mobile GUI agents. The APK artifacts are personal academic research resources and the mock apps are simplified simulations created solely for research benchmarking purposes. They are:

• NOT affiliated with, endorsed by, or connected to any of the real apps or companies referenced
• NOT intended for consumer use or distribution through any app store
• NOT designed to replicate proprietary functionality, collect user data, or compete with real apps

All app names and brand references are used purely for academic identification of the interaction patterns being evaluated. The mock apps use distinct package names (com.phoneuse.*), contain only synthetic data, and function exclusively within controlled research environments (Android emulators).

If any rights holder has concerns, please see TAKEDOWN.md for the process.

License

This project is released under a research-only license. See LICENSE for details.

• Code (scripts, runner): Free to use and modify for research
• Tasks (JSON): Free to use and cite for research
• App source (apps_source/): Available for modification, compilation, and non-commercial academic forks under the PhoneWorld Research License. See apps_source/README.md for usage and compatibility notes.
• APKs: Available through the Hugging Face gated dataset EthanLeoLYX/PhoneWorld-APKs for non-commercial academic research after accepting the dataset terms. Do not redistribute APK files; direct users to the gated dataset page instead.
• Commercial use: Contact authors for licensing

See also: TRADEMARKS.md | TAKEDOWN.md