README.md

Cognitive-Physical Decoupling — A Session-Centered Runtime for Embodied Intelligence

_{English · 中文}

📢 Changelog

Version	Date	Update
	2026-07-5	Supports benchmarking for both the policy loop and target native builtin paths; Incorporates agent verification and failure recovery server
	2026-06-27	Support for Behavior 1K; SessionVerfier for Agent verification; VerifySessionTool
	2026-06-11	Cleaned protocol files and docs; game scenario separated to general-game-agent branch; main branch now focused on sim & real
	2026-06-5	Optimize the user-friendly onboarding process; Communication Protocol Specification; More reasonable coding standards; Game Agent & Benchmarking ready
	2026-05-25	Strict separation of PolicySkillRuntime / BuiltinSkillRuntime; Game Agent & Benchmarking ready
	2026-05-20	Perception plugin system: SensorConfig / PerceptionConfig YAML + EnvironmentWriter auditable writeback
	2026-05-18	Session-Centered Runtime MVP: DummySimTarget + DummyAdapter + DummyClient serial pipeline
	2026-04-29	Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline

🤔 Why PhyAgentOS?

Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:

_{▲ Session-Centered Runtime Architecture Overview}

✨ Core Features

🚀 5-Minute Quick Start

git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS
pip install -e .            # Python ≥ 3.11
pip install -e ".[dev]"     # Dev dependencies

paos onboard

paos agent    # CLI chat
paos tui      # full-screen TUI

# Terminal 1: LIBERO TargetWS
MUJOCO_GL=egl PYTHONWARNINGS=ignore \
conda run --no-capture-output -n libero \
  python PhyAgentOS/runtime/targets/remote/libero/server.py \
  --host 0.0.0.0 --port 9002 \
  --camera-height 256 --camera-width 256 \
  --max-steps 300 --num-steps-wait 10 \
  --control-mode relative --seed 7

# Terminal 2: official OpenPI PI0.5 policy server
conda run --no-capture-output -n openpi \
  python -m PhyAgentOS.runtime.policy.openpi.native_openpi_server \
  --policy-config pi05_libero \
  --checkpoint-dir gs://openpi-assets/checkpoints/pi05_libero \
  --host 0.0.0.0 --port 8000

# Terminal 3: Agent, Watchdog, and Verification Service
paos agent --workspace ~/.PhyAgentOS/workspace -m \
  "Evaluate PI0.5 on LIBERO suite libero_spatial. Use target libero_real_remote, the libero_target_benchmark builtin runtime, target_native execution, recovery verification, task ids 0-9, init-state ids 0-49, max_steps 300, replan_every_steps 5, and policy endpoint openpi://127.0.0.1:8000."

Before starting Terminal 3, enable libero_real_remote, configure Runtime verification, and set the Target configuration to retry_instruction_mode: original. The latter keeps the original task instruction on recovery attempts; use verifier_rewrite when the policy should receive the verifier's nonempty rewrite instead. See the Runtime configuration reference for the complete global, Target, SkillRuntime, Session, benchmark, verification, and remote-host parameter reference. The LIBERO setup is included as an example.

🗂️ Protocol Files

SKILLS.md is for agent capabilities and skill discovery. SKILLRUNTIME.md is for runtime execution contracts; it is paired with TARGETS.md and SESSIONS.md.

📦 Project Structure

PhyAgentOS/
│
├── PhyAgentOS/agent/          # Track A  ─  Planner / Critic / Memory
│
├── PhyAgentOS/runtime/        # Track B  ─  Execution Plane
│   ├── watchdog/              #   WatchdogSupervisor
│   ├── sessions/              #   SessionRunner / TargetSessionHandle
│   ├── targets/               #   RolloutTarget (debug·sim·real)
│   │   └── remote/libero/     #   LIBERO benchmark TargetWS server + proxy
│   ├── skillruntime/          #   PolicySkillRuntime / BuiltinSkillRuntime
│   ├── adapters/              #   TargetAdapter / PolicyAdapter / Bridge
│   │   ├── libero/            #   LIBERO target adapter
│   │   └── openpi/            #   OpenPI policy adapters
│   ├── policy/openpi/         #   OpenPI client + LeRobot pi0-family server
│   ├── perception/            #   Perception Runtime / EnvironmentWriter
│   ├── preflight/             #   RuntimeCompatibilityPreflight
│   └── schemas/               #   Pydantic Schema
│
├── configs/runtime/           # Sensor / Perception / Contract YAML
├── scripts/                   # Utility scripts
├── workspace/                 # Agent workspace; runtime files may share it by config
├── docs/                      # Documentation
└── tests/                     # Tests

🏷️ Supported Targets

All targets are registered in TARGETS.md, identified by target_adapter:// URI. More examples & demos → Project Website

📖 Documentation

🤝 Contributing

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory & X-Era Lab

       
_{MIT License · Copyright © 2025-2026 PhyAgentOS}