README.md

June 11, 2026 · View on GitHub

Cognitive-Physical Decoupling — A Session-Centered Runtime for Embodied Intelligence

📢 Changelog

Version	Date	Update
	2026-06-11	Cleaned protocol files and docs; game scenario separated to `general-game-agent` branch; main branch now focused on sim & real
	2026-06-5	Optimize the user-friendly onboarding process; Communication Protocol Specification; More reasonable coding standards; Game Agent & Benchmarking ready
	2026-05-25	Strict separation of `PolicySkillRuntime` / `BuiltinSkillRuntime`; Game Agent & Benchmarking ready
	2026-05-20	Perception plugin system: `SensorConfig` / `PerceptionConfig` YAML + `EnvironmentWriter` auditable writeback
	2026-05-18	Session-Centered Runtime MVP: `DummySimTarget` + `DummyAdapter` + `DummyClient` serial pipeline
	2026-04-29	Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline

Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:

🔌	One Codebase, Any Hardware — Adding a new robot means implementing one Target Adapter (~100 lines); zero changes to the scheduling layer.
🛡️	Three Safety Layers — Critic validation → Strict Preflight → Target-side SafetyGuard; mandatory for real-robot deployment.
📋	Fully Auditable — State, actions, and perception results are written to Markdown + YAML files; every step is traceable and reproducible.
🔄	Zero-Friction Migration — The same Session protocol runs identically across sim and real targets.

_{▲ Session-Centered Runtime Architecture Overview}

✨ Core Features

🔄	Session-Centered Runtime	`WatchdogSupervisor` → `SessionRunner` → `SkillRuntime` → `TargetSessionHandle` execution pipeline, replacing the legacy Driver-Center architecture
🎯	Target-Configured	Three target kinds — `debug` / `simulation` / `real_robot` — registered in `TARGETS.md`, adapters attached on demand
🧩	Adapter + Bridge	`TargetAdapter` + `PolicyAdapter` + `ActionBridge` three-way decoupling with explicit observation/action contracts; `AdapterPlan` auto-composed, eliminating target×skill combinatorial explosion
⚡	Dual Skill Runtimes	`PolicySkillRuntime` maintains policy closed-loop + `BuiltinSkillRuntime` manages agent interactive loop
🛡️	Strict Preflight	Runtime validation checks (target / sensor / perception / adapter contract / action contract / tool); failures are `rejected` before execution starts
📝	File Protocol Matrix	`TARGETS.md` · `SKILLRUNTIME.md` · `SESSIONS.md` · `ENVIRONMENT.md` · `LESSONS.md` + external YAML configs
🔐	Multi-Layer Safety	Critic validation → Preflight contract checks → Target-side SafetyGuard → Operator Override
🌐	Fleet Mode	Multi-robot coordination with shared + per-robot workspaces, priority-based serial scheduling

🚀 5-Minute Quick Start

1	Install `git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS pip install -e . # Python ≥ 3.11 pip install -e ".[dev]" # Dev dependencies`
2	Initialize Workspace `paos onboard`
3	Start Agent `paos agent`
4	Optional: Connect Runtime Services `# LIBERO benchmark TargetWS machine MUJOCO_GL=egl PYTHONWARNINGS=ignore \ conda run -n liberopi python PhyAgentOS/runtime/targets/remote/libero/server.py \ --host 0.0.0.0 --port 9002 # pi0.5 policy machine conda run -n lerobot-pi python -m PhyAgentOS.runtime.policy.openpi.lerobot_pi0_server \ --model-dir /path/to/pi05/checkpoint --host 0.0.0.0 --port 8000`

paos agent and paos gateway create the runtime workspace and start the session watchdog automatically when runtime is enabled in config. Runtime targets are declared in TARGETS.md, executable runtimes in SKILLRUNTIME.md, and the Agent queues work by appending sessions to SESSIONS.md.

paos agent -m "run the configured LIBERO benchmark task"

🗂️ Protocol Files

Context Loading	File	Owner	Purpose
Always loaded into the agent system prompt	`AGENTS.md`	Agent workspace	Project-level operating rules for the agent
Always loaded into the agent system prompt	`SOUL.md`	Agent workspace	Identity, high-level behavior, and assistant style
Always loaded into the agent system prompt	`USER.md`	Agent workspace	User preferences and durable profile notes
Always loaded into the agent system prompt	`TOOLS.md`	Agent workspace	Tool usage policy and available tool guidance
Always loaded into the agent system prompt	`SKILLS.md`	Agent workspace	Agent-facing skill discovery and loading rules
Loaded when present; filtered by enabled runtime targets where applicable	`EMBODIED.md`	Agent workspace	Human-readable target capability descriptions
Loaded when present as state, not bootstrap policy	`ENVIRONMENT.md`	Agent/runtime workspace	Current target and scene/environment state
Loaded when present as memory/state	`LESSONS.md`	Agent workspace	Operational lessons and failure notes
Loaded when present as task state	`TASK.md`	Agent workspace	Multi-step task decomposition and progress
Runtime protocol; read before scheduling sessions	`RUNTIME.md`	Runtime workspace	Instructions for writing valid runtime sessions
Runtime protocol; read before scheduling sessions	`TARGETS.md`	Runtime workspace	Enabled targets, endpoint/adapter/config references, supported skill runtimes
Runtime protocol; read before scheduling sessions	`SKILLRUNTIME.md`	Runtime workspace	Policy/builtin skill runtime registry and execution contracts
Runtime queue/state; written by Agent and watchdog	`SESSIONS.md`	Runtime workspace	Pending/running/completed execution sessions and results

SKILLS.md is for agent capabilities and skill discovery. SKILLRUNTIME.md is for runtime execution contracts; it is paired with TARGETS.md and SESSIONS.md.

📦 Project Structure

PhyAgentOS/
│
├── PhyAgentOS/agent/          # Track A  ─  Planner / Critic / Memory
│
├── PhyAgentOS/runtime/        # Track B  ─  Execution Plane
│   ├── watchdog/              #   WatchdogSupervisor
│   ├── sessions/              #   SessionRunner / TargetSessionHandle
│   ├── targets/               #   RolloutTarget (debug·sim·real)
│   │   └── remote/libero/     #   LIBERO benchmark TargetWS server + proxy
│   ├── skillruntime/          #   PolicySkillRuntime / BuiltinSkillRuntime
│   ├── adapters/              #   TargetAdapter / PolicyAdapter / Bridge
│   │   ├── libero/            #   LIBERO target adapter
│   │   └── openpi/            #   OpenPI policy adapters
│   ├── policy/openpi/         #   OpenPI client + LeRobot pi0-family server
│   ├── perception/            #   Perception Runtime / EnvironmentWriter
│   ├── preflight/             #   RuntimeCompatibilityPreflight
│   └── schemas/               #   Pydantic Schema
│
├── configs/runtime/           # Sensor / Perception / Contract YAML
├── scripts/                   # Utility scripts
├── workspace/                 # Agent workspace; runtime files may share it by config
├── docs/                      # Documentation
└── tests/                     # Tests

🏷️ Supported Targets

	Kind	Location	Examples
🐛	`debug`	Local	echo / mock / dry-run — zero-hardware protocol pipeline validation
🧪	`simulation`	Remote	RoboCasa, LIBERO — benchmark evaluation & batch experience mining
🤖	`real_robot`	Remote	Franka, Go2, XLeRobot, AgileX PIPER — real-world deployment

All targets are registered in TARGETS.md, identified by target_adapter:// URI. More examples & demos → Project Website

📖 Documentation

Document	Audience	Description
🌐 Website	Everyone	Full docs, architecture details, demos
📘 User Manual	Users	Installation, deployment, and operation guide
📙 Dev Guide	Developers	Secondary development, hardware integration, plugin authoring

🤝 Contributing

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

Built on nanobot

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory