Client ↔ Policy Server Protocol

The eval client and the galaxea-fm policy server are fully decoupled — they share no Python imports, only this websocket + msgpack contract. Keep this file in sync with the copy in the server repo (galaxeafm/experiments/droid/PROTOCOL.md).

Transport:  ws://<POLICY_HOST>:<POLICY_PORT>   (msgpack-encoded frames)

Handshake:  on connect, server pushes one metadata frame {action_steps: <int>}
            (recommended: include `protocol_version`; client validates on startup)

Inference:  client → {                                  # send {} instead when last need_obs=false
              images: {
                exterior_image:    uint8 [3, H, W],
                wrist_image:       uint8 [3, H, W],
                dummy_wrist_right: uint8 [3, 224, 224]  # placeholder for bimanual schema
              },
              state: {
                right_arm:     float32 [7],             # joint_position (rad)
                right_gripper: float32 [1]              # = 1 - gripper_position
              },
              task:            <str>,                   # instruction
              frequency:       <int>,                   # control Hz (= 15)
              embodiment_type: <str>                    # e.g. "Droid_Franka"
            }
            server → {
              action: {                                 # singular; client reads right_*
                right_arm:     float [7],               # joint angles (rad); alias joint_position
                right_gripper: float [1]                # gripper; alias gripper
              },                                        # may also carry left_gripper etc.
              need_obs: <bool>,                         # if false → send {} next step (chunk cache)
              cot_text: <str>                           # optional CoT (bbox / subtask / action)
            }
                   |  { error: { code, message } }

Reset:      client → {"__reset__": true}  →  server → {"__reset__": true}

Notes

• Only predicted keys are returned. The action dict contains just the keys the model confidently predicted. When the AR head drops a key (e.g. right_gripper on some chunks) that key is absent from the dict — the server does not fill it and sends no absent_keys metadata. The client owns its expected key set and handles a missing key itself: eval/main.py::_galaxea_action_to_droid holds the current gripper position when right_gripper is absent.
• Action space is joint_position (g05/galaxea-fm output is fixed); the client must run with --action_space joint_position.
• Chunk caching: the server returns an action chunk; need_obs=false means the next step should send an empty {} (server replays the cached chunk).
• The client builds this dict in eval/main.py::_build_galaxea_raw_obs; it has its own self-contained GalaxeaPolicyClient (no dependency on the server package).