Environment Input/Output

On RoboMME, a key difference from traditional Gym-like envs is that every observation value is a list rather than a single item. This is because some RoboMME tasks use conditioning video input, and for discrete action types (e.g. waypoint or multi_choice) we also return intermediate observations for potential use with video-based policy models.

Env Input Format

We support four ACTION_SPACE types:

• joint_angle: 7 joint angles + gripper open/close
• ee_pose: 3D position (xyz) + 3D rotation (rpy) + gripper open/close
• waypoint: Same format as ee_pose, but executed in discrete keyframe steps
• multi_choice: Command dict, e.g. {"choice": "A", "point": [y, x]}. The available choices can be found in info["available_multi_choices"], where point is the pixel location on the front image. This action is designed for Video-QA research.

Note: A closed gripper is -1, and an open gripper is 1. We use absolute actions in our simulator.

Env Output Format

When calling the step function:

obs, reward, terminated, truncated, info = env.step(action)

Return	Description	Typical type
obs	Observation dict	dict[str, list]
info	Info dict	dict[str, Any]
reward	Reward value (not used)	scalar tensor
terminated	Termination flag	scalar boolean tensor
truncated	Truncation flag	scalar boolean tensor

obs dict

Key	Meaning	Typical content
maniskill_obs	The original raw env observation from ManiSkill	Raw observation dict
front_rgb_list	Front camera RGB List	Image frames, e.g. (H, W, 3)
wrist_rgb_list	Wrist camera RGB List	Image frames, e.g. (H, W, 3)
front_depth_list	Front camera depth List	Depth map, e.g. (H, W, 1)
wrist_depth_list	Wrist camera depth List	Depth map, e.g. (H, W, 1)
eef_state_list	End-effector state List	[x, y, z, roll, pitch, yaw]
joint_state_list	Robot joint state List	Joint vector, often 7-D
gripper_state_list	Robot gripper state List	2-D
front_camera_extrinsic_list	Front camera extrinsic List	Camera extrinsic matrix
wrist_camera_extrinsic_list	Wrist camera extrinsic List	Camera extrinsic matrix

To use only the current (latest) observation, use obs[key][-1].

Optional field switches (include_*)

BenchmarkEnvBuilder.make_env_for_episode(...) controls optional observation/info fields through include_* flags.

Default behavior:

• All include_* flags default to False.
• Without extra flags, the env returns only RGB and state-related fields.

Mapping:

Flag	Added key
include_maniskill_obs	obs["maniskill_obs"]
include_front_depth	obs["front_depth_list"]
include_wrist_depth	obs["wrist_depth_list"]
include_front_camera_extrinsic	obs["front_camera_extrinsic_list"]
include_wrist_camera_extrinsic	obs["wrist_camera_extrinsic_list"]
include_available_multi_choices	info["available_multi_choices"]
include_front_camera_intrinsic	info["front_camera_intrinsic"]
include_wrist_camera_intrinsic	info["wrist_camera_intrinsic"]

Special case:

• If action_space="multi_choice", front camera parameters are forced on internally:
  • front_camera_extrinsic_list
  • front_camera_intrinsic Even if the corresponding include_front_camera_* flags are False.

Example:

from robomme.env_record_wrapper import BenchmarkEnvBuilder

builder = BenchmarkEnvBuilder(
    env_id="VideoUnmaskSwap",
    dataset="test",
    action_space="joint_angle",
    gui_render=False,
)

env = builder.make_env_for_episode(
    episode_idx=0,
    max_steps=1000,
    include_maniskill_obs=False,
    include_front_depth=True,
    include_wrist_depth=False,
    include_front_camera_extrinsic=True,
    include_wrist_camera_extrinsic=False,
    include_available_multi_choices=False,
    include_front_camera_intrinsic=True,
    include_wrist_camera_intrinsic=False,
)

obs, info = env.reset()

World coordinate frame & conventions

The ee_pose/ waypoint actions and the eef_state_list observation all use the same world coordinate frame. It is right-handed: +x points forward (from the robot toward the workspace/objects), +y to the robot's left, +z up. The table-top surface is the z = 0 plane (floor/ground at z = -0.92), and the world origin (0, 0, 0) is the table-top center where task objects spawn.

                        +z (up)
                         ^
                         |
   robot base            |          world origin (0, 0, 0)
   panda_link0           |          = table-top center (z = 0)
   (-0.615, 0, 0)        |
        [R] =============O=====================>  +x  (forward: base -> objects)
                        /
                       /             table top is the z = 0 plane
                      v              (floor / ground at z = -0.92)
                    +y (robot's left;   -y = robot's right)

The robotic arm (Panda) root/base link is fixed at world (-0.615, 0, 0) with identity orientation. The RPY are extrinsic XYZ Euler angles and are unwrapped for temporal continuity across frames (so they can exceed ±π), not bounded principal values.

info dict

Key	Meaning	Typical content
task_goal	Task goal list	list[str]
simple_subgoal_online	Oracle online simple subgoal	Description of the current simple subgoal
grounded_subgoal_online	Oracle online grounded subgoal	Description of the current grounded subgoal
available_multi_choices	Current available options for multi-choice actions	A list such as {"label": "a/b/...", "action": str, "need_parameter": bool}, where need_parameter means the action requires grounding info such as [y, x]
front_camera_intrinsic	Front camera intrinsic	Camera intrinsic matrix
wrist_camera_intrinsic	Wrist camera intrinsic	Camera intrinsic matrix
status	Status flag	One of success, fail, timeout, ongoing, error