For new labs and companies contributing egocentric human demonstration data to the EgoVerse consortium.

---

Table of Contents

1. Overview
2. Dataset Practices
3. Prerequisites
4. Episode Hash Convention
5. Database Registry
6. Zarr v3 Episode Format
7. Coordinate Frame Conventions
8. Language Annotations
9. Embodiment Identifiers
10. Uploading to S3
11. Validation and Verification
12. Pre-Submission Checklist
13. Getting Access and Contact

---

1. Overview

EgoVerse is a multi-lab egocentric human demonstration dataset for robot co-training. The primary storage and training format is EgoVerse's own Zarr v3 schema.

Every contributed episode must satisfy these check lists:

Contract	What it enforces
File format	Zarr v3 store with specific key names, dtypes, and shapes
Coordinate frame	All poses expressed in a consistent reference frame
Database record	Consistent one row per episode registered in the PostgreSQL episode registry before upload
Dataset Practices	Example: reducing idle times, check for data flaws

The pipeline at a glance:

Your raw data
    └─► Convert to Zarr v3 (this guide)
    └─► Register row in app.episodes DB
    └─► Upload to s3://rldb/processed_v3/<embodiment>/<episode_hash>.zarr/
    └─► Available for download dynamically through S3MultiDataset

---

2. Dataset Practices

What to capture, and how to keep it clean.

We want to capture economically useful work performed by a proficient demonstrator.

2.1 Target Data Composition

A rough heuristic for the data in aggregate:

Dimension	Target mix
Task type	no more than ~5% navigation · ~10% mobile manipulation · ~85% manipulation
Gripper	~60% doable with a parallel-jaw gripper · ~40% doable by either a parallel-jaw gripper or a dexterous hand
Setting	~70% tabletop · ~30% non-tabletop

2.2 Capture Settings

Data can range from staged studios (realistic work captured in a controlled setting) all the way to in the wild — people doing tasks at home or in factory settings. If you are capturing real labor in the wild, it is especially important to trim down to the relevant portions.

2.3 Quality — Avoid These Failure Modes

A common failure mode is the demonstrator randomly stalling, inspecting an item, patting clothes, etc. Capture proficient, purposeful execution — not idle filler.

2.4 Trimming Rules

• Trim out aggressive head movements.
• Hand tracking must be visible in all frames — trim out any frames without hands.
• In-the-wild captures: trim down to the task-relevant portions (drop setup, breaks, wandering).

2.5 Example Tasks

Some example tasks we would prefer:

#	Task	What it covers
1	Sorting	Sort parts/components into bins by type, color, or size (factory); sort utensils into a cutlery tray, or sort a cluttered table by category (food / tool / toy), color, or shape (home).
2	Packing	Place items into a box, bin, or bag with room to spare (loose packing, not tight-fit).
3	Opening & closing containers	Drawers, cabinet doors, box lids/flaps, bags (ziploc, drawstring).
4	Tidying a cluttered table	Return scattered objects to their places / into containers (dry, no wiping).
5	Spatial arrangement	Arrange objects into an approximate target layout: line up, group by type, set out in roughly canonical positions (exact spacing not required).
6	Stacking	Stack wide, stable items (plates, bowls, books) or nest bowls; no precise tall towers.
7	Folding	Rough-fold towels / cloth / clothing in half or thirds; crisp creases not required.
8	Capping & uncapping	Place or remove loose lids on boxes/jars, or large-thread caps; no fine threading.
9	Hanging	Drape a cloth/towel over a rod, or hang a bag/mug on a large hook; generous targets.
10	Shelving	Place books/boxes onto an open shelf with free space; no tight insertion.
11	Loading & unloading	Load items into a tray, caddy, or dish rack with generous slots; unload onto the table.
12	Buttons & switches	Press large buttons, flip switches, toggle controls.
13	Retrieval by description	Pick a specified item from a mixed set and bring it to a drop zone — e.g. "the red mug," "the biggest book," "the metal one." Delivery to a zone, not a precise pose.
14	Relational placement	Place an object relative to another by instruction: to the left of, behind, between, or on top of a reference object. Approximate positions are fine.
15	Matching & pairing	Pair like items (socks, gloves, shoes), match lids to their containers, or match an object to its printed outline. Forgiving placement.
16	Reorientation & flipping	Turn objects to a target pose: stand cups upright, flip cards face-up, or rotate items so labels face front. Forgiving rotation, no exact angle.
17	Search & retrieve	Open a drawer/box, find a specified item inside, and take it out (combines opening, selection, and extraction).

---

3. Prerequisites

3.1 Hardware

EgoVerse is hardware-agnostic. Any egocentric camera with a SLAM system that provides 6-DOF pose tracking is supported. The minimum requirements are:

Item	Requirement
Egocentric camera	Any camera worn or mounted on the head/torso providing a first-person RGB stream at ≥ 30 fps. Examples: Project Aria glasses, OAK-D, ZED Mini, RealSense T265, GoPro + external SLAM.
SLAM / pose tracking	A system that outputs 6-DOF device pose in a consistent metric world frame at ≥ 30 fps, synchronized with the RGB stream. Examples: Aria MPS, ZED SDK positional tracking, ORB-SLAM3, OpenVINS, RealSense tracking firmware.
Hand tracking	Per-frame 3D hand landmark estimates (21 keypoints per hand) synchronized to the RGB stream, expressed in the same SLAM world frame. Examples: Aria MPS hand tracking, MediaPipe + depth unprojection, OAK-D depthai hand tracker, Ultraleap. If your setup does not produce hand keypoints, omit *.obs_keypoints and *.obs_wrist_pose and use only *.obs_ee_pose (e.g. derived from a robot's FK or a wrist-worn IMU).
Wrist cameras	Optional. Include as images.left_wrist / images.right_wrist if present.
Robot	Any bimanual arm or single-arm platform. See §9 for embodiment identifiers.

Minimum viable setup (no robot): egocentric camera + SLAM + hand tracking → contributes images.front_1, obs_head_pose, left/right.obs_ee_pose, left/right.obs_wrist_pose, left/right.obs_keypoints.

If your SLAM system does not run at 30 fps, ensure you upsample or interpolate pose tracks to match the RGB frame rate before writing. The training pipeline assumes all arrays are frame-aligned.

3.2 Software

# Clone and install EgoVerse
git clone git@github.com:GaTech-RL2/EgoVerse.git
cd EgoVerse
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e .

3.3 Credentials

You need two things: AWS credentials for the episode registry (PostgreSQL via Secrets Manager) and Cloudflare R2 credentials for the data bucket.

Step 1 — AWS keys (one-time, ask the consortium lead for these):

aws configure
# AccessKeyId: <provided by consortium>
# SecretAccessKey: <provided by consortium>
# Default region: us-east-2
# Output format: (leave blank)

Step 2 — Fetch R2 and DB credentials:

bash egomimic/utils/aws/setup_secret.sh
# Writes ~/.egoverse_env with R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY,
# AWS_ENDPOINT_URL_S3, SECRETS_ARN, etc.

Verify your setup:

from egomimic.utils.aws.aws_data_utils import load_env
from egomimic.utils.aws.aws_sql import create_default_engine

load_env()
engine = create_default_engine()   # should print: Tables in schema 'app': ['episodes']

---

4. Episode Hash Convention

Every episode is identified by a UTC timestamp rendered as:

YYYY-MM-DD-HH-MM-SS-ffffff

where ffffff is microseconds zero-padded to 6 digits.

Examples:

2025-10-14-04-15-30-000000
2026-01-12-03-47-29-664000

Rules:

• The episode hash is the primary key in the database. It must be globally unique.
• Use the UTC wall-clock time at the start of the recording as the hash.
• If your hardware does not produce a UTC timestamp natively, convert from device clock using a synchronized offset.
• The .zarr directory on S3 is named exactly <episode_hash>.zarr.

Python helpers:

from egomimic.utils.aws.aws_sql import episode_hash_to_timestamp_ms, timestamp_ms_to_episode_hash

# Convert a UTC epoch millisecond integer to an episode hash string
hash_str = timestamp_ms_to_episode_hash(1736651249664)
# → "2026-01-12-03-47-29-664000"

# Convert back
ts_ms = episode_hash_to_timestamp_ms("2026-01-12-03-47-29-664000")
# → 1736651249664

---

5. Database Registry

Every episode must be registered in the PostgreSQL app.episodes table before its Zarr store is uploaded. The registry is the authoritative index used by all download and training tooling.

5.1 Schema

The authoritative schema is the TableRow dataclass defined in egomimic/utils/aws/aws_sql.py. Refer to that file for the exact set of fields, defaults, and types — this guide may drift if the schema changes.

Key field notes:

• episode_hash: PRIMARY KEY, must match the .zarr directory name exactly (see §4).
• operator: hashed operator ID (e.g. SHA-256 hex digest). MUST be hashed before insertion — never store raw names/emails.
• lab: short, stable, lowercase string. Once set, do not change it (used in filters).
• task: high-level task_name that groups related episodes. Before inventing a new name, check the existing tasks in the episode registry via sql_tutorial.ipynb (df.groupby("task").size()) and reuse one if your episode fits. If no existing task matches, canonicalize your new task_name to a short, stable, lowercase string that names a semantically meaningful category (e.g. fold_clothes, object_in_container) — not a one-off trial description. Put trial-specific detail in task_description, scene, and objects.
• embodiment: must be one of the strings in §9.
• robot_name: the same human_* / eva_* string as embodiment (there is no per-platform variant — record your lab/hardware in the lab field, not here).

5.2 Inserting a Row

from egomimic.utils.aws.aws_sql import TableRow, add_episode, create_default_engine
from egomimic.utils.aws.aws_data_utils import load_env

load_env()
engine = create_default_engine()

# IMPORTANT: hash the operator identifier before inserting. Do not store raw
# names, emails, or any PII in the `operator` column.
import hashlib
operator_hash = hashlib.sha256(b"jane_doe").hexdigest()

row = TableRow(
    episode_hash   = "2026-03-15-14-22-10-000000",
    operator       = operator_hash,
    lab            = "rl2",
    task           = "fold_clothes",
    embodiment     = "human_bimanual",
    robot_name     = "human_bimanual",
    task_description = "folding a 2T baby shirt on a blue table",
    scene          = "kitchen_A",
    objects        = "baby_shirt_2T",
    num_frames     = 2712,
)

add_episode(engine, row)

add_episode raises RuntimeError on a duplicate episode_hash. Check for collisions before inserting.

5.3 Updating a Row After Upload

from egomimic.utils.aws.aws_sql import update_episode

row.zarr_processed_path = "s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr"
row.num_frames = 2712
update_episode(engine, row)

---

6. Zarr v3 Episode Format

Each episode is a Zarr v3 group (a directory ending in .zarr) containing arrays and top-level attributes.

6.1 Directory Structure

<episode_hash>.zarr/
├── zarr.json                       ← top-level group metadata + episode attrs
├── annotations/                    ← language annotations (may be empty)
│   ├── zarr.json
│   └── c/                          ← chunk data
├── images.front_1/                 ← egocentric RGB frames (required)
│   ├── zarr.json
│   └── c/
├── images.left_wrist/              ← left wrist camera RGB frames (optional)
│   ├── zarr.json
│   └── c/
├── images.right_wrist/             ← right wrist camera RGB frames (optional)
│   ├── zarr.json
│   └── c/
├── left.obs_ee_pose/               ← left end-effector pose (required for bimanual)
├── right.obs_ee_pose/              ← right end-effector pose (required for bimanual)
├── left.obs_wrist_pose/            ← left wrist pose (required if hand tracking available)
├── right.obs_wrist_pose/           ← right wrist pose (required if hand tracking available)
├── left.obs_keypoints/             ← left hand keypoints (required if hand tracking available)
├── right.obs_keypoints/            ← right hand keypoints (required if hand tracking available)
├── left.obs_gripper/               ← left gripper state (required if parallel gripper)
├── right.obs_gripper/              ← right gripper state (required if parallel gripper)
├── left.cmd_gripper/               ← left gripper command (required if parallel gripper)
├── right.cmd_gripper/              ← right gripper command (required if parallel gripper)
├── obs_head_pose/                  ← egocentric device pose (required)
├── obs_eye_gaze/                   ← eye gaze direction (if available)
└── obs_rgb_timestamps_ns/          ← per-frame capture timestamps

6.2 Required Arrays

All arrays are indexed along axis 0 by frame index. Every array must have exactly total_frames entries along axis 0 (matching the value in zarr.attrs["total_frames"]).

Images

Key	Shape	Dtype	Notes
images.front_1	(T,) of variable-length bytes	VariableLengthBytes	JPEG-encoded RGB frames
images.left_wrist	(T,) of variable-length bytes	VariableLengthBytes	Optional. Include if wrist camera present.
images.right_wrist	(T,) of variable-length bytes	VariableLengthBytes	Optional. Include if wrist camera present.

Egocentric Device Pose (all contributors)

Key	Shape	Dtype	Frame	Notes
obs_head_pose	(T, 7)	float64	SLAM world frame	6-DOF pose of the egocentric camera/device as XYZWXYZ. This is the pivot used at training time to re-express all other poses into head-relative coordinates. Required for all contributors.

Hand and Wrist Poses (if hand tracking is available)

Provide these if your setup produces 3D hand estimates. Omit the entire key (do not write zeros) if not available.

Key	Shape	Dtype	Frame	Notes
left.obs_ee_pose	(T, 7)	float64	SLAM world frame	Left hand end-effector (fingertip centroid or palm center) pose as XYZWXYZ
right.obs_ee_pose	(T, 7)	float64	SLAM world frame	Right hand end-effector pose as XYZWXYZ
left.obs_wrist_pose	(T, 7)	float64	SLAM world frame	Left wrist origin pose as XYZWXYZ
right.obs_wrist_pose	(T, 7)	float64	SLAM world frame	Right wrist origin pose as XYZWXYZ
left.obs_keypoints	(T, 63)	float64	SLAM world frame	21 hand landmarks × 3 (x, y, z); flattened row-major (see ordering below)
right.obs_keypoints	(T, 63)	float64	SLAM world frame	21 hand landmarks × 3 (x, y, z); flattened row-major

If your system only provides wrist pose (not full keypoints), include *.obs_wrist_pose and *.obs_ee_pose and omit *.obs_keypoints.

If your system provides only a single aggregate hand pose (e.g. palm center from a depth sensor), populate *.obs_ee_pose only.

Keypoint ordering (21 landmarks): Use the keypoints convention of MANO.

If you need to convert your proprietary keypoints to MANO, try using otaheri/MANO.

Robot Arm Poses (if operating alongside a robot)

Key	Shape	Dtype	Notes
left.obs_ee_pose	(T, 7)	float64	Left arm EEF pose as XYZWXYZ in robot base frame
right.obs_ee_pose	(T, 7)	float64	Right arm EEF pose as XYZWXYZ in robot base frame
left.obs_gripper	(T, 1)	float64	Left gripper aperture in [0, 1] (0 = fully closed)
right.obs_gripper	(T, 1)	float64	Right gripper aperture in [0, 1]
left.cmd_ee_pose	(T, 7)	float64	Commanded left EEF pose (if available)
right.cmd_ee_pose	(T, 7)	float64	Commanded right EEF pose (if available)
left.cmd_gripper	(T, 1)	float64	Commanded left gripper (if available)
right.cmd_gripper	(T, 1)	float64	Commanded right gripper (if available)

Timestamps and Misc

Key	Shape	Dtype	Notes
obs_rgb_timestamps_ns	(T,)	int64	UTC nanoseconds for each RGB frame
obs_eye_gaze	(T, 3)	float64	Unit gaze direction vector in SLAM world frame (x, y, z)

6.3 Top-Level Attributes (zarr.attrs)

The root group's .attrs dictionary is the episode metadata. It is written as JSON and is the primary indexing surface.

{
    "embodiment":        str,   # e.g. "human_bimanual"  (must match DB row)
    "total_frames":      int,   # number of valid frames (not padded)
    "fps":               int,   # capture frame rate (typically 30)
    "task_name":         str,   # e.g. "fold_clothes"  (must match DB row)
    "task_description":  str,   # free-text description of the trial
    "intrinsics":        dict,  # MANDATORY: {camera_key: 3x4 K matrix} dict (single-camera =
                                #   one entry, e.g. {"front_1": K}; 3x4 = the 3x3 pinhole K
                                #   with a zero last column). Projection uses the "front" entry.
    "extrinsics":  dict | None, # None, or a non-empty dict of 4x4 world<-cam transforms.
                                #   Robots key per-arm, e.g. {"left": T, "right": T}.
                                #   Egocentric human contributors omit it (None).
    "features": {
        "<key>": {
            "dtype":  str,        # numpy dtype string, or "jpeg" for images, "json" for annotations
            "shape":  list[int],  # per-frame shape (no time dimension)
            "names":  list[str],  # dimension labels (e.g. ["dim_0"] or ["height", "width", "channel"])
            # images only:
            # "dtype": "jpeg", "shape": [H, W, 3], "names": ["height", "width", "channel"]
            # annotations only:
            # "dtype": "json", "shape": [N], "names": ["json"], "format": "annotation_v1"
        },
        ...
    }
}

Rules:

• total_frames must equal len(store["images.front_1"]) and every other non-padded array.
• fps must be the actual capture rate of images.front_1. Do not set to a target rate if the actual rate differs.
• features must have one entry per array key present in the store.
• embodiment and task_name must exactly match the values in the DB row for this episode.
• intrinsics is mandatory and is always a {camera_key: 3×4 K matrix} dict in zarr.attrs (single-camera = one entry, e.g. {"front_1": K}). ZarrWriter.create_and_write raises if it is not a non-empty dict.
• extrinsics is either None or a non-empty dict of 4×4 world↔cam transforms (robots key per-arm, e.g. {"left": T, "right": T}); egocentric human contributors omit it (None). ZarrWriter.create_and_write raises if it is anything other than None or a non-empty dict.

6.4 Storage / Chunking

⚠️ USE THE ZarrWriter CLASS ⚠️

This is the only supported way to produce EgoVerse Zarr stores. It guarantees sharding and chunking match the rest of the dataset — do NOT roll your own writer.

• Numeric arrays: chunk shape (chunk_timesteps, *frame_shape) with chunk_timesteps=100, sharded to full array shape.
• Image arrays: chunk shape (1,) (one JPEG blob per chunk), sharded to full array shape.
• Annotation arrays: chunk shape (N,), sharded to (N,).
• Zarr format version: always v3 (zarr_format=3).

See example usage in eva_to_zarr.py and aria_to_zarr.py.

Camera intrinsics & extrinsics — how to store them

Do not hand-write these into zarr.attrs yourself. Pass them to ZarrWriter.create_and_write (or the ZarrWriter(...) constructor) via the intrinsics= / extrinsics= arguments; the writer serializes them into zarr.attrs under the "intrinsics" / "extrinsics" keys (§6.3).

• intrinsics is a REQUIRED dict of the form {camera_key: 3x4 K matrix} — create_and_write raises a ValueError if it is not a non-empty dict. Single-camera setups still use a dict — just one entry, e.g. {"front_1": K}. (Always a dict, so downstream code has one clear structure to handle.)
• Each value is a 3×4 K matrix: the standard 3×3 pinhole matrix with an appended zero column (i.e. [K | 0]). A bare 3×3 is rejected on the projection path — pad it with np.hstack([K_3x3, np.zeros((3, 1))]).
• Multi-camera rigs: add one entry per camera, e.g. {"front_1": K_front, "left_wrist": K_lw, "right_wrist": K_rw}. The training/viz projection uses the front-camera entry (the key containing front), so make sure that one is present and correct.
• extrinsics is **None or a non-empty dict$** \text{of} 4 \times 4 \text{world}↔\text{cam} \text{transforms} — $create_and_write raises on anything else. Robots key it per-arm, e.g. {"left": T_left, "right": T_right}; egocentric human contributors pass None (omit it).

import numpy as np
from egomimic.rldb.zarr.zarr_writer import ZarrWriter

# fx=fy=248.57, cx=320, cy=180  ->  3x4 K (note the zero last column)
K_front = np.array([
    [248.57,   0.0,   320.0, 0.0],
    [  0.0,  248.57,  180.0, 0.0],
    [  0.0,    0.0,     1.0, 0.0],
])

ZarrWriter.create_and_write(
    episode_path="path/to/<episode_hash>.zarr",
    embodiment="human_bimanual",
    numeric_data=numeric_arrays,        # left/right.obs_ee_pose, obs_head_pose, ...
    image_data=image_arrays,            # images.front_1, ...
    intrinsics={"front_1": K_front},    # REQUIRED — always a {camera_key: 3x4} dict
    # extrinsics=...,                   # REQUIRED for robot embodiments only
    fps=30,
    task_name="...",
    task_description="...",
)

6.5 Episode Preview MP4 (sibling artifact)

Alongside each <episode_hash>.zarr store, write a preview video of the egocentric RGB stream named <episode_hash>.mp4 (e.g. 2026-03-15-14-22-10-000000.mp4). The Mecka AI dataset viz looks previews up by this exact filename, so any deviation from the <episode_hash>.mp4 convention will break it.

Any standard MP4 encoder works. If it's convenient, the save_preview_mp4 helper is available — aria_to_zarr.py and eva_to_zarr.py use it (via the --save-mp4 flag) and emit the file next to the .zarr directory.

---

7. Coordinate Frame Conventions

7.1 SLAM World Frame (storage frame)

All poses are stored in the SLAM world frame produced by your pose-tracking system (e.g. Aria MPS, ZED SDK, ORB-SLAM3). This is an arbitrary fixed Euclidean frame that is consistent within a single recording session but not consistent across sessions or between different hardware setups.

• Origin: defined by the SLAM system at recording start; treat as opaque.
• Axes: right-handed, metric (meters).
• This is what you write into the Zarr arrays. Do not pre-transform poses to any other frame before writing.

The SLAM world frame origin and orientation will differ between labs and hardware. That is expected and fine — the training-time head-frame normalization (§7.2) cancels out any global offset or rotation.

7.2 Head Frame (training frame)

At training time, the pipeline automatically re-expresses all poses relative to the current egocentric device pose (obs_head_pose) using ActionChunkCoordinateFrameTransform. You do not need to do this conversion yourself; it is applied on-the-fly by the data loader.

The head frame is:

• Origin: the egocentric camera/device center at the current timestep.
• +X: right.
• +Y: down.
• +Z: forward (into the scene from the camera).

The end-effector frame uses the same convention (+X right, +Y down, +Z forward).

7.3 Wrist Frame (optional training frame)

For keypoint-based models, keypoints can optionally be further expressed relative to the wrist frame via PoseCoordinateFrameTransform. Again, this is a training-time transform; store everything in the SLAM world frame.

7.4 Frame Summary

Array	Written in	Re-expressed at train time
left.obs_ee_pose	SLAM world	Head frame
right.obs_ee_pose	SLAM world	Head frame
left.obs_wrist_pose	SLAM world	Head frame
right.obs_wrist_pose	SLAM world	Head frame
left.obs_keypoints	SLAM world	Head frame, then optionally wrist frame
right.obs_keypoints	SLAM world	Head frame, then optionally wrist frame
obs_head_pose	SLAM world	Used as the re-expression pivot; deleted from batch after transform
obs_eye_gaze	SLAM world	Not re-expressed (stored as unit direction)
Robot *.obs_ee_pose	Robot base frame	Robot base frame (no re-expression)

---

8. Language Annotations

Language annotations are optional but strongly encouraged. They are stored as a span-based structure: each annotation covers a contiguous range of frames.

8.1 Format (annotation_v1)

The annotations array in the Zarr store contains N entries, where N is the total number of annotation spans in the episode (not the number of frames). Each entry is a UTF-8-encoded JSON string:

{"text": "pick up the shirt", "start_idx": 0, "end_idx": 145}

Field	Type	Description
text	str	Natural-language description of what is happening during [start_idx, end_idx)
start_idx	int	First frame index (inclusive)
end_idx	int	Last frame index (exclusive). Must satisfy 0 <= start_idx < end_idx <= total_frames.

Rules:

• Spans may overlap.
• Spans do not need to cover the entire episode.
• text must be in English.
• Use the imperative or present-continuous form: "pick up the shirt", "folding the left sleeve", etc.
• Do not encode task-level descriptions here (those go in task_description). Use annotations for sub-step descriptions.
• An empty annotations array (shape (0,)) is valid when no annotation is available.

8.2 Annotation Granularity

Use at minimum one annotation per task phase. For fold_clothes, for example:

Phase	Example annotation text
Grasp	"grasping the shirt by the collar"
Unfold	"unfolding and laying the shirt flat"
Fold left sleeve	"folding the left sleeve towards the center"
Fold right sleeve	"folding the right sleeve towards the center"
Fold body	"folding the bottom half up to complete the fold"

8.3 Writing Annotations

Via ZarrWriter:

from egomimic.rldb.zarr.zarr_writer import ZarrWriter

annotations = [
    ("grasping the shirt by the collar",        0,   145),
    ("unfolding and laying the shirt flat",    145,   420),
    ("folding the left sleeve towards center", 420,   680),
    ("folding the right sleeve",               680,   910),
    ("folding the bottom half up",             910,  1200),
]

writer = ZarrWriter(
    episode_path="path/to/<episode_hash>.zarr",
    embodiment="human_bimanual",
    fps=30,
    task_name="fold_clothes",
    task_description="folding a 2T baby shirt",
    annotations=annotations,
)

To append annotations to an existing Zarr store:

writer = ZarrWriter(episode_path="path/to/<episode_hash>.zarr")
writer.append_annotations(
    annotation_key="annotations",
    annotations=annotations,
    mode="w",   # "w" = overwrite existing, "a" = append
)

8.4 Scale AI Annotation Format

If you are delivering data through Scale AI, annotations are generated via the Scale annotation API. The ScaleAnnotationDatasetFilter class can be used to filter episodes to only those with completed Scale annotations. Set SCALE_API_KEY in your environment.

---

9. Embodiment Identifiers

The embodiment field in the DB row and in zarr.attrs must be exactly one of the strings below. All human demonstration data is a single human embodiment — there is no per-vendor or per-hardware embodiment. The lab / hardware that produced the data is recorded separately in the SQL lab field, never in embodiment. Only the robot Eva is a distinct non-human embodiment.

embodiment string	Integer id	Description
human_right_arm	1	Egocentric human demonstration, right arm only
human_left_arm	2	Egocentric human demonstration, left arm only
human_bimanual	3	Egocentric human demonstration, two-arm
eva_right_arm	4	Eva camera + right-arm robot
eva_left_arm	5	Eva camera + left-arm robot
eva_bimanual	6	Eva camera + bimanual robot

If you are contributing egocentric human data, you use human_bimanual (or the single-arm variants) regardless of your hardware — set the lab field (e.g. lab="microagi") to identify your source.

9.1 Using the Human embodiment (no subclass, no per-vendor identifier)

There is a single concrete Human embodiment class (egomimic/rldb/embodiment/human.py) shared by all human data. You do not write a per-vendor subclass, and there is no per-vendor embodiment identifier — every human contributor uses human_* and records their source in the lab field. Camera intrinsics travel with the data (zarr.attrs, §6.3 / §6.4); per-vendor structural choices are passed as explicit arguments from the data config:

• Human.get_keymap(keymap_mode="cartesian"|"keypoints", has_head_pose=<bool>, include_aria_keypoints=<bool>)
• Human.get_transform_list(mode="cartesian"|"keypoints_headframe_ypr"|..., stride=<int>)

Onboarding human data is just two steps:

1. Write embodiment="human_bimanual" (or human_left_arm / human_right_arm) in the DB row and in zarr.attrs; record your lab/hardware in the lab field.
2. Add a data config under egomimic/hydra_configs/data/ whose key_map / transform_list point at Human.get_keymap / Human.get_transform_list with the args your data needs. Copy aria.yaml (head-mounted, stride: 3) or scale.yaml (no head pose: has_head_pose: false, stride: 1).

Notes:

• Camera intrinsics are MANDATORY and live in zarr.attrs as a {camera_key: 3x4} dict (§6.4). You no longer declare an INTRINSICS constant in code.
• has_head_pose=False if your data has no obs_head_pose; stride is the action-chunk stride (3 for ~30 fps egocentric, 1 for already-downsampled data).
• Robots subclass Embodiment directly and keep their own intrinsics/extrinsics + pipeline — see Eva in egomimic/rldb/embodiment/eva.py.

---

10. Uploading to S3

10.1 S3 Path Convention

s3://rldb/processed_v3/<embodiment_prefix>/<episode_hash>.zarr/

Embodiment	<embodiment_prefix>
human_*	human
eva_*	eva

Examples:

s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr/
s3://rldb/processed_v3/eva/2025-11-04-09-30-00-000000.zarr/

10.2 Upload with s5cmd

s5cmd is the recommended upload tool (installed as part of the Python environment).

# Upload a local .zarr directory
s5cmd --endpoint-url $AWS_ENDPOINT_URL_S3 \
      sync "/local/processed/2026-03-15-14-22-10-000000.zarr/*" \
           "s3://rldb/processed_v3/human/2026-03-15-14-22-10-000000.zarr/"

Or using the Python utility:

from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3, load_env

load_env()
upload_dir_to_s3(
    local_dir = "/local/processed/2026-03-15-14-22-10-000000.zarr",
    bucket    = "rldb",
    prefix    = "processed_v3/human/2026-03-15-14-22-10-000000.zarr",
)

10.3 Bulk Upload with Ray

For batch uploads of many episodes, use Ray to parallelize:

import ray
from egomimic.utils.aws.aws_data_utils import upload_dir_to_s3, load_env

ray.init()

@ray.remote
def upload_one(local_zarr_path: str, s3_prefix: str):
    load_env()
    upload_dir_to_s3(local_zarr_path, bucket="rldb", prefix=s3_prefix)

tasks = [
    upload_one.remote(
        f"/local/processed/{h}.zarr",
        f"processed_v3/human/{h}.zarr"
    )
    for h in episode_hashes
]
ray.get(tasks)

---

11. Validation and Verification

11.1 Automated Checks

Run these checks on every episode before uploading:

import zarr, numpy as np
import json
from pathlib import Path
from egomimic.rldb.zarr.zarr_dataset_multi import ZarrEpisode
import simplejpeg

def validate_episode(zarr_path: str) -> tuple[list[str], list[str]]:
    """Returns (errors, successes). Empty errors list = pass."""
    errors: list[str] = []
    successes: list[str] = []
    ep = ZarrEpisode(zarr_path)
    meta = ep.metadata
    T = meta["total_frames"]
    store = zarr.open(zarr_path, mode="r")

    # ── Metadata ────────────────────────────────────────────────────────────
    for field in ("embodiment", "total_frames", "fps", "task_name", "features"):
        if field not in meta:
            errors.append(f"Missing metadata field: {field}")
        else:
            successes.append(f"metadata field present: {field}")

    if meta.get("fps", 0) not in (30, 60):
        errors.append(f"Unexpected fps={meta['fps']}. Expected 30 or 60.")
    else:
        successes.append(f"fps={meta['fps']} is valid")

    # ── Embodiment identifier (must resolve to a valid id; see §9) ──────────
    from egomimic.rldb.embodiment.embodiment import get_embodiment_id
    try:
        get_embodiment_id(meta.get("embodiment", ""))
        successes.append(f"embodiment={meta.get('embodiment')} is a valid identifier")
    except (KeyError, AttributeError):
        errors.append(f"embodiment={meta.get('embodiment')!r} is not a valid identifier (see §9)")

    # ── Camera intrinsics (MANDATORY; {camera_key: 3x4 K matrix} dict) ──────
    intr = meta.get("intrinsics")
    if not isinstance(intr, dict) or not intr:
        errors.append("intrinsics: missing or not a non-empty {camera_key: 3x4} dict")
    else:
        if not any("front" in str(k).lower() for k in intr):
            errors.append(f"intrinsics: no front-camera entry (keys: {list(intr)})")
        for cam, K in intr.items():
            if np.asarray(K, dtype=float).shape != (3, 4):
                errors.append(f"intrinsics['{cam}']: expected 3x4 K, got shape {np.asarray(K).shape}")
            else:
                successes.append(f"intrinsics['{cam}']: 3x4 OK")

    # ── Camera extrinsics (OPTIONAL; None, or a non-empty dict of transforms) ─
    if "extrinsics" in meta and meta["extrinsics"] is not None:
        extr = meta["extrinsics"]
        if not isinstance(extr, dict) or not extr:
            errors.append("extrinsics: present but not a non-empty dict (must be None or a dict)")
        else:
            successes.append(f"extrinsics: non-empty dict OK (keys: {list(extr)})")

    # ── Frame counts ────────────────────────────────────────────────────────
    features = meta.get("features", {})
    for key in store.keys():
        node = store[key]
        if not isinstance(node, zarr.Array):
            continue
        if features.get(key, {}).get("dtype") == "json":
            continue
        arr_len = node.shape[0]
        if arr_len < T:
            errors.append(f"{key}: array length {arr_len} < total_frames {T}")
        else:
            successes.append(f"{key}: frame count OK ({arr_len} >= {T})")

    # ── Required keys ───────────────────────────────────────────────────────
    required = ["images.front_1", "left.obs_ee_pose", "right.obs_ee_pose"]
    for key in required:
        if key not in store:
            errors.append(f"Missing required key: {key}")
        else:
            successes.append(f"required key present: {key}")

    # ── Pose shapes and norms ───────────────────────────────────────────────
    required_poses = ("left.obs_ee_pose", "right.obs_ee_pose")
    optional_poses = ("left.obs_wrist_pose", "right.obs_wrist_pose", "obs_head_pose", "left.cmd_ee_pose", "right.cmd_ee_pose")
    for key in required_poses + optional_poses:
        if key in store:
            arr = store[key][:]
            if arr.shape != (T, 7) and arr.shape[0] >= T:
                arr = arr[:T]
            if arr.shape[-1] != 7:
                errors.append(f"{key}: expected shape (T, 7), got {arr.shape}")
                continue
            else:
                successes.append(f"{key}: shape OK (T, 7)")
            quat = arr[:, 3:7]
            norms = np.linalg.norm(quat, axis=1)
            if not np.allclose(norms, 1.0, atol=1e-4):
                bad = np.where(np.abs(norms - 1.0) > 1e-4)[0]
                errors.append(f"{key}: {len(bad)} frames with non-unit quaternions (e.g. frame {bad[0]}, norm={norms[bad[0]]:.6f})")
            else:
                successes.append(f"{key}: all quaternions unit-norm")

    # ── Gripper shapes (optional) ───────────────────────────────────────────
    for key in ("left.obs_gripper", "right.obs_gripper", "left.gripper", "right.gripper"):
        if key in store:
            arr = store[key][:]
            if arr.shape[0] < T:
                errors.append(f"{key}: array length {arr.shape[0]} < total_frames {T}")
                continue
            if arr.ndim != 2 or arr.shape[-1] != 1:
                errors.append(f"{key}: expected shape (T, 1), got {arr.shape}")
            else:
                successes.append(f"{key}: gripper shape OK (T, 1)")

    # ── Keypoint shapes ─────────────────────────────────────────────────────
    for key in ("left.obs_keypoints", "right.obs_keypoints"):
        if key in store:
            arr = store[key][:]
            if arr.shape[-1] != 63:
                errors.append(f"{key}: expected last dim 63 (21×3), got {arr.shape[-1]}")
            else:
                successes.append(f"{key}: keypoint shape OK (last dim = 63)")

    # ── Annotation format (JSON-encoded records) ────────────────────────────
    annotation_keys = [k for k, f in features.items() if f.get("dtype") == "json" and k in store]
    for key in annotation_keys:
        node = store[key]
        n = node.shape[0]
        bad = 0
        first_err = None
        for i in range(n):
            raw = node[i]
            # Unwrap any nested 0-d object/bytes ndarrays down to raw bytes.
            while isinstance(raw, np.ndarray):
                raw = raw.item() if raw.shape == () else raw.flat[0]
            if isinstance(raw, np.bytes_):
                raw = bytes(raw)
            try:
                rec = json.loads(raw.decode("utf-8") if isinstance(raw, (bytes, bytearray)) else raw)
                if not isinstance(rec, dict):
                    raise ValueError(f"record is {type(rec).__name__}, expected dict")
                for field, expected in (("text", str), ("start_idx", int), ("end_idx", int)):
                    if field not in rec:
                        raise ValueError(f"missing field '{field}'")
                    if not isinstance(rec[field], expected):
                        raise ValueError(f"field '{field}' is {type(rec[field]).__name__}, expected {expected.__name__}")
                if not (0 <= rec["start_idx"] <= rec["end_idx"] <= T):
                    raise ValueError(f"index range invalid: start={rec['start_idx']}, end={rec['end_idx']}, T={T}")
            except Exception as e:
                bad += 1
                if first_err is None:
                    first_err = (i, str(e))
        if bad:
            errors.append(f"{key}: {bad}/{n} annotations malformed (e.g. index {first_err[0]}: {first_err[1]})")
        else:
            successes.append(f"{key}: all {n} annotations well-formed")

    # ── Image decodability (spot-check first frame of each JPEG key) ────────
    jpeg_keys = [k for k, f in features.items() if f.get("dtype") == "jpeg" and k in store]
    for key in jpeg_keys:
        data = ep.read({key: (0, None)})
        try:
            frame = simplejpeg.decode_jpeg(bytes(data[key]), colorspace="RGB")
            if frame.ndim != 3 or frame.shape[2] != 3:
                errors.append(f"{key}: decoded frame has unexpected shape {frame.shape}")
            else:
                successes.append(f"{key}: frame 0 decoded OK, shape={frame.shape}")
        except Exception as e:
            errors.append(f"{key}: failed to decode frame 0: {e}")

    return errors, successes

# Usage
errors, successes = validate_episode("/storage/project/r-dxu345-0/shared/pick_place/2026-03-17-18-09-03-000000")
for s in successes:
    print("OK:", s)
if errors:
    for e in errors:
        print("ERROR:", e)
else:
    print("All checks passed.")

11.2 End-to-End Load Test

Verify the episode loads correctly through the full training pipeline before uploading:

from pathlib import Path
from egomimic.rldb.zarr.zarr_dataset_multi import LocalEpisodeResolver, MultiDataset
from egomimic.rldb.filters import DatasetFilter
from egomimic.rldb.embodiment.human import Human
import torch

# cartesian mode re-expresses every pose relative to obs_head_pose, so the
# cartesian transform REQUIRES a head pose. Head-mounted (Aria) data uses it
# directly; head-pose-less data (e.g. Scale/Mecka — scale.yaml sets
# has_head_pose: false) cannot run the cartesian transform, so load it with
# transform_list=None to validate the raw episode (the §11.1 checks are the
# primary validation in that case).
HAS_HEAD_POSE = True   # set False for head-pose-less data (e.g. Scale/Mecka)
key_map = Human.get_keymap(keymap_mode="cartesian", has_head_pose=HAS_HEAD_POSE)
transform_list = (
    Human.get_transform_list(mode="cartesian", stride=3) if HAS_HEAD_POSE else None
)

resolver = LocalEpisodeResolver(
    folder_path    = Path("/local/processed"),
    key_map        = key_map,
    transform_list = transform_list,
)

filters = DatasetFilter(filter_lambdas=[
    "lambda row: row['episode_hash'] == '2026-03-15-14-22-10-000000'"
])

ds = MultiDataset._from_resolver(resolver, filters=filters, mode="total")
loader = torch.utils.data.DataLoader(ds, batch_size=4, num_workers=0)

# Iterate the entire dataset so any decode/shape/dtype error surfaces,
# not just something in the first batch.
for batch in loader:
    pass

Expected output for a valid human bimanual episode in cartesian mode:

• actions_cartesian: `(B, 100, 12)$ — 100-\text{step} \text{action} \text{chunk}, 6 \text{DOF} \times 2 \text{arms}
• $observations.state.ee_pose`: `(B, 12)$ — \text{current} \text{EEF} \text{poses}, 6 \text{DOF} \times 2 \text{arms}
• $observations.images.front_img_1: (B, 3, H, W)— normalized RGB in[0, 1]`

11.3 Visual Verification

After the load test passes, render a quick trajectory overlay on a local episode — it projects the action chunk onto the egocentric image using the per-episode intrinsics from zarr.attrs, so a wrong K matrix or coordinate frame shows up immediately as an overlay floating off the hands. (This step uses the cartesian actions_cartesian chunk, so it applies to head-mounted data with a head pose; head-pose-less data has no cartesian chunk to project — rely on §11.1 + the §11.2 raw load for those.)

import imageio, torch
from egomimic.rldb.embodiment.human import Human
from egomimic.rldb.zarr.zarr_dataset_multi import LocalEpisodeResolver, MultiDataset
from egomimic.rldb.filters import DatasetFilter

resolver = LocalEpisodeResolver(
    folder_path    = "/local/processed",
    key_map        = Human.get_keymap(keymap_mode="cartesian"),
    transform_list = Human.get_transform_list(mode="cartesian", stride=3),
)
filters = DatasetFilter(filter_lambdas=[
    "lambda row: row['episode_hash'] == '2026-03-15-14-22-10-000000'"
])
ds = MultiDataset._from_resolver(resolver, filters=filters, mode="total")
loader = torch.utils.data.DataLoader(ds, batch_size=1)

frames = [
    Human.viz_transformed_batch(b, mode="traj", viz_batch_key="actions_cartesian")
    for b in loader
]
imageio.mimsave("overlay_check.mp4", frames, fps=30)

Then confirm:

• Trajectories project onto the hands/end-effectors in-frame (not floating off-screen or stuck at the principal point).
• Left/right arms are not swapped.
• Keypoints (if present) form anatomically plausible hand skeletons.

Do not upload an episode whose visualization is visibly misaligned.

---

12. Pre-Submission Checklist

Complete every item before considering an episode ready for upload.

Episode hash

• Episode hash is a valid UTC timestamp string (YYYY-MM-DD-HH-MM-SS-ffffff).
• Episode hash is unique — not already in the DB (episode_hash_to_table_row(engine, hash) returns None).

Zarr format

• obs_head_pose is present (required for all contributors).
• left.obs_ee_pose and right.obs_ee_pose are present if hand tracking is available.
• All obs_ee_pose arrays have shape (T, 7) and unit-norm quaternions.
• All obs_keypoints arrays have shape (T, 63).
• features dict in zarr.attrs has one entry per array key.
• embodiment and task_name in zarr.attrs match the DB row values.
• intrinsics is present in zarr.attrs as a {camera_key: 3×4} dict (single-camera = one entry) — mandatory.
• extrinsics is present in zarr.attrs for robot embodiments.
• All episode succeeds on zarr validation check code
• An embodiment class is registered in egomimic/rldb/embodiment/ (§9.1).
• A sample episode has been visually verified via zarr_data_viz.ipynb (§11.3).

Coordinate frames

• All poses are in the SLAM world frame (not head frame, not camera frame).
• Quaternion is stored in XYZWXYZ order: [tx, ty, tz, qw, qx, qy, qz].
• Translation units are meters.

Images

• Images are in RGB order (not BGR).
• JPEG quality is 85.
• Image shape matches features["images.front_1"]["shape"].

Annotations

• annotations key is present (may be empty array if no annotations available).
• All (start_idx, end_idx) spans satisfy 0 <= start_idx < end_idx <= total_frames.
• Annotation text is in English, imperative or present-continuous form.

Database

• DB row inserted before upload.
• zarr_processed_path updated to the correct S3 path after upload.
• num_frames in DB row matches total_frames in zarr.attrs.
• embodiment in DB row exactly matches the embodiment enum string (§9).

Upload

• Episode is accessible at s3://rldb/processed_v3/<prefix>/<episode_hash>.zarr/.
• sync_s3.py with an appropriate filter can download and open the episode.

---

13. Getting Access and Contact

Access Request

To get credentials for the EgoVerse data bucket and episode registry:

1. Email the consortium leads with your lab name, GitHub handle, and a brief description of the data you intend to contribute.
2. You will receive AWS credentials (for Secrets Manager access) and instructions to run setup_secret.sh.

Consortium Leads

Person	Affiliation	Role
Danfei Xu	Georgia Tech / NVIDIA GEAR	PI, consortium lead
Simar Kareer	Georgia Tech	Infrastructure, website, data pipeline
Ryan Punamiya	Georgia Tech / NVIDIA GEAR	Technical lead, format and schema

Resources

Resource	URL
Website	https://egoverse.ai
Data browser	https://partners.mecka.ai/egoverse
arXiv paper	https://arxiv.org/abs/2604.07607
GitHub	https://github.com/GaTech-RL2/EgoVerse
License	CC BY-SA 4.0
Onboarding Slack channel (GT workspace)	#egoverse-onboarding

Reporting Issues

If you encounter processing errors, S3 permission issues, or schema questions, post in #egoverse-onboarding with:

• Your episode hash(es)
• The error message or symptom
• The output of validate_episode() for the affected episode