Extending Galactic data

April 20, 2023 ยท View on GitHub

All data needed to reproduce the Galactic paper results is stored at ./data and is included in the galactic git repo, so no additional downloads or installation steps are required. This page is aimed at users looking to extend Galactic with new objects, scenes, robots, and episodes.

Shortcuts:

Requirements

To iterate on Galactic data, we recommend a Linux desktop machine meeting the above requirements (including GPU), so that you can run both Blender and Galactic seamlessly. However, if this is unavailable, you may be able to run parts of the Data Pipeline on a Macbook (steps 1-3 and steps 5-6 below) and parts on a headless Linux system (step 4).

Types of data

These terms will be referenced elsewhere in this page.

Runtime data

  • (loaded directly during training)
  • collection JSON files
    • properties of the robot and movable objects
    • data/replicacad_composite.collection.json
    • filepath specified in BatchedSimulatorConfig::collectionFilepath
  • URDF files
    • robot articulation data
    • e.g. data/URDF/opt_fetch/robots/fetch.urdf
    • filepath specified in Collection JSON urdfFilepath
  • Obsolete URDF-related mesh files
    • loaded at runtime by some legacy Habitat-sim code but not used for anything at runtime
    • also sometimes used as source data by the Data Pipeline
    • e.g. data/URDF/opt_fetch/meshes/base_link_collision.STL, spot_arm_textured/spot_arm/spot_arm/meshes/arm0.link_el0.obj
    • filepath specified in URDF files
  • Column-grid files
    • baked collision geometry for static scenes
    • e.g. data/columngrids/Baked_sc0_staging_00.0.columngrid
    • filepath base specified in episode-set JSON columnGridSetName
  • Renderer mesh data
    • data/bps_data/composite/composite.bps and composite.bps.mapping.json
    • filepath specified in BatchedSimulatorConfig::renderAssetCompositeFilepath
  • Renderer compressed textures
    • e.g. data/bps_data/composite/textures/composite_0.ktx2
    • filepaths specified automatically inside composite.bps
  • Episode-set JSON files
    • starting agent poses, target objects, arrangements of scenes, and other episode data
    • e.g. data/train/tidy_house_100_v1.episode_set.json
    • filepath specified in Habitat-lab config entry TRAIN_DATASET

Source data

  • (used to generate runtime data, but not used directly during training)
  • source 3D models
    • e.g. source_data/replica_cad_baked_lighting/stages_uncompressed/Baked_sc0_staging_00.glb, spot_arm_textured/spot_arm/spot_arm/meshes/arm0.link_el0.obj, source_data/objects/ycb/ycb/024_bowl/google_16k/textured.obj
  • source 3D scenes
    • As a prerequisite, these must be loadable in Habitat-sim's C++ viewer. See Adding a new static scene.
    • e.g. source_data/replica_cad_baked_lighting/configs/scenes/Baked_sc0_staging_00.scene_instance.json

Intermediate data

  • (temporary data generated and used by the Data Pipeline)
  • Gfx-replay JSON files
    • generated by the Habitat-sim viewer using --gala-write-scene-gfx-replay
    • used by the procedural episode generator
    • e.g. data/replays/Baked_sc0_staging_00.replay.json
  • Composite GLB files
    • generated by scripts/blender_make_composite_glb.py
    • used to generated renderer mesh and compressed texture data

Data Pipeline

Some runtime data is baked (generated) from source data; this is the Data Pipeline. You will need to re-run the data pipeline if you are extending Galactic, so this section will be referenced elsewhere in this page.

Here, we give repro steps to rebuild the runtime data used in our paper, as a reference. This runtime data is already stored in the Galactic repo under ./data.

  1. Install dependencies.
# Install and build Galactic using instructions from README.md

# Verify that the bps3D preprocess binary is built and runnable
./habitat-sim/build_relwithdebinfo/deps/bps3D/bin/preprocess

# Install Blender v2.93.5 or newer; instructions vary by OS

# Verify Blender installation
blender --version

# Install the KTX texture compression utility
git clone --recursive https://github.com/KhronosGroup/KTX-Software.git
cd KTX-Software
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build ./
cd ../../

# Verify the KTX utility is built and runnable
./KTX-Software/build/tools/toktx/toktx
  1. Download/extract all source data.
# Download selected datasets via Habitat downloader
cd source_data
python -m habitat_sim.utils.datasets_download --data-path ./ --uids replica_cad_baked_lighting ycb

# Extract some additional source data
unzip extra_source_data.zip

cd ../
  1. Build the composite GLB file.
# Run our Blender script to pack all source 3D models into a composite GLB file. This step can include interactive inspection/troubleshooting in the Blender GUI.
blender --python ./scripts/blender_make_composite_glb.py
  1. Regenerate renderer mesh and texture data.
# If the contents or order of textures has changed, you must delete this folder. If they haven't, you can skip this step to avoid re-compressing textures (which is slow).
rm -rf ./data/bps_data/composite/textures

# Convert to bps3D optimized mesh format and compress textures.
./scripts/glb_to_bps.sh

# Optionally, clean up temp files
rm -rf ./temp ./data/bps_data/composite/textures/*.png
  1. Delete and regenerate column-grid files and gfx-replay JSON files
rm -rf data/columngrids/*.columngrid
rm -rf data/replays/*.replay.json
./scripts/preprocess_replicacad_scenes.sh
  1. Optional: rebuild source-controlled compressed data
cd source_data
zip -r extra_source_data.zip debug_models preprocessed_fetch_meshes fixed_replicacad_models
cd ../data
zip columngrids.zip ./columngrids/*.columngrid

Adding a new 3D model

3D models are used to visualize robots, movable objects, and static scenes in Galactic. The instructions here will be referenced elsewhere in this page. It's recommended you have some familiarity with Blender or other 3D-authoring software.

  1. Identify your 3D model file, e.g. your_model.glb.
    • Run Blender, import your model, and ensure it looks correct there.
    • If you encounter problems here, you may need to convert your model to another format or work with a 3D artist to find a workaround.
  2. Modify scripts/blender_make_composite_glb.py to include your new 3D model.
    • Add an ImportItem for your 3D model.
    • A current limitation of the Data Pipeline is that an imported model must either (1) contain exactly one mesh, or (2) contain multiple meshes sharing a single material, which can be merged using do_join_all.
  3. Re-run Step 3 and 4 of the Data Pipeline (above) to bake renderer mesh and texture data.
    • For this step, you may want to iterate on decimate_fraction and vertex_merge_distance to optimize your baked mesh data (lowering the vertex count to increase training speed).

Adding a new movable object

A movable object in Galactic can be grasped, moved, and dropped by the robot during simulation. Galactic supports only free rigid objects (not soft or articulated objects; not constrained articulated objects like doors or drawers). Movable objects are often called "free objects" in code and data files. In our paper, we used YCB objects. Adding a new movable object includes updating the following runtime data: renderer mesh and texture data, collection JSON files, and episode-set JSON files.

  1. Find a 3D model for your object.
  2. Add the 3D model to Galactic. See Adding a new 3D model.
    • In blender_make_composite_glb.py, see YCB object 024_bowl as an example.
  3. Add an entry for your object to your collection JSON file, in the freeObjects section.
    • For collisionBox, you can directly author this. Or, you can copy-paste from the terminal output when running blender_make_composite_glb.py; the 3D model's extents will be printed to the terminal in this exact JSON format.
  4. See Adding new episodes for how to use your new movable object.

Adding a new static scene

A static scene in Galactic is usually a house interior populated with furniture. It represents all the non-movable pieces of the scene. In our paper, we used ReplicaCAD scenes. Adding a new static scene includes updating the following runtime data: column-grid files, renderer mesh and texture data, and episode-set JSON files.

  1. Identify your source 3D scene, e.g. your_scene.glb or your_scene.scene_instance.json.
    • Prerequisite: your scene must be loadable in Habitat-sim's viewer! Various formats are supported. See Habitat-sim documentation for help.
    • Load your_scene in the Habitat-sim C++ viewer at habitat-sim/build_relwithdebinfo/utils/viewer/viewer. Verify that it looks correct.
  2. Run the viewer again with --enable-physics, --gala-write-scene-gfx-replay and --gala-generate-column-grids. This creates data/replays/your_scene.replay.json and a set of three column-grid files (data/columngrids/your_scene.0.columngrid, etc.)
    • See also scripts/preprocess_replicacad_scenes.sh for reference.
  3. Inspect your_scene.replay.json to understand all the 3D models in your scene.
    • A ReplicaCAD scene includes a stage model (Baked_sc0_staging_00.glb) and many other models (e.g. kitchencupboard_base.glb, kitchencupboard_doorWhole_L.glb).
  4. Add all of these 3D models to Galactic. See Adding a new 3D model.
    • In blender_make_composite_glb.py, see existing ReplicaCAD and MP3D examples.
  5. See Adding new episodes for how to use your new static scene.

Known Issues

  • Galactic currently only supports robot movement in the ground plane at y=0. Multi-floor scenes or scenes with the floor at a different height aren't yet supported.

Adding a Matterport3D example scene as a static scene

A Matterport3D scene GLB file contains multiple meshes with multiple materials, so it can't be added to Galactic as a single 3D model. Instead, we use a special do_generate_gfx_replay option in blender_make_composite_glb.py.

  1. Download the MP3D example scene 17DRP5sb8fy:
    • cd source_data; python -m habitat_sim.utils.datasets_download --data-path ./ --uids mp3d_example_scene
  2. Generate column-grid files:
    • cd habitat-sim; build_relwithdebinfo/utils/viewer/viewer 17DRP5sb8fy --dataset "../source_data/scene_datasets/mp3d_example/mp3d.scene_dataset_config.json" --enable-physics --gala-generate-column-grids
    • Note: unlike the instructions for Adding a new static scene, we don't use the viewer to generate a gfx-replay file.
    • Verify data/columngrids/17DRP5sb8fy.0.columngrid, 17DRP5sb8fy.1.columngrid and 17DRP5sb8fy.2.columngrid were written.
  3. Modify blender_make_composite_glb.py:
    • Set the global variable include_mp3d_example=True.
    • Consider disabling ReplicaCAD 3D models: include_replicacad_stages=False and include_replicacad_articulated_objects=False.
  4. Re-run steps 3 and 4 of the Data Pipeline.
    • Verify that replays/17DRP5sb8fy.replay.json was written.
  5. The MP3D example scene is now ready to use in new episodes. See Adding new episodes.

Adding new episodes

The task used in our paper is GeometricGoal rearrangement. An episode specifies (1) a static scene, (2) a starting arrangement of movable objects, (3) the robot agent start pose, and (4) the target object index and goal pose. See Converting Habitat 2.0 episodes for documentation on how episodes were generated for our paper.

Generating meaningful episodes for the rearrangement task is nontrivial and depends heavily on the static scene, movable objects, and robot. In this section, as a simple starting point, we'll edit and run Galactic's C++ procedural episode generator to produce a new episode-set JSON file.

  1. Choose a set of static scenes and movable objects for your episodes. We recommend you start with a single scene and single movable object.
    • You can re-use a ReplicaCAD scene used in our paper, or see Adding a new static scene.
    • You can re-use a YCB object used in our paper, or see Adding a new movable object.
  2. You should also choose a robot for your episodes.
    • You can re-use the Fetch robot used in our paper, or see Adding a new robot.
    • Your robot is specified in your collection JSON file. See also BatchedSimulator.h BatchedSimulatorConfig::collectionFilepath.
  3. Modify EpisodeGenerator.cpp generateBenchmarkEpisodeSet
    • Call addStaticScene for your desired static scene and addFreeObject for your desired movable object.
    • Changes to C++ code require rebuilding the simulator. See here.
  4. Modify EpisodeGenerator.cpp setFetchJointStartPositions.
    • This function sets the robot start joint pose for each episode.
    • This function is currently hard-coded for the Fetch robot. It starts with joint poses from your collection JSON file (see Adding a new robot) and then overrides a subset of joints to random values, based on the action space defined in your collection JSON file.
  5. Modify EpisodeGenerator.cpp addEpisode.
    • Find a reasonable axis-aligned rectangular volume in your scene where objects should be spawned. Find a similar volume where the robot agent should start. For example, you could inspect the scene in Blender, although beware that software uses a different coordinate convention that swaps y and z axes.
    • Set objectSpawnRange and robotSpawnRange appropriately.
    • Beware that the robot base's random starting position/yaw is influenced by your collection JSON file's freeObjects robotProxy collisionBox. See Adding a new robot.
  6. Configure and run the episode generator to produce a new episode-set JSON file.
    • See sim_test.py BatchedEnvTester.__init__ for an example of configuring and invoking the episode generator as part of intializing BatchedSimulator.
    • Set EpisodeGeneratorConfig.save_filepath to your_episodes.episode_set.json
  7. Test your new episodes in the BatchedSimTest viewer. See BatchedSimTest interactive viewer.
    • The r key will reset the environment to the next episode, allowing you to cycle through all episodes in the set.

Tips for advanced episode-generation:

  • your_episodes.episode_set.json contains these top-level sections: renderAssets, staticScenes, freeObjects, freeObjectSpawns, episodes.
  • Consider creating new episode-set JSON files using your_episodes.episode_set.json as a starting point.
  • freeObjectSpawns and episodes sections should be customized for new episodes.
  • renderAssets, staticScenes, and freeObjects sections should be re-used exactly as-is in new episode-set JSON files (i.e. copy-paste these sections).
  • In particular, direct edits to staticScenes should be avoided. Instead, follow the steps at Adding a new static scene. (If you change staticScenes.renderAssetInstances but don't rebuild column-grid files, you'll get a desync between the scene's visuals and its collision geometry.)

Adding a new robot

Galactic supports a kinematically-simulated, articulated robot agent, including an abstracted suction gripper for grasping movable objects. In our paper, we use the Fetch robot. However, beware that the Fetch data isn't integrated into Galactic in a clean way, so don't use it as a reference. Instead, in this section, we'll add a Spot robot with arm attachment as a proper example. Adding a new robot includes updating the following runtime data: URDF files, renderer mesh and texture data, collection JSON files, and episode-set JSON files.

  1. Download the latest Habitat-sim datasets-download script from here.
  2. Download the Spot URDF files.
    • python path/to/datasets_download.py --uids hab_spot_arm
    • These files will be a combination of source and runtime data. These should be placed under data/URDF/hab_spot_arm.
  3. Inspect hab_spot_arm/urdf/hab_spot_arm.urdf to understand all the 3D models used for visuals.
    • These are usually specified in JSON under link.visual.mesh sections.
    • Meshes used for collision can be ignored (in JSON under link.collision.geometry.mesh).
  4. Add all these 3D models to Galactic. See Adding a new 3D model.
    • If your robot uses non-mesh visuals (e.g. a cylinder geometric primitive), you'll need to create 3D models to approximate these primitives, as the Data Pipeline doesn't support geometric primitives.
  5. Edit your collection JSON file.
    • freeObjects robotProxy collisionBox is a bounding box used to help place the robot during procedural episode generation.
    • Set robots urdfFilepath to ../data/URDF/hab_spot_arm/urdf/hab_spot_arm.urdf.
    • startJointPositions specifies a default pose for your robot. The array length must exactly match the degrees of freedom of your robot.
    • actionMap configures the action space of your robot. Often, you'll be training a robot policy that controls the robot base movement, abstracted grasping, plus a subset of the robot's joints.
    • The gripper section lets you specify details of the suction gripper.
    • The links section requires you to specify collision geometry as spheres attached to certain links. You can leave this section empty for now.
  6. Revise your episode-set JSON file.
    • See Adding new episodes.
    • A rearrangement episode specifies a robot starting pose in terms of joint degrees of freedom, so the existing Galactic episode set (built for Fetch) won't work well for other robots.
    • An episode also specifies a robot starting base position/yaw, so depending on your robot's shape, the existing Galactic episode set again won't work well, e.g. your robot will spawn in penetration with the environment and be unable to simulate.
  7. Test your robot in the BatchedSimTest viewer. See BatchedSimTest interactive viewer.
    • Explore the different hotkeys and functionality here. These are not currently well-documented so this will require some patience! E.g. use the free camera to visually inspect your robot and use WASD plus number keys to move the robot.
    • See calls to bsim.setCamera in BatchedSimTest.cpp. You may need to modify these to get a reasonable camera for use in the viewer.
  8. Iterate on your collision geometry and gripper in the BatchedSimTest viewer.
    • Collsion JSON gripper and links sections above require careful authoring of many collision spheres.
    • We recommend you run the BatchedSimTest viewer while editing your collision JSON file, to rapidly iterate. Collision spheres are visible in ./latest_debugenv0.bmp while using the viewer. Press F5 to hot-reload the collecion JSON file and update this bitmap.
  9. Author your camera sensor properties: field-of-view and robot attachment.
    • Galactic supports a single RGB+depth camera attached to a link of your robot.
    • These aren't specified in a data file but are instead specified in Python at runtime. See calls to BatchedSimulator.set_camera in sim_test.py and batched_env.py.
    • There is currently a bug in how FOV is handled that can result in improper frustum culling. We recommend using 60 degrees as the FOV.

Known Issues

  • In BatchedSimulator.cpp, globalFixup and rootTransforms[b] include a hard-coded 90-degree rotation about the X axis. These are runtime hacks for the Fetch 3D models, which were improperly preprocessed. These hacks should be removed when integrating other robots. We'll generalize this as a config param soon.
  • For some robots, rootTransforms[b] may need an additional Y offset so that the robot appears at the correct floor height.

Converting Habitat 2.0 episodes

Galactic can convert Habitat 2.0 rearrangement episode datasets by converting the Habitat 2.0 JSON file into a Galactic compatible format. In the paper, we converted the Tidy House dataset from the Home Assistant Benchmark and used this for training in Galactic. To convert a Habitat 2.0 dataset to Galactic format run the following command:

python scripts/gala_dataset_conversion.py --in-path path_to_hab2_dataset.json.gz --out-path output_gala_dataset.episode_set.json --cfg-path path_to_hab2_task_config.yaml --ref-gala-episode-set-path data/tidy_house_10ep.episode_set.json

BatchedSimTest interactive viewer

This is a hacky interactive viewer built as a console unit test. It's intended to be used on a desktop Linux system. When run in the terminal, it blocks and waits for keyboard input, for example, use WASD to move the robot. After every keypress, it writes a new image to ./latest_env0.bmp (but it doesn't render this to the screen!). You should open this image in an image-viewer program that supports hot-reload. Position the image-viewer window alongside your terminal app--in this way, you get something similar to an interactive GUI 3D app.

There are keys to move the robot base and some of the Fetch arm links. For full keyboard controls, see BatchedSimTest.cpp L357.

There are some options starting at L192 (edit the source and rebuild as necessary). If you set doSaveAllFramesForVideo to true, the program will save a sequence of bitmaps, which can be made into a video using utils/make_video_from_image_files.py.

By default, a debug view is rendered to ./latest_debugenv0.bmp, which you should also have open.