RGB-objects 🛑🟩🔷 for robotic manipulation

January 22, 2023 · View on GitHub

This folder provides the RGB-objects introduced in the paper Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes and related code used by the RGB-Stacking environment.

The RGB-objects family

The RGB-objects are all obtained by applying a certain deformation to a cube, which is our seed object, referred to as s0 in code. We have defined four major axes of deformation of the seed object, which result in different shapes. These shapes can also be thought of as the vertical extrusion of a 2D shape, which is also the name of each axis. The names of the axes in code are numerical:

Axis 2 - Polygon: deformation obtained by transforming the extruded planar shape (i.e. the square) into a regular polygon.
Axis 3 - Trapezoid: deformation obtained by progressively morphing the planar square to a isosceles trapezoid.
Axis 5 - Parallelogram: deformation obtained by changing the orientation of the extrusion axis, from vertical (i.e. orthogonal to the plane of the planar shape) to progressively more slanted axes.
Axis 6 - Rectangle: deformation by uniformly scaling the object along the X, Y or Z-axis. Note that the x-, y-,z- Rectangle axes are the same so we refer to these as a single major Rectangle axis.

These deformations and their combinations define a parametric family of objects. For our Skill Generalization task (see the paper) we designate the axes of all pairwise combined deformations as the "training axes" and the ones of single deformation - the major axes above - as the "held-out axes". Pairwise mixing of some of the major axes leads to objects that are duplicates and therefore we omitted certain axes and objects.

Based on the resulting 15 axes, we created a training object set, which consists of 103 different shapes, and a held-out set. The diagram and gifs below shows a depiction of all the objects in the benchmark. The seed object is at the center; all the other objects are the result of deformations of this cube. The held-out objects (major axes) are enclosed in the upper teal sector of the diagram; the training objects (pairwise mixing of two major axes) are enclosed in the blue sector. Some objects cannot be grasped with a parallel gripper with 85mm aperture (i.e. the Robotiq 2F-85); these objects are transparent and were omitted in our experiments.

disk of objects

STL assets

Under the meshes directory there are 152 STL files representing the RGB-object dataset.

Assets directories structure

All mesh assets are located in the meshes directory where they are further split between following subdirectories:

train directory with 103 objects.
test_triplets directory with 13 objects including the seed s0, to form 5 test triplets.
heldout directory with 36 objects.

File name convention

All asset files are in STL format and have an *.stl extension. The file name starts with the object ID and is followed by a sequence of parameters that uniquely describe the object. For example, b3_sds4_shr48_drf0_hlw0_shx0_shy0_scx46_scy49_scz63.stl is the filename for the b3 object.

Visualizing STL assets

meshlab is a convenient tool to work with the provided meshes. To load a mesh from a command line:

$ cd props/rgb_objects/assets/rgb_v1.3/meshes
$ meshlab test_triplets/b3_sds4_shr48_drf0_hlw0_shx0_shy0_scx46_scy49_scz63.stl

A map of the object IDs is provided here:

Held-Out Axis 2, Polygon:

Held-Out Axis 3, Trapezoid:

Held-Out Axis 5, Parallelogram:

Held-Out Axis 6, Rectangle:

Training Axis 23, Polygon & Trapezoid:

Training Axis 25, Polygon & Parallelogram:

Training Axis 26, Polygon & Rectangle:

Training Axis 35, Trapezoid & Parallelogram:

Training Axis 36, Trapezoid & x-Rectangle:

Training Axis 37, Trapezoid & y-Rectangle:

Training Axis 38, Trapezoid & z-Rectangle:

Training Axis 56, Parallelogram & x-Rectangle:

Training Axis 57, Parallelogram & y-Rectangle:

Training Axis 58, Parallelogram & z-Rectangle:

Training Axis 67, x-Rectangle & y-Rectangle:

The abbreviations used for the parameters that define each object are explained below:

sds: integer scalar [-]. The number of edges used to generate the regular 2D shape. By default, every 2D shape is a regular one, which means that it has equal sides and equal angles.
shr: real-valued [%]. The percentage of 'shrinking' applied to the 2D shape. In the 2D plane, the shape is y-shrunk along the x-axis according to the formula: $y_{new} = (1 - shr \cdot x)*y_{old}$
drf: real-valued [deg]. The amount of draft used in the extrusion. Not used in this dataset, drf=0 for every object.
hlw: real-valued [%]. The percentage of material to be removed from each of the generated faces of the solid (to make it hollow). In this dataset, no hollow object was created by this transformation, hlw=0 for every object.
shx: real-valued [deg]. The solid angle of the extrusion axis along the x-axis.
shy: real-valued [deg]. The solid angle of the extrusion axis along the y-axis.
scx: real-valued [mm]. The global x-scale.
scy: real-valued [mm]. The global y-scale.
scz: real-valued [mm]. The global z-scale.

The values of these parameters are constrained in such a way that each valid RGB-shape has a one-to-one mapping with the associated parameters.

Test Triplets

We also provide the 5 fixed test triplets we have used during evaluation in our work.

rgb_test_triplet1: ('r3', 's0', 'b2')
rgb_test_triplet2: ('r5', 'g2', 'b3')
rgb_test_triplet3: ('r6', 'g3', 'b5')
rgb_test_triplet4: ('s0', 'g5', 'b6')
rgb_test_triplet5: ('r2', 'g6', 's0')

Usage example

RGB-objects are implemented in props/rgb_objects package, can be initialized and used as following.

from dm_robotics.manipulation.props.rgb_objects import rgb_object

 color_set = [
      [1, 0, 0, 1],  # RED
      [0, 1, 0, 1],  # GREEN
      [0, 0, 1, 1],  # BLUE
  ]

# Only a name of the object and its color are required. Optional parameters
# include scaling, mass etc.
prop_1 = rgb_object.RgbObjectProp(obj_id='r3', color=color_set[0])
prop_2 = rgb_object.RgbObjectProp(obj_id='g2', color=color_set[1])
prop_3 = rgb_object.RgbObjectProp(obj_id='b1', color=color_set[2])

# A list of all available objects could be accessed with
ids_list = rgb_object.RGB_OBJECTS_FULL_SET

Objects could be created individually as in the above snippet. Alternatively, some object triplets have been defined in rgb_object.PROP_TRIPLETS.

object_set = 'rgb_test_triplet1'
_, obj_ids = rgb_object.PROP_SETS[object_set]
task_props = []
for i, obj_id in enumerate(objects):
  prop = rgb_object.RgbObjectProp(
        obj_id=obj_id, color=color_set[i], name=f'rgb_object_{obj_id}')
  task_props.append(prop)

Citing

If you use rgb_objects in your work, please cite the accompanying paper:

@inproceedings{lee2021rgbstacking,
    title={Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapes},
    author={Alex X. Lee and
            Coline Devin and
            Yuxiang Zhou and
            Thomas Lampe and
            Konstantinos Bousmalis and
            Jost Tobias Springenberg and
            Arunkumar Byravan and
            Abbas Abdolmaleki and
            Nimrod Gileadi and
            David Khosid and
            Claudio Fantacci and
            Jose Enrique Chen and
            Akhil Raju and
            Rae Jeong and
            Michael Neunert and
            Antoine Laurens and
            Stefano Saliceti and
            Federico Casarini and
            Martin Riedmiller and
            Raia Hadsell and
            Francesco Nori},
    booktitle={Conference on Robot Learning (CoRL)},
    year={2021},
    url={https://openreview.net/forum?id=U0Q8CrtBJxJ}
}