SpatialQA_depthmap_instruction.md

SpatialBot uses depth information from sensors if possible. Convert depth information to millimeter unit so DepthAPI and SpatialBot can accurately query depth information.

Otherwise, we convert RGB to RGB-D with ZoeDepth and save depth information in uint16 by default, which covers 0-65535mm. (You can also do it in e.g. uint64 to cover more distance ranges).

After downloading ZoeDepth, please replace save_raw_16bit function in zoedepth/utils/misc.py with:

def save_raw_16bit(depth, fpath="raw.png"):
    if isinstance(depth, torch.Tensor):
        depth = depth.squeeze().cpu().numpy()
    assert isinstance(depth, np.ndarray), "Depth must be a torch tensor or numpy array"
    assert depth.ndim == 2, "Depth must be 2D"
    depth = depth * 1000
    depth = depth.astype(np.uint16)
    depth = Image.fromarray(depth)
    depth.save(fpath)

Use our codes to run ZoeDepth for a directory:

python zoe.py --img-dir </path/of/rgb> --save-dir </path/to/save/depth>