Data Format

NOTE: the data format and this documentation has been adapted from our NSAVP dataset.

All sensor (and pseudo ground truth) data is provided in the HDF5 file format (abbreviated H5). For an introduction to the H5 file format, we recommend this video series. We recommend using the HDFView GUI to view the contents of the H5 files. Commandline tools h5ls and h5dump can also be used to print the contents of the H5 files. See the Setup section of the conversion documentation page for instructions to build a Docker image that contains HDFView, h5ls, and h5dump. The provided conversion scripts demonstrate reading from and writing to H5 files in C++. To work with H5 files in Python we recommend h5py or PyTables. The TRNeRF codebase demonstrates reading from the H5 files with h5py.

The TRNeRF dataset includes 9 sequences:

• cam_calibration: A sequence for intrinsic and extrinsic calibration of the four cameras. In this sequence, the cameras are stationary while viewing the multi-spectral Aprilgrid calibration board as it is slowly moved in front of them.
• imu_noise_calibration: A sequence for intrinsic calibration of the IMU. This sequence contains 18 hours of data collected from the IMU while it was stationary on a damped surface.
• imu_cam_calibration: A sequence for extrinsic calibration between the IMU and the monochrome cameras. In this sequence, the data collection rig is moved in six degrees of freedom while viewing the stationary multi-spectral Aprilgrid calibration board.
• <speed>_<scene>: Six sequences with three different camera speeds (slow, medium, and fast) in two scenes (indoor and outdoor). At the start of each recording of the scene the data collection rig is moved in six degrees of freedom at a moderate speed to aid IMU initialization. Each sequence also includes two recordings of approximate hot and cold uniform temperature sources, captured before and after the recording of the scene. The recordings of these uniform temperature sources allow two-point non-uniformity corrections (NUCs) to be performed.

A summary of the files provided in the non-calibration (<speed>_<scene>) sequences is given in the table below:

Filename	Description	Sensor Model	Applicable Sequences
adk_left.h5	Left Thermal	FLIR 40640U050-6PAAX	All
adk_right.h5	Right Thermal	FLIR 40640U050-6PAAX	All
mono_left.h5	Left Monochrome	FLIR BFS-PGE-16S2M	All
mono_right.h5	Right Monochrome	FLIR BFS-PGE-16S2M	All
imu.h5	IMU	VectorNav VN-100	All
pseudo_ground_truth_poses.h5	Left Monochrome Pseudo Ground Truth Poses	NA	All
two_point_nuc_results_adk_left.h5	Left Thermal Two-Point NUC Results	NA	Slow
two_point_nuc_results_adk_right.h5	Right Thermal Two-Point NUC Results	NA	Slow
pseudo_ground_truth_adk_left.h5	Left Thermal Pseudo Ground Truth Images	NA	Medium/Fast
pseudo_ground_truth_adk_right.h5	Right Thermal Pseudo Ground Truth Images	NA	Medium/Fast

All data is precisely time synchronized: images captured simultaneously have identical timestamps and there is approximately zero time offset between the cameras and the IMU.

The pseudo ground truth poses and images cover only the subsets of the sequences used for training in the TRNeRF paper. See t_start_list and t_stop_list in the pseudo ground truth pose generation script for the exact timestamps at the beginning and end of the subsets used in each sequence.

The calibration sequences contain the subset of H5 files necessary to perform the calibration and the calibration results as YAML files. For example, the imu_cam_calibration sequence includes the files imu.h5, mono_left.h5, mono_right.h5 and transformation_imu_to_mono_left.yaml. The calibration result files are described in more detail below.

The cam_calibration sequence additionally includes the Aprilgrid configuration file aprilgrid.yaml, which describes properties of our multi-spectral Aprilgrid board. This file is required for performing calibration and also for computing the detection metric.

1. General H5 Format

All sensor data was originally collected as a ROS1 rosbag and was converted to H5 files using the following strategy:

• A H5 file is created for each topic's namespace with filename <namespace>.h5, with forward slashes replaced by underscores in nested namespaces (e.g. /mono_left/image_raw is written to mono_left.h5 and /adk/left/image_raw is written to adk_left.h5).
• H5 groups are created for each topic's name (e.g. the topic /mono_left/image_raw is written to the H5 group /image_raw in the mono_left.h5 file) and the message type of the topic is attached as a H5 scalar attribute to this group (e.g. the H5 group /image_raw has a H5 attribute named ros_message_type with value sensor_msgs/Image).
• Constant message fields are written as H5 scalar attributes attached to their corresponding H5 group (e.g. the height field of a sensor_msgs/Image message on the topic /mono_left/image_raw is written as a H5 attribute named height attached to the /image_raw H5 group).
• Variable message fields are written as H5 datasets in their corresponding H5 groups using descriptive names that do not necessarily match the message field name (e.g. the data field of a sensor_msgs/Image message on the topic /mono_left/image_raw is written to the H5 dataset /image_raw/images).
• Timestamps are written in nanoseconds as unsigned 64-bit integer H5 datasets.
• Where applicable, units or descriptors are written as H5 scalar attributes attached to H5 datasets (e.g. the H5 dataset /image_raw/timestamps will have an attribute named units with value nanoseconds).
• The first dimension of each H5 dataset indexes across messages and chunking is performed along the first dimension only. Additionally, any two chunked datasets from the same message have the same chunk size along the first dimension such that data from all message fields can be read in as full chunks.

The pseudo ground truth H5 files mimick the structure of the sensor data H5 files.

We provide a script for converting the H5 files back to the ROS1 rosbag format. See the section Converting H5 Files to a ROS1 Rosbag File in the conversion documentation page.

2. Specific H5 Format Details

2.1. Thermal Camera Data

The thermal camera data files, adk_left.h5 and adk_right.h5, contain the following:

• image_raw: A group, derived from sensor_msgs/Image messages, including the following:
  • height, width, encoding, is_bigendian, step: Attributes for constant messages fields.
  • images: A three dimensional dataset (messages x rows x columns) of raw thermal images. Note that the firmware version we use with our ADK cameras embeds telemetry information in the first four pixels of the first row in the image. After extracting this information we write the value of the fifth pixel to the first four.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.
• image_meta: A group, derived from image_meta_msgs/ImageMeta messages (a custom message type), including multiple one dimensional datasets of various image metadata parameters. In particular, this includes:
  • driver_timestamps: A one dimensional dataset of timestamps in nanoseconds. Note that each timestamp matches exactly with the image the metadata corresponds to.
  • ffc_flags: A one dimensional dataset of hexadecimal codes where the least significant digit indicates whether a shutter-based flat field correction (FFC), also called a non-uniformity correction (NUC), is not being performed (0x0000), imminent (0x0001), in progress (0x0002), or complete (0x0003). When the FFC is in progress (0x0002), the camera returns duplicates of the last image captured before the FFC. We have removed these duplicate images and their corresponding meta data. Note that shutter-based NUCs were disabled during the recording of the non-calibration (<speed>_<scene>) sequences.

The pseudo ground truth thermal image files, pseudo_ground_truth_adk_left and pseudo_ground_truth_adk_right, have only a single group image_rendered with the same structure as the image_raw group described above.

2.2. Monochrome Camera Data

The monochrome camera data files, mono_left.h5 and mono_right.h5, contain the following:

• image_raw: A group, derived from sensor_msgs/Image messages, including the following:
  • height, width, encoding, is_bigendian, step: Attributes for constant messages fields
  • images: A three dimensional dataset (messages x rows x columns) of raw monochrome images.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.
• image_meta: A group, derived from image_meta_msgs/ImageMeta messages (a custom message type), including multiple one dimensional datasets of various image metadata parameters. In particular, this includes:
  • driver_timestamps: A one dimensional dataset of timestamps in nanoseconds. Note that each timestamp matches exactly with the image the metadata corresponds to.
  • exposure_time_us: A one dimensional dataset of exposure times in microseconds.
  • gain: A one dimensional dataset of gain values in dB.

2.3. IMU Data

The IMU data file, imu.h5, contains the following:

• imu: A group, derived from sensor_msgs/Imu messages, including the following:
  • angular_velocities: A two dimensional dataset (messages x coordinate axes) of angular velocities in the IMU frame in radians/second. The order of the coordinate axes across the columns is x, y, z.
  • linear_accelerations: A two dimensional dataset (messages x coordinate axes) of linear accelerations in the IMU frame in meters/second^2. The order of the coordinate axes across the columns is x, y, z.
  • quaternions: A two dimensional dataset (messages x quaternion components) of quaternion rotations from the body frame to the North East Down (NED) frame. The order of the components across the columns is x, y, z, w. Note that this orientation is estimated onboard the VectorNav VN-100 via an EKF and may be unreliable.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.
• magnetic_field: A group, derived from sensor_msgs/MagneticField messages, including the following:
  • magnetic_fields: A two dimensional dataset (messages x coordinate axes) of magnetic field measurements in the IMU frame in Gauss. The order of the coordinate axes across the columns is x, y, z.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.
• fluid_pressure: A group, derived from sensor_msgs/FluidPressure messages, including the following:
  • fluid_pressures: A one dimensional dataset of pressure measurements in kilopascals.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.
• temperature: A group, derived from sensor_msgs/Temperature messages, including the following:
  • temperatures: A one dimensional dataset of temperature measurements in Celsius.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.

The angular velocities, linear accelerations, and magnetic field measurements are calibrated but uncompensated. That is, factory calibrations were applied but online bias estimates and hard/soft iron compensation were not applied.

Note that in the imu_noise_calibration sequence the imu.h5 file contains only the imu group.

2.4. Pseudo Ground Truth Pose Data

The pseudo ground truth data file, pseudo_ground_truth_poses.h5, contains transformations from the COLMAP world frame to the left monochrome camera frame. The COLMAP world frame is consistent across each sequence (slow, medium, and fast) within a given scene.

Specifically, the file contains the following:

• poses_mono_left: A group, derived from geometry_msgs/PoseStamped messages, including the following:
  • positions: A two dimensional dataset (messages x coordinate axes) of positions in meters. The order of the coordinates across the columns is x, y, z.
  • quaternions: A two dimensional dataset (messages x quaternion components) of quaternion rotations. The order of the components across the columns is x, y, z, w.
  • timestamps: A one dimensional dataset of timestamps in nanoseconds.

2.5. Two-Point NUC Results

The two-point NUC result files, two_point_nuc_results_adk_left.h5 and two_point_nuc_results_adk_right.h5, contain the following:

• gain: A two dimensional dataset of pixelwise gains.
• offset: A two dimensional dataset of pixelwise offsets.

3. Calibration Results

There are three calibration result files:

• camchain.yaml: The result of applying Kalibr to the cam_calibration sequence. This file contains the intrinsic and extrinsic calibration results for all four cameras in the Kalibr camchain.yaml format.
• imu.yaml: The result of applying the allan_variance_ros package to the imu_noise_calibration sequence. This file contains the IMU intrinsic calibration results in the Kalibr imu.yaml format.
• transformation_imu_to_mono_left.yaml: The result of applying Kalibr to the imu_cam_calibration sequence. This file contains a single key transformation_imu_to_mono_left which gives the transformation from the IMU frame to the left monochrome camera frame. In combination with camchain.yaml a transformation between any two of the sensor frames can be obtained.