Data Format Conventions for RGB, Depth, Timestamps, and Camera Poses in LingBot-Map

LingBot-Map unifies diverse SLAM datasets under a single convention: timestamp-named RGB/depth files, floating-point seconds for temporal data, and 4×4 camera-to-world matrices built from scalar-last quaternions.

The LingBot-Map repository standardizes input formats across benchmarks including TUM, KITTI, DROID-W, and Oxford-Spires. Mastering these data format conventions for RGB, depth, timestamps, and camera poses enables seamless integration of custom sequences and consistent evaluation across mapping algorithms.

On-Disk File Layout

The repository expects a specific directory hierarchy where filenames encode temporal alignment.

RGB Images

RGB frames are stored as 8-bit unsigned integer arrays in either PNG or JPEG format. The canonical path is rgb/<timestamp>.png, where the filename represents the capture time in seconds (e.g., 1305031452.791720.png). Images retain their original resolution (commonly 640×480) and are resized to the target load_img_size during loading. According to the source code in benchmark/datasets/base.py, the loader returns images as HxWx3 uint8 arrays.

Depth Maps

Depth data uses the OpenEXR format (.exr) to preserve floating-point precision. Located at depth/<timestamp>.exr, these files contain 32-bit float values representing metric distances in meters. The timestamp in the filename exactly matches the corresponding RGB frame, ensuring one-to-one correspondence without additional lookup tables.

Optional Binary Masks

Area-of-interest masks follow the same timestamp convention at mask/<timestamp>.png. These 8-bit binary images indicate valid regions for evaluation and mapping.

Timestamp Representation and Association

Internally, timestamps are handled as float64 NumPy arrays representing Unix time in seconds. The BaseDataset.get_timestamps() method returns this array, which drives the temporal association logic.

For ground-truth alignment, the system employs nearest-neighbor matching with dataset-specific tolerances. In benchmark/datasets/tum.py, the ASSOC_TOLERANCE constant is set to 0.02 seconds. The association logic in TumDataset.load_frame_data searches for the ground-truth pose with the minimum temporal distance:

gt_idx = _nearest_index(gt_ts, rgb_ts)
if gt_idx is not None and abs(gt_ts[gt_idx] - rgb_ts) <= ASSOC_TOLERANCE:
    pose = data['gt_poses'][gt_idx].astype(np.float32)

Frames lacking a pose within tolerance receive pose = None, and the benchmark writes a NaN row to the trajectory file.

Camera Pose Format

Camera poses are stored as 4×4 camera-to-world (C2W) transformation matrices using the float64 dtype during storage, though loaders typically cast to np.float32 after loading.

Ground Truth File Structure

Ground-truth files (e.g., groundtruth.txt or traj_gt.txt) contain space-delimited lines with the format:

timestamp tx ty tz qx qy qz qw

Where (tx, ty, tz) represents the translation vector and (qx, qy, qz, qw) is a scalar-last quaternion. The parsers in benchmark/datasets/tum.py and related modules convert these to rotation matrices using scipy.spatial.transform.Rotation.from_quat, then construct the 4×4 matrix:

c2w = np.eye(4, dtype=np.float64)
c2w[:3, :3] = R
c2w[:3, 3] = [tx, ty, tz]

Coordinate System Convention

The repository assumes the OpenCV/ROS coordinate convention where the camera Z-axis points forward, Y-axis points down, and X-axis points right. This C2W matrix transforms points from camera coordinates to world coordinates.

Camera Intrinsics

Intrinsic parameters are stored in intrinsics.txt or hard-coded per dataset. Each line contains seven float values:

timestamp fx fy cx cy width height

The pinhole parameters (fx, fy, cx, cy) are scaled automatically when images are resized to the target resolution. Dataset-specific loaders implement this scaling in their _load_scene_data methods.

Implementation in Code

Pose Association Logic

The TumDataset class in benchmark/datasets/tum.py implements the authoritative association pattern. It builds timestamp arrays for both RGB frames and ground-truth poses, then performs vectorized nearest-neighbor searches. The kitti.py and droid_w.py modules adapt this pattern for their respective timestamp formats (KITTI uses times.txt while DROID-W uses Unix timestamps in JPEG filenames).

File Loading Utilities

The BaseDataset class in benchmark/dataset/base.py provides shared utilities for image loading, timestamp extraction, and intrinsics scaling. Depth loading leverages OpenCV's EXR support via cv2.imread with cv2.IMREAD_UNCHANGED to preserve the 32-bit float depth values.

Practical Code Examples

Loading a TUM Frame with RGB and Pose

from benchmark.datasets.tum import TumDataset

# Initialize dataset (root points to unpacked TUM folder)

ds = TumDataset(raw_data_root="/data/TUM-RGBD", load_img_size=518)

scene = ds.get_scenes()[0]          # e.g., "rgbd_dataset_freiburg1_desk"

frame = ds.load_frame_data(scene, frame_id=0)

rgb_image = frame["rgb"]            # HxWx3 uint8, resized to target width

intrinsics = frame["intrinsics"]    # [fx, fy, cx, cy] scaled to new resolution

pose_matrix = frame["pose"]         # 4x4 C2W matrix or None if unassociated

Reading Depth Data by Timestamp

import cv2
from pathlib import Path

def load_depth(scene_dir: Path, timestamp: float) -> np.ndarray:
    depth_path = scene_dir / "depth" / f"{timestamp:.6f}.exr"
    # OpenEXR read via cv2 preserves float32 meter values

    depth = cv2.imread(str(depth_path), cv2.IMREAD_UNCHANGED)
    return depth  # Shape: HxW, dtype: float32, units: meters

Converting Poses to ROS-compatible Format

import numpy as np
from scipy.spatial.transform import Rotation

def pose_to_ros(pose):
    """Convert 4x4 C2W matrix to ROS position and quaternion."""
    position = pose[:3, 3]
    rot = Rotation.from_matrix(pose[:3, :3])
    # ROS uses scalar-last (x, y, z, w) format

    quaternion = rot.as_quat()  # Already returns [qx, qy, qz, qw]

    return position, quaternion

Writing Benchmark Trajectory Files

def write_trajectory(timestamps, poses, out_path):
    """Write trajectory in LingBot-Map format: timestamp + flattened 4x4 matrix."""
    with open(out_path, "w") as f:
        for ts, pose in zip(timestamps, poses):
            if pose is None:
                continue
            # Flatten first 3 rows (rotation + translation) row-major

            flat_pose = pose[:3, :].reshape(-1)
            row = [ts] + flat_pose.tolist()
            f.write(" ".join(f"{v:.6f}" for v in row) + "\n")

Summary

• RGB images use 8-bit PNG/JPEG files named with timestamps in seconds, located in rgb/ directories.
• Depth maps are 32-bit float OpenEXR (.exr) files in depth/ directories, storing metric distances.
• Timestamps are handled as float64 seconds and associated with ground-truth poses using nearest-neighbor matching with tolerance (typically 0.02s).
• Camera poses are stored as 4×4 C2W matrices derived from scalar-last quaternions (qx, qy, qz, qw) and translation vectors in ground-truth text files.
• Intrinsics follow the 7-value format timestamp fx fy cx cy width height and are automatically scaled during image resizing.

Frequently Asked Questions

What file format does LingBot-Map use for depth images?

LingBot-Map stores depth maps as single-channel OpenEXR (.exr) files containing 32-bit floating-point values. These files reside in the depth/ subdirectory and use the same timestamp-based filenames as their corresponding RGB images. The format preserves sub-millimeter precision necessary for accurate 3D reconstruction.

How are timestamps matched between RGB frames and ground-truth poses?

The association uses a nearest-neighbor search with a dataset-specific tolerance constant. For TUM sequences, the ASSOC_TOLERANCE is set to 0.02 seconds in benchmark/datasets/tum.py. The loader finds the ground-truth pose with the closest timestamp; if the absolute difference exceeds the tolerance, the frame receives a None pose.

What coordinate convention does the pose matrix follow?

Poses are 4×4 camera-to-world (C2W) matrices that follow the OpenCV convention. The rotation component transforms from camera to world coordinates, while the translation vector represents the camera origin in world space. The ground-truth files store rotations as scalar-last quaternions (qx, qy, qz, qw), which the loader converts to rotation matrices via scipy.spatial.transform.Rotation.

Can I use custom intrinsics with resized images?

Yes. The BaseDataset class automatically scales intrinsics when load_img_size differs from the original image dimensions. The scaling factor is applied to the focal lengths (fx, fy) and principal point (cx, cy) based on the ratio between the target width and the original width specified in the intrinsics file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}