# Data Format Conventions for RGB, Depth, Timestamps, and Camera Poses in LingBot-Map

> Explore LingBot-Map data format conventions for RGB, depth, timestamps, and camera poses. Learn about timestamp-named files, scalar-last quaternions, and camera-to-world matrices.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: api-reference
- Published: 2026-07-02

---

**LingBot-Map unifies diverse SLAM datasets under a single convention: timestamp-named RGB/depth files, floating-point seconds for temporal data, and 4×4 camera-to-world matrices built from scalar-last quaternions.**

The LingBot-Map repository standardizes input formats across benchmarks including TUM, KITTI, DROID-W, and Oxford-Spires. Mastering these data format conventions for RGB, depth, timestamps, and camera poses enables seamless integration of custom sequences and consistent evaluation across mapping algorithms.

## On-Disk File Layout

The repository expects a specific directory hierarchy where filenames encode temporal alignment.

### RGB Images

RGB frames are stored as 8-bit unsigned integer arrays in either PNG or JPEG format. The canonical path is `rgb/<timestamp>.png`, where the filename represents the capture time in seconds (e.g., `1305031452.791720.png`). Images retain their original resolution (commonly 640×480) and are resized to the target `load_img_size` during loading. According to the source code in [`benchmark/datasets/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/base.py), the loader returns images as `HxWx3 uint8` arrays.

### Depth Maps

Depth data uses the OpenEXR format (`.exr`) to preserve floating-point precision. Located at `depth/<timestamp>.exr`, these files contain 32-bit float values representing metric distances in meters. The timestamp in the filename exactly matches the corresponding RGB frame, ensuring one-to-one correspondence without additional lookup tables.

### Optional Binary Masks

Area-of-interest masks follow the same timestamp convention at `mask/<timestamp>.png`. These 8-bit binary images indicate valid regions for evaluation and mapping.

## Timestamp Representation and Association

Internally, timestamps are handled as `float64` NumPy arrays representing Unix time in seconds. The `BaseDataset.get_timestamps()` method returns this array, which drives the temporal association logic.

For ground-truth alignment, the system employs nearest-neighbor matching with dataset-specific tolerances. In [`benchmark/datasets/tum.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/tum.py), the `ASSOC_TOLERANCE` constant is set to `0.02` seconds. The association logic in `TumDataset.load_frame_data` searches for the ground-truth pose with the minimum temporal distance:

```python
gt_idx = _nearest_index(gt_ts, rgb_ts)
if gt_idx is not None and abs(gt_ts[gt_idx] - rgb_ts) <= ASSOC_TOLERANCE:
    pose = data['gt_poses'][gt_idx].astype(np.float32)

```

Frames lacking a pose within tolerance receive `pose = None`, and the benchmark writes a NaN row to the trajectory file.

## Camera Pose Format

Camera poses are stored as 4×4 camera-to-world (C2W) transformation matrices using the `float64` dtype during storage, though loaders typically cast to `np.float32` after loading.

### Ground Truth File Structure

Ground-truth files (e.g., [`groundtruth.txt`](https://github.com/Robbyant/lingbot-map/blob/main/groundtruth.txt) or [`traj_gt.txt`](https://github.com/Robbyant/lingbot-map/blob/main/traj_gt.txt)) contain space-delimited lines with the format:

```

timestamp tx ty tz qx qy qz qw

```

Where `(tx, ty, tz)` represents the translation vector and `(qx, qy, qz, qw)` is a **scalar-last quaternion**. The parsers in [`benchmark/datasets/tum.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/tum.py) and related modules convert these to rotation matrices using `scipy.spatial.transform.Rotation.from_quat`, then construct the 4×4 matrix:

```python
c2w = np.eye(4, dtype=np.float64)
c2w[:3, :3] = R
c2w[:3, 3] = [tx, ty, tz]

```

### Coordinate System Convention

The repository assumes the OpenCV/ROS coordinate convention where the camera Z-axis points forward, Y-axis points down, and X-axis points right. This C2W matrix transforms points from camera coordinates to world coordinates.

## Camera Intrinsics

Intrinsic parameters are stored in [`intrinsics.txt`](https://github.com/Robbyant/lingbot-map/blob/main/intrinsics.txt) or hard-coded per dataset. Each line contains seven float values:

```

timestamp fx fy cx cy width height

```

The pinhole parameters `(fx, fy, cx, cy)` are scaled automatically when images are resized to the target resolution. Dataset-specific loaders implement this scaling in their `_load_scene_data` methods.

## Implementation in Code

### Pose Association Logic

The `TumDataset` class in [`benchmark/datasets/tum.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/tum.py) implements the authoritative association pattern. It builds timestamp arrays for both RGB frames and ground-truth poses, then performs vectorized nearest-neighbor searches. The [`kitti.py`](https://github.com/Robbyant/lingbot-map/blob/main/kitti.py) and [`droid_w.py`](https://github.com/Robbyant/lingbot-map/blob/main/droid_w.py) modules adapt this pattern for their respective timestamp formats (KITTI uses [`times.txt`](https://github.com/Robbyant/lingbot-map/blob/main/times.txt) while DROID-W uses Unix timestamps in JPEG filenames).

### File Loading Utilities

The `BaseDataset` class in [`benchmark/dataset/base.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/dataset/base.py) provides shared utilities for image loading, timestamp extraction, and intrinsics scaling. Depth loading leverages OpenCV's EXR support via `cv2.imread` with `cv2.IMREAD_UNCHANGED` to preserve the 32-bit float depth values.

## Practical Code Examples

### Loading a TUM Frame with RGB and Pose

```python
from benchmark.datasets.tum import TumDataset

# Initialize dataset (root points to unpacked TUM folder)

ds = TumDataset(raw_data_root="/data/TUM-RGBD", load_img_size=518)

scene = ds.get_scenes()[0]          # e.g., "rgbd_dataset_freiburg1_desk"

frame = ds.load_frame_data(scene, frame_id=0)

rgb_image = frame["rgb"]            # HxWx3 uint8, resized to target width

intrinsics = frame["intrinsics"]    # [fx, fy, cx, cy] scaled to new resolution

pose_matrix = frame["pose"]         # 4x4 C2W matrix or None if unassociated

```

### Reading Depth Data by Timestamp

```python
import cv2
from pathlib import Path

def load_depth(scene_dir: Path, timestamp: float) -> np.ndarray:
    depth_path = scene_dir / "depth" / f"{timestamp:.6f}.exr"
    # OpenEXR read via cv2 preserves float32 meter values

    depth = cv2.imread(str(depth_path), cv2.IMREAD_UNCHANGED)
    return depth  # Shape: HxW, dtype: float32, units: meters

```

### Converting Poses to ROS-compatible Format

```python
import numpy as np
from scipy.spatial.transform import Rotation

def pose_to_ros(pose):
    """Convert 4x4 C2W matrix to ROS position and quaternion."""
    position = pose[:3, 3]
    rot = Rotation.from_matrix(pose[:3, :3])
    # ROS uses scalar-last (x, y, z, w) format

    quaternion = rot.as_quat()  # Already returns [qx, qy, qz, qw]

    return position, quaternion

```

### Writing Benchmark Trajectory Files

```python
def write_trajectory(timestamps, poses, out_path):
    """Write trajectory in LingBot-Map format: timestamp + flattened 4x4 matrix."""
    with open(out_path, "w") as f:
        for ts, pose in zip(timestamps, poses):
            if pose is None:
                continue
            # Flatten first 3 rows (rotation + translation) row-major

            flat_pose = pose[:3, :].reshape(-1)
            row = [ts] + flat_pose.tolist()
            f.write(" ".join(f"{v:.6f}" for v in row) + "\n")

```

## Summary

- **RGB images** use 8-bit PNG/JPEG files named with timestamps in seconds, located in `rgb/` directories.
- **Depth maps** are 32-bit float OpenEXR (`.exr`) files in `depth/` directories, storing metric distances.
- **Timestamps** are handled as `float64` seconds and associated with ground-truth poses using nearest-neighbor matching with tolerance (typically 0.02s).
- **Camera poses** are stored as 4×4 C2W matrices derived from scalar-last quaternions (`qx, qy, qz, qw`) and translation vectors in ground-truth text files.
- **Intrinsics** follow the 7-value format `timestamp fx fy cx cy width height` and are automatically scaled during image resizing.

## Frequently Asked Questions

### What file format does LingBot-Map use for depth images?

LingBot-Map stores depth maps as single-channel OpenEXR (`.exr`) files containing 32-bit floating-point values. These files reside in the `depth/` subdirectory and use the same timestamp-based filenames as their corresponding RGB images. The format preserves sub-millimeter precision necessary for accurate 3D reconstruction.

### How are timestamps matched between RGB frames and ground-truth poses?

The association uses a nearest-neighbor search with a dataset-specific tolerance constant. For TUM sequences, the `ASSOC_TOLERANCE` is set to 0.02 seconds in [`benchmark/datasets/tum.py`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/datasets/tum.py). The loader finds the ground-truth pose with the closest timestamp; if the absolute difference exceeds the tolerance, the frame receives a `None` pose.

### What coordinate convention does the pose matrix follow?

Poses are 4×4 camera-to-world (C2W) matrices that follow the OpenCV convention. The rotation component transforms from camera to world coordinates, while the translation vector represents the camera origin in world space. The ground-truth files store rotations as scalar-last quaternions (`qx, qy, qz, qw`), which the loader converts to rotation matrices via `scipy.spatial.transform.Rotation`.

### Can I use custom intrinsics with resized images?

Yes. The `BaseDataset` class automatically scales intrinsics when `load_img_size` differs from the original image dimensions. The scaling factor is applied to the focal lengths `(fx, fy)` and principal point `(cx, cy)` based on the ratio between the target width and the original width specified in the intrinsics file.