How BSSLoader Handles Image Loading and Resizing in lingbot-map

BSSLoader automatically rescales RGB, depth, and mask images using a ResizeContext when loading data from BSS-formatted directories, ensuring all modalities share consistent geometric transforms and updated camera intrinsics.

The BSSLoader class in the lingbot-map repository serves as the central data access layer for benchmark datasets stored in BSS format. When processing robotics perception data, developers need to understand how this loader handles image loading and resizing to maintain geometric accuracy across RGB frames, depth maps, and segmentation masks. The implementation couples file I/O with optional spatial transformations, allowing batch processing with consistent camera models.

BSSLoader Architecture and Initialization

The loader acts as a bridge between raw BSS artifacts and processed tensors, orchestrating reads from RGB, depth, and mask directories while applying unified geometric transforms.

Constructor and ResizeContext Storage

When instantiating BSSLoader, the class stores a BSSArtifact reference and an optional ResizeContext parameter. According to the source code in benchmark/benchmark/core/loader.py (lines 30–46), if resize_context is None or its mode is set to 'none', the loader returns images at their native resolution without modification.

loader = BSSLoader(artifact, resize_context=resize_ctx)  # resize_ctx may be None

This design allows the same loader instance to toggle between high-resolution analysis and downsampled benchmarking simply by changing the context parameter.

Step-by-Step Image Loading Flow

The loading process follows a deterministic pipeline that queries metadata, iterates over frames, and conditionally applies resizing before returning lists of arrays.

Frame Discovery via Metadata

Before loading pixels, get_num_frames() reads the .complete.json metadata file to determine the valid frame indices for iteration. This method, found at lines 56–74 in benchmark/benchmark/core/loader.py, ensures that the loader only attempts to read existing files and establishes the iteration range for subsequent list-loading methods.

RGB Image Loading and Resizing

The load_rgb_list() method (lines 71–100) constructs file paths using zero-padded indexing (f"{idx:06d}.png") and delegates actual file reading to load_rgb(), a thin wrapper around cv2.imread. When a ResizeContext is present and enabled, each loaded image is passed to ResizeContext.apply() before being appended to the result list.

This guarantees that all RGB frames in the returned list share identical dimensions defined by the resize parameters.

Depth Map Loading with Nearest-Neighbor Interpolation

Depth data requires special handling to preserve metric precision. The load_depth_list() method (lines 106–130) loads .exr files via load_exr() and, when resizing is active, invokes ResizeContext.apply_nearest() instead of the standard apply() method.

Nearest-neighbor interpolation is critical here because linear or cubic interpolation would introduce invalid depth values at object boundaries, corrupting subsequent 3D back-projection calculations.

Mask Loading and Boolean Conversion

For segmentation masks, load_mask_list() (lines 91–115) reads PNG files and converts them to boolean arrays. Like depth maps, masks use ResizeContext.apply_nearest() to avoid generating fractional opacity values that could break downstream boolean logic or contour detection algorithms.

ResizeContext Internals and Transform Pipeline

The geometric transformation logic resides in benchmark/benchmark/geometry/resize.py, where ResizeContext encapsulates scaling factors, crop offsets, and canvas dimensions.

Lazy Transform Computation

ResizeContext.apply() lazily initializes a ResizeTransform object via _compute_resize_transform() the first time an image of a specific size is encountered. This cached transform stores the mathematical operations required to map the original image dimensions to the target size, accounting for aspect ratio preservation and alignment constraints (e.g., align=4 for tensor-friendly dimensions).

The Three-Step Resize Pipeline

As implemented in lines 64–78 of resize.py, _apply_resize_transform() executes a rigid three-step sequence:

1. Resize – Scale the image using the computed factors
2. Crop – Remove pixels if the scaled image exceeds the target canvas
3. Place – Position the result on a zero-filled canvas of the exact target size

This pipeline ensures consistent results regardless of aspect ratio differences between the source data and target resolution.

Intrinsics Adjustment for Geometric Consistency

When camera calibration data is loaded via load_intrinsics() (lines 145–158 in loader.py), the loader automatically invokes adjust_intrinsics() using the same ResizeTransform applied to the images. This updates the focal lengths and principal point coordinates to match the resized image coordinates, preserving the mathematical relationship between pixel space and camera space.

Without this adjustment, 3D reconstruction algorithms would use incorrect projection matrices, leading to skewed point clouds and inaccurate pose estimation.

Practical Code Examples

The following patterns demonstrate how to configure BSSLoader for different processing scenarios:

from benchmark.core.storage import BSSArtifact
from benchmark.core.loader import BSSLoader
from benchmark.geometry.resize import ResizeContext

# Load RGB images at native resolution (no resizing)

artifact = BSSArtifact("/path/to/bss_dataset")
loader = BSSLoader(artifact)                     # resize_context is None → no scaling

rgb_images = loader.load_rgb_list()            # List[ndarray] of shape (H, W, 3)

# Load images scaled to fit an area budget (e.g., max 320×240 pixels)

resize_ctx = ResizeContext(mode="area_budget", area_budget=76800, align=4)
loader = BSSLoader(artifact, resize_context=resize_ctx)
rgb_images = loader.load_rgb_list()            # Resized RGB

depth_maps = loader.load_depth_list()          # Resized depth (nearest-neighbor)

# Load adjusted intrinsics for the same resize transform

intrinsics = loader.load_intrinsics()            # Already corrected for the resize

Summary

• BSSLoader couples file I/O from benchmark/benchmark/core/loader.py with the geometric transforms defined in benchmark/benchmark/geometry/resize.py to produce consistently sized output.
• ResizeContext applies to RGB via interpolation-based scaling, but uses nearest-neighbor for depth and masks to preserve data integrity.
• Camera intrinsics are automatically adjusted using the same ResizeTransform, ensuring that 3D geometry remains valid after downsampling.
• The loader determines frame counts by parsing .complete.json metadata, preventing attempts to load non-existent files.
• When resize_context is None or mode is 'none', images return at their native resolution with no computational overhead.

Frequently Asked Questions

What image formats does BSSLoader support?

BSSLoader supports PNG files for RGB images and masks (loaded via cv2.imread), and OpenEXR (.exr) files for depth data (loaded via load_exr() in benchmark/io/image.py). These formats preserve the necessary bit depth for computer vision tasks, with EXR specifically handling the floating-point precision required for metric depth.

Why does BSSLoader use nearest-neighbor interpolation for depth maps?

Nearest-neighbor interpolation prevents the introduction of interpolated depth values at object boundaries. Linear or cubic interpolation would create fractional depth values that do not correspond to real surfaces in the scene, corrupting 3D back-projection calculations and point cloud generation.

How does BSSLoader maintain geometric consistency when resizing images?

The loader maintains consistency by caching a ResizeTransform object that stores scaling factors and crop offsets. This same transform is applied to all modalities (RGB, depth, mask) and is also used to adjust camera intrinsics via adjust_intrinsics(), ensuring that the mathematical relationship between pixels and 3D space remains valid after resizing.

What happens if ResizeContext is set to 'none' mode?

When ResizeContext is initialized with mode='none' or passed as None to the BSSLoader constructor, the loader skips all transformation logic. Images return at their native resolution read directly from disk, and intrinsics remain unmodified, providing zero-overhead access to the original dataset.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}