How LingBot-Map's Sky Segmentation Masking Functions Within the Reconstruction Pipeline

LingBot-Map removes sky regions from depth-confidence maps using the apply_sky_segmentation helper, zeroing out sky pixel confidences before they enter the 3D reconstruction pipeline.

The Robbyant/lingbot-map repository implements a specialized sky segmentation masking function that filters unreliable sky measurements from depth-confidence tensors before reconstruction. This preprocessing step ensures that only geometrically valid, non-sky pixels contribute to the final 3D map. The implementation centers on the apply_sky_segmentation utility found in lingbot_map/vis/sky_segmentation.py.

Sky Segmentation Architecture and Mask Generation

The sky segmentation masking process follows a two-stage pipeline: mask generation (or cache retrieval) followed by confidence tensor filtering.

ONNX Model Inference and Caching

When cached masks are unavailable, load_or_create_sky_masks initializes the ONNX sky-segmentation model (skyseg.onnx), automatically downloaded from HuggingFace if missing. The system processes each frame through the inference chain segment_sky_from_array → run_skyseg, producing soft score maps where higher values indicate sky regions. The helper _result_map_to_non_sky_conf converts these raw outputs into non-sky confidence maps and rescales them to the target resolution. If a cached mask directory exists, the function bypasses inference and loads pre-computed 8-bit masks (.png files) directly.

Binary Masking and Confidence Filtering

apply_sky_segmentation constructs a binary mask by thresholding soft scores at _SKYSEG_SOFT_THRESHOLD (0.1). It then performs element-wise multiplication between this mask and the input confidence tensor conf of shape (S, H, W) (where S represents the number of frames). This operation zeroes all confidence values associated with sky pixels, ensuring downstream reconstruction components ignore unreliable depth measurements from the sky region.

Integration Points in the Reconstruction Pipeline

The sky segmentation masking function operates as a transparent preprocessing layer within LingBot-Map's visualization and reconstruction stack.

Visualization Wrapper Integration

In lingbot_map/vis/viser_wrapper.py, the confidence map undergoes sky masking immediately after model inference. This ensures that downstream reconstruction modules receive only reliable, sky-filtered measurements before entering the 3D mapping stage.

Point Cloud Viewer Protection

The masking function also protects the lingbot_map/vis/point_cloud_viewer.py module. By filtering sky pixels before point cloud generation, the system prevents infinite-depth sky artifacts from corrupting the geometric representation used for visualization and further processing.

Practical Implementation

The following example demonstrates how to apply sky segmentation masking to a confidence tensor before reconstruction:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# conf: (num_frames, H, W) confidence scores from the depth model

conf = np.load("confidence.npy")

# Mask sky using image files from a folder

conf_no_sky = apply_sky_segmentation(
    conf,
    image_folder="data/images",          # folder with the RGB images

    skyseg_model_path="skyseg.onnx",     # will be downloaded if absent

    sky_mask_dir="cache/sky_masks",      # where cached masks are stored

    sky_mask_visualization_dir="cache/sky_vis",  # optional visual checks

)

# conf_no_sky can now be fed to the reconstruction module

Key Source Files

• lingbot_map/vis/sky_segmentation.py – Core sky-segmentation utilities (apply_sky_segmentation, mask generation, ONNX inference).
• lingbot_map/vis/viser_wrapper.py – Applies sky masking to confidence maps before visualization and downstream processing.
• lingbot_map/vis/point_cloud_viewer.py – Uses the same masking step when rendering point clouds for inspection.

Summary

• Sky segmentation masking in LingBot-Map filters unreliable sky measurements from depth-confidence tensors using the apply_sky_segmentation utility.
• The system caches 8-bit masks as .png files and falls back to ONNX inference via skyseg.onnx when cached data is unavailable.
• A threshold of 0.1 (_SKYSEG_SOFT_THRESHOLD) converts soft segmentation scores into binary masks that zero out sky regions in the confidence tensor.
• Integration occurs in viser_wrapper.py and point_cloud_viewer.py, ensuring only non-sky pixels contribute to 3D reconstruction.

Frequently Asked Questions

What is the purpose of sky segmentation masking in LingBot-Map?

Sky segmentation masking eliminates unreliable depth measurements from sky regions that typically lack geometric features. By zeroing out confidence values for sky pixels in the (S, H, W) confidence tensor, the system prevents infinite-depth artifacts from corrupting the 3D reconstruction while preserving valid measurements from terrestrial objects.

How does LingBot-Map cache sky segmentation results?

The load_or_create_sky_masks helper checks for existing 8-bit mask .png files in the specified sky_mask_dir. If present, it loads these cached masks directly, bypassing ONNX model inference. If absent, the system runs inference on the input images, saves the results to the cache directory, and returns the newly generated masks.

What model does LingBot-Map use for sky segmentation?

LingBot-Map uses an ONNX format segmentation model stored as skyseg.onnx. The repository automatically downloads this model from HuggingFace if the file is not found locally. The model produces soft score maps that the system converts to binary masks using the 0.1 confidence threshold.

Where is the sky segmentation applied in the reconstruction pipeline?

The apply_sky_segmentation function is invoked in lingbot_map/vis/viser_wrapper.py immediately after confidence map computation, and in lingbot_map/vis/point_cloud_viewer.py before point cloud rendering. These integration points ensure that sky-filtered data flows into all downstream reconstruction and visualization components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}