How to Integrate the ONNX Sky Segmentation Model into the Ling-Bot-Map Reconstruction Pipeline

The ONNX sky segmentation model integrates into the Ling-Bot-Map reconstruction pipeline by generating binary masks that zero-out confidence scores for sky pixels, preventing depth-less sky regions from contaminating the dense 3-D point cloud.

Ling-Bot-Map is an open-source framework for dense 3-D reconstruction from video sequences. During reconstruction, each frame produces a confidence map (conf) indicating the reliability of depth estimates. Since sky regions contain no valid depth information, the pipeline optionally runs an ONNX sky-segmentation model to detect and suppress these pixels before final point cloud generation.

Architecture Overview

The sky-segmentation logic is woven into three primary visualization modules. Each entry point checks the mask_sky flag and delegates processing to the core segmentation utilities.

Prediction Preprocessing

In lingbot_map/vis/viser_wrapper.py, the viser_wrapper() function prepares depth-based point clouds for visualization. When mask_sky=True, it invokes apply_sky_segmentation(conf, image_folder) to filter the confidence map before rendering (lines 82-84).

Interactive Viewer

The PointCloudViewer class in lingbot_map/vis/point_cloud_viewer.py processes prediction dictionaries via _process_pred_dict(). If sky masking is enabled, the confidence map is filtered through apply_sky_segmentation() before the interactive Viser UI launches (lines 73-79).

Sky-Mask Generation Core

The central orchestration happens in lingbot_map/vis/sky_segmentation.py. The apply_sky_segmentation() function (lines 373-426) coordinates model loading, mask caching, and confidence-map multiplication. It loads cached masks or generates new ones via run_skyseg(), then multiplies the binary sky mask with the original confidence scores to zero-out sky regions.

Data Flow Pipeline

The integration follows a strict preprocessing sequence:

1. Input Acquisition: RGB frames are read from image_folder, and depth inference produces a raw point cloud with confidence map conf of shape (S, H, W).

2. Mask Retrieval: apply_sky_segmentation() calls load_or_create_sky_masks() (lines 210-294), which checks for cached masks under <image_folder>_sky_masks. If missing, it executes run_skyseg() on each image.

3. ONNX Inference: The run_skyseg() function (lines 46-72) loads the model via onnxruntime.InferenceSession, preprocesses the BGR image, and forwards it through the model to generate a sky-score map (higher values indicate sky). The output is normalized to an 8-bit score map.

4. Confidence Suppression: The sky-score map is inverted to a non-sky confidence map and thresholded (sky_mask_binary > _SKYSEG_SOFT_THRESHOLD). The original confidence map is multiplied element-wise: conf = conf * sky_mask_binary, setting sky-region scores to zero.

5. Point Cloud Filtering: Subsequent filtering steps discard points with zero confidence, ensuring the final reconstruction contains no sky artifacts.

Implementation Details

Model Caching Strategy

To avoid redundant inference, load_or_create_sky_masks() implements a disk-based cache. Masks are stored as binary images in a parallel directory structure. If the cache exists, the function reads masks directly from disk; otherwise, it triggers ONNX inference and persists the results for future runs.

Automatic Model Download

The download_skyseg_model() helper (lines 430-556) ensures the ONNX file is available at runtime. If skyseg.onnx is missing, the function automatically downloads it from HuggingFace, eliminating manual setup requirements.

ONNX Runtime Configuration

The model session is initialized once per pipeline execution. The run_skyseg() function handles BGR-to-RGB conversion, resizing to _SKYSEG_INPUT_SIZE, and normalization before feeding tensors to the onnxruntime inference session.

Code Examples

Enable Sky Masking in the Interactive Viewer

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

# pred_dict contains images, depth maps, and raw confidence

viewer = PointCloudViewer(
    pred_dict=pred_dict,
    mask_sky=True,                # Enable ONNX sky segmentation

    image_folder="data/images",   # Source RGB frames for mask generation

    port=8080,
)
viewer.animate()  # Launches Viser UI with sky-filtered point cloud

Stand-Alone Confidence Map Filtering

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# conf shape: (S, H, W) from depth inference

masked_conf = apply_sky_segmentation(
    conf,
    image_folder="data/images",
    skyseg_model_path="skyseg.onnx",  # Optional: auto-downloaded if omitted

)

Direct ONNX Inference (Advanced)

import onnxruntime
import cv2
from lingbot_map.vis.sky_segmentation import run_skyseg, _SKYSEG_INPUT_SIZE

# Initialize session once

session = onnxruntime.InferenceSession("skyseg.onnx")

# Load BGR image

img = cv2.imread("data/images/frame_000001.jpg")

# Generate sky-score map (uint8, higher values = sky)

sky_score = run_skyseg(session, _SKYSEG_INPUT_SIZE, img)

Summary

• Integration Points: The ONNX model is invoked in viser_wrapper.py and point_cloud_viewer.py when mask_sky=True, delegating to sky_segmentation.py for core logic.
• Caching Mechanism: Masks are cached under <image_folder>_sky_masks to avoid re-running inference on identical frames.
• Confidence Modulation: The pipeline multiplies the binary sky mask with the confidence map, effectively zeroing out sky pixels before point cloud generation.
• Automatic Setup: The model downloads automatically from HuggingFace if not present locally, using download_skyseg_model().

Frequently Asked Questions

How does the pipeline handle missing ONNX model files?

If skyseg.onnx is not found in the working directory, the download_skyseg_model() function automatically downloads the pre-trained weights from HuggingFace. This ensures the reconstruction pipeline works without manual model procurement.

Can I disable sky segmentation for specific frames?

Yes. Sky segmentation is optional and controlled by the mask_sky boolean parameter. Setting mask_sky=False skips the apply_sky_segmentation() call entirely, preserving the original confidence map values for all frames.

What is the performance impact of running the ONNX model?

The model runs once per unique frame and caches results to disk. Subsequent reconstructions using the same image_folder read masks from cache, reducing the overhead to simple disk I/O. The initial inference uses onnxruntime for optimized CPU or GPU execution.

Which file contains the threshold for sky detection?

The soft threshold for binarizing the sky-score map is defined as _SKYSEG_SOFT_THRESHOLD in lingbot_map/vis/sky_segmentation.py. The apply_sky_segmentation() function applies this threshold to convert the continuous sky-score map into a binary mask before confidence modulation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}