How the Sky Masking Pipeline Works in LingBot-Map Using ONNX Segmentation

LingBot-Map removes sky regions from 3-D reconstructions by running an ONNX-based segmentation model to generate per-frame sky masks, caching them for reuse, and applying them to confidence tensors to zero-out sky pixels.

The LingBot-Map repository implements an efficient sky masking pipeline that integrates ONNX segmentation models directly into its 3-D reconstruction workflow. By automatically downloading model weights from Hugging Face and caching intermediate masks, the system ensures deterministic, reproducible results while filtering out sky contamination from geometry completion outputs.

Pipeline Architecture Overview

The sky masking pipeline operates in four distinct stages implemented in lingbot_map/vis/sky_segmentation.py. First, the system acquires the ONNX model file skyseg.onnx via automatic download if not present locally. Second, it initializes a reusable onnxruntime.InferenceSession for efficient batch processing. Third, it generates per-frame sky masks with intelligent caching to avoid redundant computation. Finally, it applies these masks to the geometry completion network's confidence scores using a soft threshold to exclude sky pixels from the reconstruction.

Model Acquisition and ONNX Session Initialization

If the skyseg.onnx model file is missing from the local filesystem, the download_skyseg_model() function fetches it from Hugging Face using streaming requests and stores it locally. This ensures the pipeline works out-of-the-box without manual weight downloads.

Once the model file is available, the load_or_create_sky_masks() function creates an onnxruntime.InferenceSession object that persists for the entire pipeline execution. This session reuse eliminates the overhead of repeatedly loading the model into memory when processing video sequences or large image batches.

Per-Frame Mask Generation and Caching

The load_or_create_sky_masks() function handles mask generation through a sophisticated caching mechanism that supports both image arrays and file-based inputs. For each frame, the system first checks the cache directory (conventionally named <image_folder>_sky_masks) for an existing mask file. If a valid cache entry exists with matching dimensions, the pipeline loads the precomputed mask directly; otherwise, it proceeds with ONNX inference.

Input Preprocessing and ONNX Inference

When generating new masks, the pipeline processes images through the segment_sky_from_array() or segment_sky() helper functions. The input frame is rescaled to the model's expected input size defined by _SKYSEG_INPUT_SIZE = (320, 320). The preprocessing pipeline converts the image to RGB, normalizes it using ImageNet mean and standard deviation values, and rearranges the layout to CHW format suitable for ONNX runtime.

The run_skyseg() function executes the actual inference, feeding the preprocessed tensor into the ONNX session. The raw output score map is then rescaled back to the original image resolution and converted to a non-sky confidence map via _result_map_to_non_sky_conf(), producing values in the range [0, 1] where higher values indicate non-sky regions.

Cache Management and Versioning

Generated masks are saved as 8-bit PNG files in the cache directory for future reuse. The _prepare_sky_mask_cache() function manages cache integrity by creating a version file (.skyseg_cache_version) that guarantees compatibility across different pipeline versions. This deterministic caching ensures that repeated runs on the same dataset execute instantly after the initial pass.

Optional visualizations are produced via _save_sky_mask_visualization(), generating side-by-side panels showing the original image, the binary mask, and an overlay with red tint highlighting detected sky regions.

Applying Sky Masks to Confidence Tensors

The apply_sky_segmentation() function integrates sky masks into the 3-D reconstruction pipeline. It accepts a confidence tensor conf of shape (S, H, W) representing per-frame confidence scores from the geometry completion network, where S is the number of frames and H, W are spatial dimensions.

The function retrieves masks matching the tensor's spatial size through load_or_create_sky_masks(), then binarizes them using the soft threshold constant _SKYSEG_SOFT_THRESHOLD = 0.1. The binary mask is multiplied element-wise into the confidence scores, effectively zeroing out regions classified as sky while preserving non-sky confidence values for downstream processing.

Implementation Examples

The following example demonstrates masking a confidence tensor using the automatic pipeline:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` is the per‑frame confidence tensor produced by the GCT pipeline

# (shape: num_frames × H × W).  Provide the folder that holds the corresponding

# RGB images; the function will download the model, generate / load masks, and

# mask out the sky.

conf_masked = apply_sky_segmentation(
    conf,
    image_folder="example/university",      # folder with the original RGB frames

    skyseg_model_path="skyseg.onnx",        # optional – will be auto‑downloaded if missing

    sky_mask_dir="example/university_sky_masks",   # optional custom cache location

    sky_mask_visualization_dir="example/university_vis"  # optional visualisations

)

# `conf_masked` now has sky regions set to zero and can be fed to the next stage.

For manual mask generation without applying them to confidence tensors:

# Manually generate sky masks for a list of image files:

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

image_files = ["frame_000001.png", "frame_000002.png"]
sky_masks = load_or_create_sky_masks(
    image_paths=image_files,
    skyseg_model_path="skyseg.onnx",
    sky_mask_dir="cached_masks",
    target_shape=(480, 640)   # resize masks to match downstream resolution

)

# `sky_masks` is a NumPy array of shape (len(image_files), 480, 640) with values in [0, 1].

Summary

• Automatic model management: The pipeline downloads skyseg.onnx from Hugging Face automatically via download_skyseg_model() and manages ONNX runtime sessions efficiently.
• Intelligent caching: Mask generation results are stored as 8-bit PNGs in versioned cache directories, eliminating redundant inference on subsequent runs.
• Standardized preprocessing: Images are resized to _SKYSEG_INPUT_SIZE = (320, 320) with ImageNet normalization before ONNX inference.
• Soft-threshold masking: The apply_sky_segmentation() function uses _SKYSEG_SOFT_THRESHOLD = 0.1 to binarize masks and multiply them into confidence tensors of shape (S, H, W).
• Visualization support: Optional side-by-side visualizations showing original frames, binary masks, and sky overlays aid debugging and validation.

Frequently Asked Questions

How does LingBot-Map handle missing ONNX model files?

If skyseg.onnx is not present locally, the download_skyseg_model() function automatically streams the weights from Hugging Face and stores them in the specified path. This ensures the sky masking pipeline works immediately without manual setup.

What is the input resolution expected by the sky segmentation model?

The ONNX model expects inputs of size 320×320 pixels as defined by _SKYSEG_INPUT_SIZE. The pipeline automatically rescales input images to this resolution, performs inference, then rescales the output mask back to the original image dimensions.

How does the caching mechanism prevent redundant computation?

The load_or_create_sky_masks() function checks for existing mask files in the cache directory (typically <image_folder>_sky_masks) before running inference. It validates cache compatibility using a version file (.skyseg_cache_version) and only generates new masks for uncached or invalid entries, significantly speeding up repeated pipeline runs.

What happens to sky regions in the final confidence output?

The apply_sky_segmentation() function multiplies the confidence tensor by a binary mask where sky regions are set to zero. Using a soft threshold of 0.1, pixels with confidence below this value are considered sky and removed from the 3-D reconstruction, ensuring only non-sky geometry contributes to the final output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}