Sky Masking for Outdoor Scenes in LingBot-Map: ONNX-Based Segmentation Guide

Yes, LingBot-Map performs sky masking for outdoor scenes using an ONNX-based segmentation model implemented in sky_segmentation.py that automatically detects and zeros out sky pixels in confidence maps.

LingBot-Map is an open-source 3D reconstruction framework designed to handle challenging outdoor environments. The repository includes a dedicated sky masking for outdoor scenes pipeline that filters sky regions from confidence volumes to improve mapping accuracy and visualization quality.

Core Sky Segmentation Architecture

The sky masking system centers on [lingbot_map/vis/sky_segmentation.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py), which implements a complete segmentation workflow using a pretrained model hosted on HuggingFace.

Model Acquisition and Initialization

The download_skyseg_model() function (lines 30-38) manages automatic model retrieval, verifying local availability before downloading the ONNX file from remote storage. This ensures the segmentation pipeline functions without manual configuration.

Mask Generation and Caching

The load_or_create_sky_masks() function (lines 10-24) serves as the primary entry point for mask generation. This utility either loads cached masks from disk or processes input images through segment_sky() and segment_sky_from_array() to produce per-frame binary arrays. The implementation caches results to avoid redundant computation on subsequent runs.

Confidence Map Filtering

To apply masks to reconstruction data, the apply_sky_segmentation() function (lines 73-78) multiplies the input confidence volume—typically a NumPy array of shape (S, H, W)—by a binary sky mask thresholded at 0.1. This operation sets sky pixel confidences to zero while preserving non-sky regions for downstream processing.

Integration with Visualization Pipelines

The sky masking functionality is tightly coupled with LingBot-Map's visualization stack to ensure consistent rendering.

In [lingbot_map/vis/viser_wrapper.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py) (lines 21-84), confidence maps pass through apply_sky_segmentation() prior to Viser interface rendering. Similarly, [lingbot_map/vis/point_cloud_viewer.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) (lines 33-175) invokes the same masking routine when generating confidence overlays for point cloud displays, preventing sky artifacts from contaminating 3D visualizations.

Practical Implementation Examples

Masking Confidence Volumes

To filter sky regions from reconstruction confidence data:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# conf is a (S, H, W) confidence volume from the reconstruction pipeline

conf_masked = apply_sky_segmentation(
    conf,
    image_folder="/path/to/outdoor/images",
    skyseg_model_path="skyseg.onnx",  # Auto-downloaded if missing

    sky_mask_dir="/tmp/sky_masks",
    sky_mask_visualization_dir="/tmp/sky_viz"
)

Generating Standalone Sky Masks

For custom preprocessing workflows, generate masks independently:

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

# Returns array of shape (S, H, W) with 1=non-sky, 0=sky

sky_masks = load_or_create_sky_masks(
    image_folder="/path/to/outdoor/images",
    skyseg_model_path="skyseg.onnx",
    sky_mask_dir="/tmp/sky_masks"
)

Key Source Files and Functions

Understanding the module structure enables effective customization:

• [lingbot_map/vis/sky_segmentation.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py): Contains download_skyseg_model(), load_or_create_sky_masks(), and apply_sky_segmentation() implementing the core ONNX-based segmentation logic.
• [lingbot_map/vis/viser_wrapper.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py): Integrates sky masking into the Viser visualization pipeline (lines 21-84).
• [lingbot_map/vis/point_cloud_viewer.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py): Applies sky masking for point cloud confidence overlays (lines 33-175).
• [lingbot_map/vis/utils.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/utils.py): Provides image I/O utilities supporting the segmentation workflow.

Summary

• LingBot-Map implements sky masking for outdoor scenes through an ONNX-based segmentation model with automatic HuggingFace model downloading.
• The [sky_segmentation.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) module provides apply_sky_segmentation() for filtering confidence volumes and load_or_create_sky_masks() for standalone mask generation.
• Binary masks use a 0.1 threshold to identify sky pixels (0=sky, 1=non-sky), effectively removing atmospheric regions from 3D reconstruction data.
• Visualization components in [viser_wrapper.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py) and [point_cloud_viewer.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) automatically apply these masks during rendering.
• The system caches generated masks to disk, eliminating redundant computation across multiple sessions.

Frequently Asked Questions

Does LingBot-Map require manual annotation for sky masking?

No, the system uses a pretrained ONNX model accessed via download_skyseg_model(). The model downloads automatically from HuggingFace on first use, requiring no manual labeling or training data from the user.

What input formats does the sky segmentation pipeline accept?

The pipeline accepts folders containing RGB image sequences. The load_or_create_sky_masks() function processes these through segment_sky() or segment_sky_from_array(), returning NumPy arrays of shape (S, H, W) where S represents the frame count.

Can I adjust the threshold for sky detection sensitivity?

The apply_sky_segmentation() function applies a default threshold of 0.1 for binarizing probability maps. While the high-level API uses this fixed threshold, users can access raw segmentation probabilities through lower-level functions in [sky_segmentation.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) to implement custom thresholding logic.

Is sky masking applied automatically in visualization tools?

Yes, both [viser_wrapper.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py) and [point_cloud_viewer.py](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) automatically invoke apply_sky_segmentation() when processing confidence maps. This ensures sky regions are filtered before rendering without requiring explicit user intervention in standard workflows.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}