LingBot-Map Sky Segmentation Model: ONNX Implementation and 3D Visualization Benefits

LingBot-Map uses a lightweight ONNX sky segmentation model (skyseg.onnx) downloaded from HuggingFace to automatically detect and mask sky regions, eliminating noisy background points for cleaner, more focused 3D point cloud visualizations.

LingBot-Map integrates an automated sky segmentation model to enhance the quality of 3D reconstructions by filtering out atmospheric noise. According to the Robbyant/lingbot-map source code, this system leverages a pre-trained ONNX model that segments sky pixels with minimal computational overhead. By applying these masks during visualization, the library ensures that only relevant scene geometry appears in the final point cloud renderings.

The ONNX Sky Segmentation Model

LingBot-Map employs skyseg.onnx, a lightweight neural network optimized for real-time sky detection. The model is automatically retrieved from the public HuggingFace repository JianyuanWang/skyseg when the visualization pipeline initializes.

The download logic resides in lingbot_map/vis/sky_segmentation.py between lines 30–38. This utility checks for the model file locally and fetches it only when missing, ensuring seamless setup without manual configuration. Once downloaded, the ONNX runtime executes inference directly on input images to generate per-pixel confidence maps indicating sky probability.

How Sky Masking Works in LingBot-Map

The sky masking pipeline follows a three-stage process implemented in the load_or_create_sky_masks function:

1. Generate confidence maps – The ONNX model processes each input frame to produce a floating-point "sky-likeness" score for every pixel.

2. Apply soft thresholding – Values exceeding _SKYSEG_SOFT_THRESHOLD = 0.1 are classified as sky and converted into a binary mask (sky_mask_binary).

3. Mask the confidence volume – The original confidence scores (conf) undergo element-wise multiplication with the binary mask (conf = conf * sky_mask_binary), zeroing out all sky-associated points while preserving scene geometry.

This multiplication occurs within the apply_sky_segmentation function, which handles batch processing for video sequences where confidence volumes have shape (S, H, W) representing frames, height, and width.

Visualization Improvements from Sky Masking

Removing sky regions delivers measurable benefits to 3D point cloud visualization:

• Cleaner geometry – Atmospheric points often contain depth estimation errors and color inconsistencies. Masking them eliminates visual clutter, making buildings, furniture, and landscape features stand out distinctly.

• Higher-quality confidence filtering – Without sky masking, confidence thresholds might incorrectly filter valid scene points while retaining low-confidence sky pixels. The binary mask ensures threshold operations affect only relevant scene content.

• Better rendering performance – Eliminating sky points reduces the total vertex count transmitted to the browser viewer. This decreases GPU memory usage and improves frame rates during interactive exploration.

The Viser wrapper in lingbot_map/vis/viser_wrapper.py automatically exploits these benefits when the mask_sky parameter is enabled, requiring no additional post-processing from the user.

Code Implementation Examples

Enabling Sky Masking in the Viser Wrapper

The high-level visualization interface accepts a boolean flag to activate segmentation:

from lingbot_map.vis import viser_wrapper

# `pred_dict` contains the standard LingBot-Map predictions

viser_server = viser_wrapper(
    pred_dict,
    port=8080,
    init_conf_threshold=50.0,
    use_point_map=False,
    background_mode=False,
    mask_sky=True,                 # ← Enable sky segmentation

    image_folder="path/to/images"  # Source RGB images for masking

)

Direct Application to Confidence Volumes

For custom workflows, apply masks directly to confidence arrays:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` has shape (S, H, W) representing per-frame confidence scores

conf_masked = apply_sky_segmentation(
    conf,
    image_folder="path/to/images",
    skyseg_model_path="skyseg.onnx",   # Auto-downloaded if not present

)

Key Source Files and Functions

The sky segmentation system spans four primary files in the visualization module:

• lingbot_map/vis/sky_segmentation.py – Contains the apply_sky_segmentation function, model download utilities, and the _SKYSEG_SOFT_THRESHOLD constant. This file orchestrates ONNX inference and mask application.

• lingbot_map/vis/viser_wrapper.py – High-level visualizer that exposes the mask_sky parameter and internally invokes apply_sky_segmentation before transmitting data to the browser.

• lingbot_map/vis/point_cloud_viewer.py – Alternative viewer implementation demonstrating sky mask integration outside the Viser ecosystem.

• lingbot_map/vis/__init__.py – Public API surface that exports apply_sky_segmentation for direct imports.

Summary

• LingBot-Map utilizes skyseg.onnx, an ONNX model from HuggingFace that downloads automatically on first use.
• The load_or_create_sky_masks function generates binary masks using a threshold of 0.1 to identify sky pixels.
• Masking occurs via element-wise multiplication with confidence volumes, zeroing out atmospheric points while preserving scene geometry.
• Sky removal improves visualization quality by eliminating noisy background points, optimizing confidence thresholding, and reducing rendering overhead.
• Both the high-level viser_wrapper (with mask_sky=True) and the low-level apply_sky_segmentation function provide access to this functionality.

Frequently Asked Questions

What type of deep learning model does LingBot-Map use for sky segmentation?

LingBot-Map uses a lightweight ONNX format convolutional network specifically trained for semantic sky segmentation. The model file skyseg.onnx is sourced from the HuggingFace repository JianyuanWang/skyseg and runs via the ONNX runtime for efficient CPU-based inference.

How does sky masking improve confidence thresholding in 3D visualizations?

Without sky masking, atmospheric regions often exhibit low or erratic confidence scores that interfere with threshold-based filtering. By applying the binary sky mask before threshold operations, LingBot-Map ensures that confidence values below init_conf_threshold represent actual scene geometry rather than irrelevant sky pixels, preventing accidental preservation of noisy atmospheric data.

Can I use the sky segmentation feature without launching the Viser web viewer?

Yes. The apply_sky_segmentation function in lingbot_map/vis/sky_segmentation.py operates independently of the visualization layer. You can import this function directly to mask confidence volumes programmatically, then feed the results into custom rendering pipelines or export workflows without activating the Viser server.

Where is the skyseg.onnx model cached locally?

The model downloads to the local working directory or a designated cache path specified during the apply_sky_segmentation call via the skyseg_model_path parameter. If the file does not exist at the specified path, the library automatically fetches it from HuggingFace and stores it for subsequent reuse.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}