# LingBot-Map Sky Segmentation Model: ONNX Implementation and 3D Visualization Benefits

> Explore LingBot-Map's ONNX sky segmentation model for cleaner 3D point cloud visualizations. Automatically mask skies and remove background noise for enhanced clarity and focus.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-29

---

**LingBot-Map uses a lightweight ONNX sky segmentation model (`skyseg.onnx`) downloaded from HuggingFace to automatically detect and mask sky regions, eliminating noisy background points for cleaner, more focused 3D point cloud visualizations.**

LingBot-Map integrates an automated **sky segmentation model** to enhance the quality of 3D reconstructions by filtering out atmospheric noise. According to the Robbyant/lingbot-map source code, this system leverages a pre-trained ONNX model that segments sky pixels with minimal computational overhead. By applying these masks during visualization, the library ensures that only relevant scene geometry appears in the final point cloud renderings.

## The ONNX Sky Segmentation Model

LingBot-Map employs **`skyseg.onnx`**, a lightweight neural network optimized for real-time sky detection. The model is automatically retrieved from the public HuggingFace repository *JianyuanWang/skyseg* when the visualization pipeline initializes.

The download logic resides in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) between lines 30–38. This utility checks for the model file locally and fetches it only when missing, ensuring seamless setup without manual configuration. Once downloaded, the ONNX runtime executes inference directly on input images to generate per-pixel confidence maps indicating sky probability.

## How Sky Masking Works in LingBot-Map

The sky masking pipeline follows a three-stage process implemented in the `load_or_create_sky_masks` function:

1. **Generate confidence maps** – The ONNX model processes each input frame to produce a floating-point "sky-likeness" score for every pixel.

2. **Apply soft thresholding** – Values exceeding `_SKYSEG_SOFT_THRESHOLD = 0.1` are classified as sky and converted into a binary mask (`sky_mask_binary`).

3. **Mask the confidence volume** – The original confidence scores (`conf`) undergo element-wise multiplication with the binary mask (`conf = conf * sky_mask_binary`), zeroing out all sky-associated points while preserving scene geometry.

This multiplication occurs within the `apply_sky_segmentation` function, which handles batch processing for video sequences where confidence volumes have shape `(S, H, W)` representing frames, height, and width.

## Visualization Improvements from Sky Masking

Removing sky regions delivers measurable benefits to 3D point cloud visualization:

- **Cleaner geometry** – Atmospheric points often contain depth estimation errors and color inconsistencies. Masking them eliminates visual clutter, making buildings, furniture, and landscape features stand out distinctly.

- **Higher-quality confidence filtering** – Without sky masking, confidence thresholds might incorrectly filter valid scene points while retaining low-confidence sky pixels. The binary mask ensures threshold operations affect only relevant scene content.

- **Better rendering performance** – Eliminating sky points reduces the total vertex count transmitted to the browser viewer. This decreases GPU memory usage and improves frame rates during interactive exploration.

The Viser wrapper in [`lingbot_map/vis/viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py) automatically exploits these benefits when the `mask_sky` parameter is enabled, requiring no additional post-processing from the user.

## Code Implementation Examples

### Enabling Sky Masking in the Viser Wrapper

The high-level visualization interface accepts a boolean flag to activate segmentation:

```python
from lingbot_map.vis import viser_wrapper

# `pred_dict` contains the standard LingBot-Map predictions

viser_server = viser_wrapper(
    pred_dict,
    port=8080,
    init_conf_threshold=50.0,
    use_point_map=False,
    background_mode=False,
    mask_sky=True,                 # ← Enable sky segmentation

    image_folder="path/to/images"  # Source RGB images for masking

)

```

### Direct Application to Confidence Volumes

For custom workflows, apply masks directly to confidence arrays:

```python
import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` has shape (S, H, W) representing per-frame confidence scores

conf_masked = apply_sky_segmentation(
    conf,
    image_folder="path/to/images",
    skyseg_model_path="skyseg.onnx",   # Auto-downloaded if not present

)

```

## Key Source Files and Functions

The sky segmentation system spans four primary files in the visualization module:

- **[`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py)** – Contains the `apply_sky_segmentation` function, model download utilities, and the `_SKYSEG_SOFT_THRESHOLD` constant. This file orchestrates ONNX inference and mask application.

- **[`lingbot_map/vis/viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py)** – High-level visualizer that exposes the `mask_sky` parameter and internally invokes `apply_sky_segmentation` before transmitting data to the browser.

- **[`lingbot_map/vis/point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py)** – Alternative viewer implementation demonstrating sky mask integration outside the Viser ecosystem.

- **[`lingbot_map/vis/__init__.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/__init__.py)** – Public API surface that exports `apply_sky_segmentation` for direct imports.

## Summary

- LingBot-Map utilizes **`skyseg.onnx`**, an ONNX model from HuggingFace that downloads automatically on first use.
- The `load_or_create_sky_masks` function generates binary masks using a threshold of **0.1** to identify sky pixels.
- Masking occurs via element-wise multiplication with confidence volumes, zeroing out atmospheric points while preserving scene geometry.
- Sky removal improves visualization quality by eliminating noisy background points, optimizing confidence thresholding, and reducing rendering overhead.
- Both the high-level `viser_wrapper` (with `mask_sky=True`) and the low-level `apply_sky_segmentation` function provide access to this functionality.

## Frequently Asked Questions

### What type of deep learning model does LingBot-Map use for sky segmentation?

LingBot-Map uses a lightweight **ONNX format** convolutional network specifically trained for semantic sky segmentation. The model file `skyseg.onnx` is sourced from the HuggingFace repository *JianyuanWang/skyseg* and runs via the ONNX runtime for efficient CPU-based inference.

### How does sky masking improve confidence thresholding in 3D visualizations?

Without sky masking, atmospheric regions often exhibit low or erratic confidence scores that interfere with threshold-based filtering. By applying the binary sky mask before threshold operations, LingBot-Map ensures that confidence values below `init_conf_threshold` represent actual scene geometry rather than irrelevant sky pixels, preventing accidental preservation of noisy atmospheric data.

### Can I use the sky segmentation feature without launching the Viser web viewer?

Yes. The `apply_sky_segmentation` function in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) operates independently of the visualization layer. You can import this function directly to mask confidence volumes programmatically, then feed the results into custom rendering pipelines or export workflows without activating the Viser server.

### Where is the skyseg.onnx model cached locally?

The model downloads to the local working directory or a designated cache path specified during the `apply_sky_segmentation` call via the `skyseg_model_path` parameter. If the file does not exist at the specified path, the library automatically fetches it from HuggingFace and stores it for subsequent reuse.