# How to Integrate the ONNX Sky Segmentation Model into the Ling-Bot-Map Reconstruction Pipeline

> Integrate the ONNX sky segmentation model into your Ling-Bot-Map pipeline. Generate binary masks to remove sky pixels for cleaner 3-D point clouds and improved reconstruction accuracy.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-04

---

**The ONNX sky segmentation model integrates into the Ling-Bot-Map reconstruction pipeline by generating binary masks that zero-out confidence scores for sky pixels, preventing depth-less sky regions from contaminating the dense 3-D point cloud.**

Ling-Bot-Map is an open-source framework for dense 3-D reconstruction from video sequences. During reconstruction, each frame produces a **confidence map** (`conf`) indicating the reliability of depth estimates. Since sky regions contain no valid depth information, the pipeline optionally runs an **ONNX sky-segmentation model** to detect and suppress these pixels before final point cloud generation.

## Architecture Overview

The sky-segmentation logic is woven into three primary visualization modules. Each entry point checks the `mask_sky` flag and delegates processing to the core segmentation utilities.

### Prediction Preprocessing

In [`lingbot_map/vis/viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/viser_wrapper.py), the `viser_wrapper()` function prepares depth-based point clouds for visualization. When `mask_sky=True`, it invokes `apply_sky_segmentation(conf, image_folder)` to filter the confidence map before rendering (lines 82-84).

### Interactive Viewer

The `PointCloudViewer` class in [`lingbot_map/vis/point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/point_cloud_viewer.py) processes prediction dictionaries via `_process_pred_dict()`. If sky masking is enabled, the confidence map is filtered through `apply_sky_segmentation()` before the interactive Viser UI launches (lines 73-79).

### Sky-Mask Generation Core

The central orchestration happens in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py). The `apply_sky_segmentation()` function (lines 373-426) coordinates model loading, mask caching, and confidence-map multiplication. It loads cached masks or generates new ones via `run_skyseg()`, then multiplies the binary sky mask with the original confidence scores to zero-out sky regions.

## Data Flow Pipeline

The integration follows a strict preprocessing sequence:

1. **Input Acquisition**: RGB frames are read from `image_folder`, and depth inference produces a raw point cloud with confidence map `conf` of shape `(S, H, W)`.

2. **Mask Retrieval**: `apply_sky_segmentation()` calls `load_or_create_sky_masks()` (lines 210-294), which checks for cached masks under `<image_folder>_sky_masks`. If missing, it executes `run_skyseg()` on each image.

3. **ONNX Inference**: The `run_skyseg()` function (lines 46-72) loads the model via `onnxruntime.InferenceSession`, preprocesses the BGR image, and forwards it through the model to generate a sky-score map (higher values indicate sky). The output is normalized to an 8-bit score map.

4. **Confidence Suppression**: The sky-score map is inverted to a non-sky confidence map and thresholded (`sky_mask_binary > _SKYSEG_SOFT_THRESHOLD`). The original confidence map is multiplied element-wise: `conf = conf * sky_mask_binary`, setting sky-region scores to zero.

5. **Point Cloud Filtering**: Subsequent filtering steps discard points with zero confidence, ensuring the final reconstruction contains no sky artifacts.

## Implementation Details

### Model Caching Strategy

To avoid redundant inference, `load_or_create_sky_masks()` implements a disk-based cache. Masks are stored as binary images in a parallel directory structure. If the cache exists, the function reads masks directly from disk; otherwise, it triggers ONNX inference and persists the results for future runs.

### Automatic Model Download

The `download_skyseg_model()` helper (lines 430-556) ensures the ONNX file is available at runtime. If `skyseg.onnx` is missing, the function automatically downloads it from HuggingFace, eliminating manual setup requirements.

### ONNX Runtime Configuration

The model session is initialized once per pipeline execution. The `run_skyseg()` function handles BGR-to-RGB conversion, resizing to `_SKYSEG_INPUT_SIZE`, and normalization before feeding tensors to the `onnxruntime` inference session.

## Code Examples

### Enable Sky Masking in the Interactive Viewer

```python
from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

# pred_dict contains images, depth maps, and raw confidence

viewer = PointCloudViewer(
    pred_dict=pred_dict,
    mask_sky=True,                # Enable ONNX sky segmentation

    image_folder="data/images",   # Source RGB frames for mask generation

    port=8080,
)
viewer.animate()  # Launches Viser UI with sky-filtered point cloud

```

### Stand-Alone Confidence Map Filtering

```python
import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# conf shape: (S, H, W) from depth inference

masked_conf = apply_sky_segmentation(
    conf,
    image_folder="data/images",
    skyseg_model_path="skyseg.onnx",  # Optional: auto-downloaded if omitted

)

```

### Direct ONNX Inference (Advanced)

```python
import onnxruntime
import cv2
from lingbot_map.vis.sky_segmentation import run_skyseg, _SKYSEG_INPUT_SIZE

# Initialize session once

session = onnxruntime.InferenceSession("skyseg.onnx")

# Load BGR image

img = cv2.imread("data/images/frame_000001.jpg")

# Generate sky-score map (uint8, higher values = sky)

sky_score = run_skyseg(session, _SKYSEG_INPUT_SIZE, img)

```

## Summary

- **Integration Points**: The ONNX model is invoked in [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py) and [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py) when `mask_sky=True`, delegating to [`sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/sky_segmentation.py) for core logic.
- **Caching Mechanism**: Masks are cached under `<image_folder>_sky_masks` to avoid re-running inference on identical frames.
- **Confidence Modulation**: The pipeline multiplies the binary sky mask with the confidence map, effectively zeroing out sky pixels before point cloud generation.
- **Automatic Setup**: The model downloads automatically from HuggingFace if not present locally, using `download_skyseg_model()`.

## Frequently Asked Questions

### How does the pipeline handle missing ONNX model files?

If `skyseg.onnx` is not found in the working directory, the `download_skyseg_model()` function automatically downloads the pre-trained weights from HuggingFace. This ensures the reconstruction pipeline works without manual model procurement.

### Can I disable sky segmentation for specific frames?

Yes. Sky segmentation is optional and controlled by the `mask_sky` boolean parameter. Setting `mask_sky=False` skips the `apply_sky_segmentation()` call entirely, preserving the original confidence map values for all frames.

### What is the performance impact of running the ONNX model?

The model runs once per unique frame and caches results to disk. Subsequent reconstructions using the same `image_folder` read masks from cache, reducing the overhead to simple disk I/O. The initial inference uses `onnxruntime` for optimized CPU or GPU execution.

### Which file contains the threshold for sky detection?

The soft threshold for binarizing the sky-score map is defined as `_SKYSEG_SOFT_THRESHOLD` in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py). The `apply_sky_segmentation()` function applies this threshold to convert the continuous sky-score map into a binary mask before confidence modulation.