# How Sky Segmentation with ONNX Improves Point Cloud Visualization Quality

> Enhance point cloud visualization with ONNX sky segmentation. Learn how this technique filters sky regions to prevent spurious points and improve 3D reconstruction quality.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: performance
- Published: 2026-07-31

---

**Sky segmentation with ONNX improves point cloud visualization by filtering out sky regions from confidence maps using a lightweight neural network, preventing spurious points from contaminating the 3D reconstruction.**

The LingBot-Map repository employs an ONNX-based sky segmentation pipeline to enhance the quality of 3D reconstructions. By identifying and masking sky pixels before point cloud generation, the system removes low-confidence artifacts that typically degrade outdoor scene visualizations.

## The ONNX Sky Segmentation Pipeline

The implementation in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) processes input images through a portable ONNX model to generate binary sky masks. This approach avoids heavyweight deep-learning frameworks at runtime while delivering precise segmentation results.

### Model Initialization with ONNX Runtime

The system creates an inference session using `onnxruntime.InferenceSession` within the `load_or_create_sky_masks` function (lines 52-55). This initialization loads the pretrained sky segmentation model into memory, preparing it for batched inference across the input image sequence.

### Image Preprocessing and Normalization

Before inference, images undergo strict preprocessing to match the model's training distribution (lines 54-60):

- **Resizing**: Input images are resized to 320×320 pixels to match the model's expected input dimensions
- **Color space conversion**: Images convert from BGR to RGB format
- **Normalization**: Pixel values are normalized using ImageNet statistics (mean **[0.485, 0.456, 0.406]** and standard deviation **[0.229, 0.224, 0.225]**)
- **Tensor reshaping**: Data is rearranged into NCHW format (batch, channels, height, width) for optimal ONNX runtime performance

### Inference and Mask Generation

The model outputs a raw sky-probability map that undergoes post-processing (lines 62-72). The system linearly scales these probabilities to an 8-bit score map (0-255 range), then resizes the result back to the original image resolution.

To create the final **non-sky confidence map**, the pipeline inverts the probability values using `1.0 - mask` (lines 92-95). This inversion ensures that sky regions receive near-zero confidence values while preserving high confidence for valid scene geometry.

## Integrating Sky Segmentation into Point Cloud Visualization

The segmentation masks integrate directly into the point cloud generation workflow by modifying the confidence tensors that drive 3D reconstruction.

### Applying Masks to Confidence Tensors

The `apply_sky_segmentation` function (lines 23-27) multiplies the binary sky mask (thresholded at **mask > 0.1**) with the per-frame confidence tensor `conf`. This operation zeroes out confidence values belonging to sky pixels, causing the subsequent point cloud generator to ignore those points entirely.

By removing sky points from the confidence maps, the resulting point cloud becomes cleaner and denser, eliminating the sparse, floating artifacts that typically appear when depth estimation models incorrectly interpret sky regions as distant geometry.

### Caching and Debugging Utilities

The repository includes robust caching mechanisms to avoid redundant computation. The `_prepare_sky_mask_cache` function (lines 35-44) stores generated masks on disk, while optional visualization outputs (lines 83-104) save side-by-side comparisons for debugging purposes.

## Practical Implementation Examples

The following examples demonstrate how to apply sky segmentation to your own datasets using the LingBot-Map utilities:

```python
import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` is the per‑frame confidence map produced by the depth‑estimation model

# Shape: (num_frames, H, W)

conf = np.load("confidence.npy")          # example source

# Apply sky segmentation using the ONNX model (automatically downloads if missing)

conf_no_sky = apply_sky_segmentation(
    conf,
    image_folder="datasets/oxford_images",   # folder with the original RGB frames

    skyseg_model_path="skyseg.onnx",          # will be downloaded from HuggingFace if absent

    sky_mask_dir="cache/sky_masks",           # optional cache directory

    sky_mask_visualization_dir="cache/vis",   # optional visual debugging output

)

# `conf_no_sky` can now be fed into the point-cloud viewer (e.g., PointCloudViewer)

```

For offline batch processing or debugging, generate sky masks directly:

```python

# Directly generate sky masks (useful for debugging or offline caching)

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

sky_masks = load_or_create_sky_masks(
    image_folder="datasets/oxford_images",
    skyseg_model_path="skyseg.onnx",
    target_shape=(480, 640),   # resize masks to match your depth map resolution

)

# `sky_masks` is an array of shape (num_frames, H, W) with values in [0, 1]

```

## Summary

- **ONNX runtime integration** in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py) enables lightweight, framework-agnostic inference without PyTorch or TensorFlow dependencies
- **ImageNet-standard preprocessing** at 320×320 resolution ensures optimal model performance across diverse outdoor datasets
- **Threshold-based masking** at 0.1 probability effectively isolates sky regions while preserving valid scene geometry
- **Confidence map multiplication** zeroes out sky pixels before point cloud generation, eliminating spurious floating points
- **Disk caching** via `_prepare_sky_mask_cache` prevents redundant computation across multiple visualization sessions

## Frequently Asked Questions

### What makes ONNX suitable for sky segmentation in point cloud pipelines?

ONNX provides a portable, runtime-efficient format that eliminates heavy deep-learning framework dependencies. According to the LingBot-Map source code, the `onnxruntime.InferenceSession` loads quickly and executes the sky segmentation model with minimal memory overhead, making it ideal for preprocessing steps that run before 3D reconstruction.

### How does the 0.1 threshold affect point cloud quality?

The **mask > 0.1** threshold in `apply_sky_segmentation` (lines 23-27) creates a binary decision boundary that conservatively classifies pixels as sky. This threshold balances between removing hazy sky gradients near horizons (which often produce noisy depth estimates) and preserving thin structures like tree branches that might otherwise be misclassified as sky.

### Can the sky segmentation handle different input resolutions?

Yes. While the model internally processes 320×320 images, the pipeline resizes outputs back to the original resolution before applying them to confidence maps. The `target_shape` parameter in `load_or_create_sky_masks` allows explicit control over final mask dimensions to match your specific depth map resolution.

### Where does LingBot-Map store cached sky masks?

The system stores cached masks in the directory specified by `sky_mask_dir` (defaulting to a local cache folder). The `_prepare_sky_mask_cache` function manages these disk operations, enabling rapid reloading of previously computed masks without re-running ONNX inference.