How Sky Segmentation with ONNX Improves Point Cloud Visualization Quality

Sky segmentation with ONNX improves point cloud visualization by filtering out sky regions from confidence maps using a lightweight neural network, preventing spurious points from contaminating the 3D reconstruction.

The LingBot-Map repository employs an ONNX-based sky segmentation pipeline to enhance the quality of 3D reconstructions. By identifying and masking sky pixels before point cloud generation, the system removes low-confidence artifacts that typically degrade outdoor scene visualizations.

The ONNX Sky Segmentation Pipeline

The implementation in lingbot_map/vis/sky_segmentation.py processes input images through a portable ONNX model to generate binary sky masks. This approach avoids heavyweight deep-learning frameworks at runtime while delivering precise segmentation results.

Model Initialization with ONNX Runtime

The system creates an inference session using onnxruntime.InferenceSession within the load_or_create_sky_masks function (lines 52-55). This initialization loads the pretrained sky segmentation model into memory, preparing it for batched inference across the input image sequence.

Image Preprocessing and Normalization

Before inference, images undergo strict preprocessing to match the model's training distribution (lines 54-60):

• Resizing: Input images are resized to 320×320 pixels to match the model's expected input dimensions
• Color space conversion: Images convert from BGR to RGB format
• Normalization: Pixel values are normalized using ImageNet statistics (mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225])
• Tensor reshaping: Data is rearranged into NCHW format (batch, channels, height, width) for optimal ONNX runtime performance

Inference and Mask Generation

The model outputs a raw sky-probability map that undergoes post-processing (lines 62-72). The system linearly scales these probabilities to an 8-bit score map (0-255 range), then resizes the result back to the original image resolution.

To create the final non-sky confidence map, the pipeline inverts the probability values using 1.0 - mask (lines 92-95). This inversion ensures that sky regions receive near-zero confidence values while preserving high confidence for valid scene geometry.

Integrating Sky Segmentation into Point Cloud Visualization

The segmentation masks integrate directly into the point cloud generation workflow by modifying the confidence tensors that drive 3D reconstruction.

Applying Masks to Confidence Tensors

The apply_sky_segmentation function (lines 23-27) multiplies the binary sky mask (thresholded at mask > 0.1) with the per-frame confidence tensor conf. This operation zeroes out confidence values belonging to sky pixels, causing the subsequent point cloud generator to ignore those points entirely.

By removing sky points from the confidence maps, the resulting point cloud becomes cleaner and denser, eliminating the sparse, floating artifacts that typically appear when depth estimation models incorrectly interpret sky regions as distant geometry.

Caching and Debugging Utilities

The repository includes robust caching mechanisms to avoid redundant computation. The _prepare_sky_mask_cache function (lines 35-44) stores generated masks on disk, while optional visualization outputs (lines 83-104) save side-by-side comparisons for debugging purposes.

Practical Implementation Examples

The following examples demonstrate how to apply sky segmentation to your own datasets using the LingBot-Map utilities:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` is the per‑frame confidence map produced by the depth‑estimation model

# Shape: (num_frames, H, W)

conf = np.load("confidence.npy")          # example source

# Apply sky segmentation using the ONNX model (automatically downloads if missing)

conf_no_sky = apply_sky_segmentation(
    conf,
    image_folder="datasets/oxford_images",   # folder with the original RGB frames

    skyseg_model_path="skyseg.onnx",          # will be downloaded from HuggingFace if absent

    sky_mask_dir="cache/sky_masks",           # optional cache directory

    sky_mask_visualization_dir="cache/vis",   # optional visual debugging output

)

# `conf_no_sky` can now be fed into the point-cloud viewer (e.g., PointCloudViewer)

For offline batch processing or debugging, generate sky masks directly:

# Directly generate sky masks (useful for debugging or offline caching)

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

sky_masks = load_or_create_sky_masks(
    image_folder="datasets/oxford_images",
    skyseg_model_path="skyseg.onnx",
    target_shape=(480, 640),   # resize masks to match your depth map resolution

)

# `sky_masks` is an array of shape (num_frames, H, W) with values in [0, 1]

Summary

• ONNX runtime integration in lingbot_map/vis/sky_segmentation.py enables lightweight, framework-agnostic inference without PyTorch or TensorFlow dependencies
• ImageNet-standard preprocessing at 320×320 resolution ensures optimal model performance across diverse outdoor datasets
• Threshold-based masking at 0.1 probability effectively isolates sky regions while preserving valid scene geometry
• Confidence map multiplication zeroes out sky pixels before point cloud generation, eliminating spurious floating points
• Disk caching via _prepare_sky_mask_cache prevents redundant computation across multiple visualization sessions

Frequently Asked Questions

What makes ONNX suitable for sky segmentation in point cloud pipelines?

ONNX provides a portable, runtime-efficient format that eliminates heavy deep-learning framework dependencies. According to the LingBot-Map source code, the onnxruntime.InferenceSession loads quickly and executes the sky segmentation model with minimal memory overhead, making it ideal for preprocessing steps that run before 3D reconstruction.

How does the 0.1 threshold affect point cloud quality?

The mask > 0.1 threshold in apply_sky_segmentation (lines 23-27) creates a binary decision boundary that conservatively classifies pixels as sky. This threshold balances between removing hazy sky gradients near horizons (which often produce noisy depth estimates) and preserving thin structures like tree branches that might otherwise be misclassified as sky.

Can the sky segmentation handle different input resolutions?

Yes. While the model internally processes 320×320 images, the pipeline resizes outputs back to the original resolution before applying them to confidence maps. The target_shape parameter in load_or_create_sky_masks allows explicit control over final mask dimensions to match your specific depth map resolution.

Where does LingBot-Map store cached sky masks?

The system stores cached masks in the directory specified by sky_mask_dir (defaulting to a local cache folder). The _prepare_sky_mask_cache function manages these disk operations, enabling rapid reloading of previously computed masks without re-running ONNX inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}