How Sky Segmentation Filters Sky Points from a Point Cloud: The lingbot-map Implementation

Sky segmentation filters sky points by running an ONNX inference model to generate per-pixel probability masks, converting those masks into binary confidence filters, and element-wise multiplying them against the depth confidence tensor to zero out sky regions before point-cloud reconstruction.

The Robbyant/lingbot-map repository implements a dedicated sky segmentation pipeline that automatically removes sky pixels from RGB-D data to prevent artifacts in 3-D reconstructions. This system processes input frames through a pretrained neural network, caches the resulting masks, and applies them to confidence scores that drive point-cloud generation. Understanding how sky segmentation filters sky points from a point cloud requires examining the inference pipeline, the thresholding logic, and the integration points with the visualization system.

The Sky Segmentation Pipeline

The core implementation resides in lingbot_map/vis/sky_segmentation.py. The pipeline consists of four distinct stages: model inference, confidence conversion, disk caching, and application to the confidence tensor.

ONNX Model Inference

The system loads a pretrained segmentation model using onnxruntime.InferenceSession. In the run_skyseg function (line 46 of sky_segmentation.py), input images are converted from RGB to BGR format and resized to 320×320 before inference. The model outputs an 8-bit score map where higher values indicate sky regions and lower values represent non-sky areas such as buildings, vegetation, or ground planes.

The inference produces a raw score map that requires normalization to the [0, 255] range before further processing.

Confidence Map Conversion

Raw model outputs are transformed into usable confidence values by _result_map_to_non_sky_conf (line 92). This function inverts the probability scores to create a non-sky confidence map:

• Sky probability is normalized to [0, 1]
• Non-sky confidence is calculated as 1.0 - sky_score

This inversion ensures that subsequent multiplication operations suppress sky regions while preserving structural points. The resulting floating-point array represents the probability that each pixel belongs to the non-sky class.

Mask Caching Strategy

For batch processing efficiency, the system implements an aggressive caching mechanism via load_or_create_sky_masks (line 210) and _prepare_sky_mask_cache. When processing a dataset:

1. The function checks for existing cached masks on disk
2. If unavailable, it invokes segment_sky_from_array or segment_sky to generate masks
3. Results are written to the configured cache directory to avoid recomputation in subsequent runs

This caching proves essential when iterating on point-cloud parameters, as sky segmentation remains constant for static camera positions.

Applying the Sky Filter to Point Clouds

The critical filtering step occurs when the confidence tensor meets the sky mask. This happens in apply_sky_segmentation at line 73 of sky_segmentation.py.

Binary Mask Thresholding

The confidence tensor conf enters with shape (S, H, W), where S represents the number of frames. The system loads cached sky masks and resizes them to match (H, W) if dimensional mismatches exist. A soft threshold converts the continuous sky probability into a binary decision:

sky_mask_binary = (sky_mask_array > _SKYSEG_SOFT_THRESHOLD).astype(np.float32)
conf = conf * sky_mask_binary

The default _SKYSEG_SOFT_THRESHOLD of 0.1 provides a conservative filter that removes most sky pixels while preserving edge cases near horizon lines. Pixels exceeding this threshold receive a value of 0 in the confidence tensor, effectively filtering out sky points from any subsequent point-cloud reconstruction algorithms that rely on these confidence scores.

Integration with the Visualization Pipeline

The viser_wrapper module (lines 82-84) exposes this functionality to end users through the mask_sky parameter. When enabled, the wrapper:

• Loads the confidence map from depth estimation
• Invokes apply_sky_segmentation with the appropriate image folder and model path
• Passes the filtered confidence to the point-cloud view generator

This integration ensures that sky points never enter the 3-D reconstruction buffer, improving visual quality and reducing noise in downstream processing tasks such as mesh extraction or semantic labeling.

Practical Implementation Examples

Filtering Confidence Scores Directly

To apply sky segmentation to existing confidence data:

import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` – confidence scores from depth estimation, shape (S, H, W)

conf = np.random.rand(10, 480, 640).astype(np.float32)

# Apply sky segmentation using images stored in a folder

filtered_conf = apply_sky_segmentation(
    conf,
    image_folder="data/images",          # folder with the RGB frames

    skyseg_model_path="skyseg.onnx",     # automatically downloaded if missing

    sky_mask_dir="cache/sky_masks",      # optional cache dir

    sky_mask_visualization_dir="cache/visuals",  # optional visualisation

)

# `filtered_conf` now has zeros where sky was detected

Generating Masks for Batch Processing

For scenarios requiring mask inspection before point-cloud generation:

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

sky_masks = load_or_create_sky_masks(
    image_folder="data/images",
    skyseg_model_path="skyseg.onnx",
    target_shape=(480, 640),   # resize to match depth map resolution

)

# `sky_masks` shape → (num_frames, 480, 640) with values in [0, 1]

Summary

• ONNX Inference: The system uses skyseg.onnx with onnxruntime to generate per-pixel sky probabilities from 320×320 BGR inputs in run_skyseg.
• Confidence Inversion: The _result_map_to_non_sky_conf function converts sky scores to non-sky confidence via 1.0 - sky_score arithmetic.
• Persistent Caching: Masks are cached to disk via load_or_create_sky_masks to eliminate redundant inference on subsequent runs.
• Threshold-Based Filtering: apply_sky_segmentation uses a default threshold of 0.1 to create binary masks that zero out sky regions in the confidence tensor through element-wise multiplication.
• Pipeline Integration: The viser_wrapper enables sky masking through the mask_sky flag, ensuring filtered confidence drives the final point-cloud view.

Frequently Asked Questions

What neural network architecture powers the sky segmentation?

The repository utilizes a pretrained ONNX model (skyseg.onnx) loaded through onnxruntime.InferenceSession. While the specific architecture (likely a lightweight CNN or segmentation transformer) is abstracted by the ONNX format, the model accepts 320×320 BGR inputs and outputs 8-bit segmentation maps where high values indicate sky regions.

How does the soft threshold affect point cloud quality?

The _SKYSEG_SOFT_THRESHOLD default of 0.1 balances aggressive sky removal with horizon preservation. Lower thresholds (e.g., 0.05) retain more sky edge pixels but may introduce noise, while higher thresholds (e.g., 0.2) risk clipping legitimate distant structures. The threshold applies in apply_sky_segmentation when converting the soft probability mask to the binary filter.

Can sky segmentation handle arbitrary image resolutions?

Yes. The load_or_create_sky_masks function accepts a target_shape parameter (H, W) that resizes cached masks to match your depth map resolution. The underlying ONNX model always processes 320×320 inputs, but the resulting masks are bilinearly resampled to match the confidence tensor dimensions before application.

Where does the system store cached sky masks?

The _prepare_sky_mask_cache function writes masks to the directory specified by sky_mask_dir (defaulting to a subdirectory of the image folder). These cached arrays persist as NumPy binary files, allowing load_or_create_sky_masks to skip ONNX inference on subsequent runs unless the source images change.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}