# How Sky Segmentation Filters Sky Points from a Point Cloud: The lingbot-map Implementation

> Learn how sky segmentation filters sky points from point clouds using ONNX inference models and confidence filters for accurate 3D reconstruction in lingbot-map.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-26

---

**Sky segmentation filters sky points by running an ONNX inference model to generate per-pixel probability masks, converting those masks into binary confidence filters, and element-wise multiplying them against the depth confidence tensor to zero out sky regions before point-cloud reconstruction.**

The Robbyant/lingbot-map repository implements a dedicated **sky segmentation** pipeline that automatically removes sky pixels from RGB-D data to prevent artifacts in 3-D reconstructions. This system processes input frames through a pretrained neural network, caches the resulting masks, and applies them to confidence scores that drive point-cloud generation. Understanding how sky segmentation filters sky points from a point cloud requires examining the inference pipeline, the thresholding logic, and the integration points with the visualization system.

## The Sky Segmentation Pipeline

The core implementation resides in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py). The pipeline consists of four distinct stages: model inference, confidence conversion, disk caching, and application to the confidence tensor.

### ONNX Model Inference

The system loads a pretrained segmentation model using `onnxruntime.InferenceSession`. In the `run_skyseg` function (line 46 of [`sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/sky_segmentation.py)), input images are converted from RGB to BGR format and resized to 320×320 before inference. The model outputs an 8-bit score map where higher values indicate sky regions and lower values represent non-sky areas such as buildings, vegetation, or ground planes.

The inference produces a raw score map that requires normalization to the [0, 255] range before further processing.

### Confidence Map Conversion

Raw model outputs are transformed into usable confidence values by `_result_map_to_non_sky_conf` (line 92). This function inverts the probability scores to create a **non-sky confidence map**:

- Sky probability is normalized to [0, 1]
- Non-sky confidence is calculated as `1.0 - sky_score`

This inversion ensures that subsequent multiplication operations suppress sky regions while preserving structural points. The resulting floating-point array represents the probability that each pixel belongs to the non-sky class.

### Mask Caching Strategy

For batch processing efficiency, the system implements an aggressive caching mechanism via `load_or_create_sky_masks` (line 210) and `_prepare_sky_mask_cache`. When processing a dataset:

1. The function checks for existing cached masks on disk
2. If unavailable, it invokes `segment_sky_from_array` or `segment_sky` to generate masks
3. Results are written to the configured cache directory to avoid recomputation in subsequent runs

This caching proves essential when iterating on point-cloud parameters, as sky segmentation remains constant for static camera positions.

## Applying the Sky Filter to Point Clouds

The critical filtering step occurs when the confidence tensor meets the sky mask. This happens in `apply_sky_segmentation` at line 73 of [`sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/sky_segmentation.py).

### Binary Mask Thresholding

The confidence tensor `conf` enters with shape `(S, H, W)`, where *S* represents the number of frames. The system loads cached sky masks and resizes them to match `(H, W)` if dimensional mismatches exist. A soft threshold converts the continuous sky probability into a binary decision:

```python
sky_mask_binary = (sky_mask_array > _SKYSEG_SOFT_THRESHOLD).astype(np.float32)
conf = conf * sky_mask_binary

```

The default `_SKYSEG_SOFT_THRESHOLD` of **0.1** provides a conservative filter that removes most sky pixels while preserving edge cases near horizon lines. Pixels exceeding this threshold receive a value of 0 in the confidence tensor, effectively filtering out sky points from any subsequent point-cloud reconstruction algorithms that rely on these confidence scores.

### Integration with the Visualization Pipeline

The `viser_wrapper` module (lines 82-84) exposes this functionality to end users through the `mask_sky` parameter. When enabled, the wrapper:

- Loads the confidence map from depth estimation
- Invokes `apply_sky_segmentation` with the appropriate image folder and model path
- Passes the filtered confidence to the point-cloud view generator

This integration ensures that sky points never enter the 3-D reconstruction buffer, improving visual quality and reducing noise in downstream processing tasks such as mesh extraction or semantic labeling.

## Practical Implementation Examples

### Filtering Confidence Scores Directly

To apply sky segmentation to existing confidence data:

```python
import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` – confidence scores from depth estimation, shape (S, H, W)

conf = np.random.rand(10, 480, 640).astype(np.float32)

# Apply sky segmentation using images stored in a folder

filtered_conf = apply_sky_segmentation(
    conf,
    image_folder="data/images",          # folder with the RGB frames

    skyseg_model_path="skyseg.onnx",     # automatically downloaded if missing

    sky_mask_dir="cache/sky_masks",      # optional cache dir

    sky_mask_visualization_dir="cache/visuals",  # optional visualisation

)

# `filtered_conf` now has zeros where sky was detected

```

### Generating Masks for Batch Processing

For scenarios requiring mask inspection before point-cloud generation:

```python
from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

sky_masks = load_or_create_sky_masks(
    image_folder="data/images",
    skyseg_model_path="skyseg.onnx",
    target_shape=(480, 640),   # resize to match depth map resolution

)

# `sky_masks` shape → (num_frames, 480, 640) with values in [0, 1]

```

## Summary

- **ONNX Inference**: The system uses `skyseg.onnx` with onnxruntime to generate per-pixel sky probabilities from 320×320 BGR inputs in `run_skyseg`.
- **Confidence Inversion**: The `_result_map_to_non_sky_conf` function converts sky scores to non-sky confidence via `1.0 - sky_score` arithmetic.
- **Persistent Caching**: Masks are cached to disk via `load_or_create_sky_masks` to eliminate redundant inference on subsequent runs.
- **Threshold-Based Filtering**: `apply_sky_segmentation` uses a default threshold of 0.1 to create binary masks that zero out sky regions in the confidence tensor through element-wise multiplication.
- **Pipeline Integration**: The `viser_wrapper` enables sky masking through the `mask_sky` flag, ensuring filtered confidence drives the final point-cloud view.

## Frequently Asked Questions

### What neural network architecture powers the sky segmentation?

The repository utilizes a pretrained ONNX model (`skyseg.onnx`) loaded through `onnxruntime.InferenceSession`. While the specific architecture (likely a lightweight CNN or segmentation transformer) is abstracted by the ONNX format, the model accepts 320×320 BGR inputs and outputs 8-bit segmentation maps where high values indicate sky regions.

### How does the soft threshold affect point cloud quality?

The `_SKYSEG_SOFT_THRESHOLD` default of 0.1 balances aggressive sky removal with horizon preservation. Lower thresholds (e.g., 0.05) retain more sky edge pixels but may introduce noise, while higher thresholds (e.g., 0.2) risk clipping legitimate distant structures. The threshold applies in `apply_sky_segmentation` when converting the soft probability mask to the binary filter.

### Can sky segmentation handle arbitrary image resolutions?

Yes. The `load_or_create_sky_masks` function accepts a `target_shape` parameter (H, W) that resizes cached masks to match your depth map resolution. The underlying ONNX model always processes 320×320 inputs, but the resulting masks are bilinearly resampled to match the confidence tensor dimensions before application.

### Where does the system store cached sky masks?

The `_prepare_sky_mask_cache` function writes masks to the directory specified by `sky_mask_dir` (defaulting to a subdirectory of the image folder). These cached arrays persist as NumPy binary files, allowing `load_or_create_sky_masks` to skip ONNX inference on subsequent runs unless the source images change.