# How the Sky Masking Pipeline Works in LingBot-Map Using ONNX Segmentation

> Explore LingBot-Map's sky masking pipeline. Learn how ONNX segmentation creates sky masks, caches them, and zeros out sky pixels in 3D reconstructions. Understand the full process.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-27

---

**LingBot-Map removes sky regions from 3-D reconstructions by running an ONNX-based segmentation model to generate per-frame sky masks, caching them for reuse, and applying them to confidence tensors to zero-out sky pixels.**

The LingBot-Map repository implements an efficient sky masking pipeline that integrates ONNX segmentation models directly into its 3-D reconstruction workflow. By automatically downloading model weights from Hugging Face and caching intermediate masks, the system ensures deterministic, reproducible results while filtering out sky contamination from geometry completion outputs.

## Pipeline Architecture Overview

The sky masking pipeline operates in four distinct stages implemented in [`lingbot_map/vis/sky_segmentation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/vis/sky_segmentation.py). First, the system acquires the ONNX model file `skyseg.onnx` via automatic download if not present locally. Second, it initializes a reusable `onnxruntime.InferenceSession` for efficient batch processing. Third, it generates per-frame sky masks with intelligent caching to avoid redundant computation. Finally, it applies these masks to the geometry completion network's confidence scores using a soft threshold to exclude sky pixels from the reconstruction.

## Model Acquisition and ONNX Session Initialization

If the `skyseg.onnx` model file is missing from the local filesystem, the `download_skyseg_model()` function fetches it from Hugging Face using streaming requests and stores it locally. This ensures the pipeline works out-of-the-box without manual weight downloads.

Once the model file is available, the `load_or_create_sky_masks()` function creates an `onnxruntime.InferenceSession` object that persists for the entire pipeline execution. This session reuse eliminates the overhead of repeatedly loading the model into memory when processing video sequences or large image batches.

## Per-Frame Mask Generation and Caching

The `load_or_create_sky_masks()` function handles mask generation through a sophisticated caching mechanism that supports both image arrays and file-based inputs. For each frame, the system first checks the cache directory (conventionally named `<image_folder>_sky_masks`) for an existing mask file. If a valid cache entry exists with matching dimensions, the pipeline loads the precomputed mask directly; otherwise, it proceeds with ONNX inference.

### Input Preprocessing and ONNX Inference

When generating new masks, the pipeline processes images through the `segment_sky_from_array()` or `segment_sky()` helper functions. The input frame is rescaled to the model's expected input size defined by `_SKYSEG_INPUT_SIZE = (320, 320)`. The preprocessing pipeline converts the image to RGB, normalizes it using ImageNet mean and standard deviation values, and rearranges the layout to CHW format suitable for ONNX runtime.

The `run_skyseg()` function executes the actual inference, feeding the preprocessed tensor into the ONNX session. The raw output score map is then rescaled back to the original image resolution and converted to a non-sky confidence map via `_result_map_to_non_sky_conf()`, producing values in the range [0, 1] where higher values indicate non-sky regions.

### Cache Management and Versioning

Generated masks are saved as 8-bit PNG files in the cache directory for future reuse. The `_prepare_sky_mask_cache()` function manages cache integrity by creating a version file (`.skyseg_cache_version`) that guarantees compatibility across different pipeline versions. This deterministic caching ensures that repeated runs on the same dataset execute instantly after the initial pass.

Optional visualizations are produced via `_save_sky_mask_visualization()`, generating side-by-side panels showing the original image, the binary mask, and an overlay with red tint highlighting detected sky regions.

## Applying Sky Masks to Confidence Tensors

The `apply_sky_segmentation()` function integrates sky masks into the 3-D reconstruction pipeline. It accepts a confidence tensor `conf` of shape **(S, H, W)** representing per-frame confidence scores from the geometry completion network, where S is the number of frames and H, W are spatial dimensions.

The function retrieves masks matching the tensor's spatial size through `load_or_create_sky_masks()`, then binarizes them using the soft threshold constant `_SKYSEG_SOFT_THRESHOLD = 0.1`. The binary mask is multiplied element-wise into the confidence scores, effectively zeroing out regions classified as sky while preserving non-sky confidence values for downstream processing.

## Implementation Examples

The following example demonstrates masking a confidence tensor using the automatic pipeline:

```python
import numpy as np
from lingbot_map.vis.sky_segmentation import apply_sky_segmentation

# `conf` is the per‑frame confidence tensor produced by the GCT pipeline

# (shape: num_frames × H × W).  Provide the folder that holds the corresponding

# RGB images; the function will download the model, generate / load masks, and

# mask out the sky.

conf_masked = apply_sky_segmentation(
    conf,
    image_folder="example/university",      # folder with the original RGB frames

    skyseg_model_path="skyseg.onnx",        # optional – will be auto‑downloaded if missing

    sky_mask_dir="example/university_sky_masks",   # optional custom cache location

    sky_mask_visualization_dir="example/university_vis"  # optional visualisations

)

# `conf_masked` now has sky regions set to zero and can be fed to the next stage.

```

For manual mask generation without applying them to confidence tensors:

```python

# Manually generate sky masks for a list of image files:

from lingbot_map.vis.sky_segmentation import load_or_create_sky_masks

image_files = ["frame_000001.png", "frame_000002.png"]
sky_masks = load_or_create_sky_masks(
    image_paths=image_files,
    skyseg_model_path="skyseg.onnx",
    sky_mask_dir="cached_masks",
    target_shape=(480, 640)   # resize masks to match downstream resolution

)

# `sky_masks` is a NumPy array of shape (len(image_files), 480, 640) with values in [0, 1].

```

## Summary

- **Automatic model management**: The pipeline downloads `skyseg.onnx` from Hugging Face automatically via `download_skyseg_model()` and manages ONNX runtime sessions efficiently.
- **Intelligent caching**: Mask generation results are stored as 8-bit PNGs in versioned cache directories, eliminating redundant inference on subsequent runs.
- **Standardized preprocessing**: Images are resized to `_SKYSEG_INPUT_SIZE = (320, 320)` with ImageNet normalization before ONNX inference.
- **Soft-threshold masking**: The `apply_sky_segmentation()` function uses `_SKYSEG_SOFT_THRESHOLD = 0.1` to binarize masks and multiply them into confidence tensors of shape **(S, H, W)**.
- **Visualization support**: Optional side-by-side visualizations showing original frames, binary masks, and sky overlays aid debugging and validation.

## Frequently Asked Questions

### How does LingBot-Map handle missing ONNX model files?

If `skyseg.onnx` is not present locally, the `download_skyseg_model()` function automatically streams the weights from Hugging Face and stores them in the specified path. This ensures the sky masking pipeline works immediately without manual setup.

### What is the input resolution expected by the sky segmentation model?

The ONNX model expects inputs of size **320×320 pixels** as defined by `_SKYSEG_INPUT_SIZE`. The pipeline automatically rescales input images to this resolution, performs inference, then rescales the output mask back to the original image dimensions.

### How does the caching mechanism prevent redundant computation?

The `load_or_create_sky_masks()` function checks for existing mask files in the cache directory (typically `<image_folder>_sky_masks`) before running inference. It validates cache compatibility using a version file (`.skyseg_cache_version`) and only generates new masks for uncached or invalid entries, significantly speeding up repeated pipeline runs.

### What happens to sky regions in the final confidence output?

The `apply_sky_segmentation()` function multiplies the confidence tensor by a binary mask where sky regions are set to zero. Using a soft threshold of 0.1, pixels with confidence below this value are considered sky and removed from the 3-D reconstruction, ensuring only non-sky geometry contributes to the final output.