How to Enable Sky Masking in LingBot-Map and Install ONNX Runtime

Enable sky masking in LingBot-Map by installing the onnxruntime package and running the demo with the --mask_sky flag alongside an --image_folder path, which automatically downloads the sky segmentation model and filters sky points from outdoor reconstructions.

LingBot-Map (Robbyant/lingbot-map) is an open-source mapping framework that supports optional sky masking to improve visual quality in outdoor scenes. This feature uses a lightweight ONNX model to segment sky regions and exclude them from point cloud generation. To enable sky masking in LingBot-Map and install the ONNX runtime, you need to configure the segmentation pipeline and install the appropriate inference library for your hardware.

How Sky Masking Works in LingBot-Map

The sky masking pipeline is implemented across three core modules and operates lazily—only loading the ONNX model when first requested.

Entry Points and Command-Line Flags

The --mask_sky argument is defined in demo.py at line 409:

parser.add_argument("--mask_sky", action="store_true",
                    help="Apply sky segmentation to filter out sky points")

At line 592, this flag is passed to the visualization wrapper:

viser_wrapper(pred_dict, port=8080, mask_sky=args.mask_sky,
              image_folder=args.image_folder)

The offline rendering pipeline in demo_render/batch_demo.py supports the same flag, allowing batch processing with sky filtering.

Visualization Wrapper Integration

In lingbot_map/vis/viser_wrapper.py, the function signature accepts the Boolean parameter:

viser_wrapper(..., mask_sky: bool = False, image_folder: Optional[str] = None)

When mask_sky is True and an image_folder is provided, the wrapper calls apply_sky_segmentation (lines 82-85) after converting depth maps to world coordinates. The resulting confidence map modifies the per-point confidence values, dimming or omitting points where sky is detected.

ONNX Model Inference and Caching

The segmentation logic resides in lingbot_map/vis/sky_segmentation.py. Key implementation details include:

• Lazy Loading: The ONNX inference session initializes only when apply_sky_segmentation is first called
• Automatic Download: The model downloads from https://huggingface.co/JianyuanWang/skyseg/resolve/main/skyseg.onnx on first use
• Caching: Masks are cached in <image_folder>_sky_masks/ with a version stamp (_SKYSEG_CACHE_VERSION) to avoid recomputation
• Thresholding: Sky pixels are identified using _SKYSEG_SOFT_THRESHOLD = 0.1, where values below this threshold indicate sky regions

The function segment_sky_from_array processes each image through the ONNX runtime, rescales the output to the target resolution, and returns a confidence map that the visualizer uses to filter points.

Installing ONNX Runtime for Sky Segmentation

The sky segmentation module imports onnxruntime (lines 19-22 in sky_segmentation.py). Without this dependency, the code prints a warning and disables masking.

Choose the appropriate package based on your hardware:

Platform	Package	Install Command
CPU Only	onnxruntime	pip install onnxruntime
NVIDIA GPU (CUDA)	onnxruntime-gpu	pip install onnxruntime-gpu

The GPU version is recommended for large image sets or batch rendering pipelines, as it significantly accelerates inference speed.

Step-by-Step Activation Guide

1. Install the ONNX Runtime

For CPU-only systems:

pip install onnxruntime

For CUDA-enabled GPUs:

pip install onnxruntime-gpu

2. Run the Interactive Demo with Sky Masking

Execute demo.py with the required flags:

python demo.py \
    --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford \
    --mask_sky

The first run will download the skyseg.onnx model (approximately cached in example/oxford_sky_masks/), compute segmentation masks, and render the point cloud with sky points filtered.

3. Run Batch Rendering with Custom Cache Directories

For offline processing via demo_render/batch_demo.py:

python demo_render/batch_demo.py \
    --video_path /data/drive.mp4 \
    --output_folder /data/outputs/drive/ \
    --model_path /path/to/lingbot-map-long.pt \
    --config demo_render/config/outdoor_drive.yaml \
    --mask_sky \
    --sky_mask_dir /data/outputs/drive/sky_masks \
    --sky_mask_visualization_dir /data/outputs/drive/sky_viz

4. Verify the Implementation

Check that the cache directory was created at <image_folder>_sky_masks/ containing the computed masks. Subsequent runs will reuse these cached files unless the _SKYSEG_CACHE_VERSION changes in the source code.

Summary

• Sky masking in LingBot-Map filters sky points from outdoor reconstructions using an ONNX segmentation model
• Activation requires both the --mask_sky flag and an --image_folder path to provide source images
• Installation of onnxruntime (CPU) or onnxruntime-gpu (CUDA) is mandatory for the feature to function
• Caching occurs automatically in <image_folder>_sky_masks/, avoiding redundant model inference on subsequent runs
• Key files involved are demo.py, lingbot_map/vis/viser_wrapper.py, and lingbot_map/vis/sky_segmentation.py

Frequently Asked Questions

What is the difference between onnxruntime and onnxruntime-gpu?

The onnxruntime package runs inference on CPU only, while onnxruntime-gpu leverages NVIDIA CUDA for hardware acceleration. For LingBot-Map, the GPU version processes large outdoor datasets significantly faster, though both produce identical segmentation results.

Where does LingBot-Map store the downloaded sky segmentation model?

The model downloads automatically from the Hugging Face repository JianyuanWang/skyseg and caches in a directory named <image_folder>_sky_masks/ adjacent to your image folder. You can specify a custom location using the --sky_mask_dir flag.

Can I enable sky masking without providing an image folder?

No. The viser_wrapper function in lingbot_map/vis/viser_wrapper.py explicitly checks for both mask_sky=True and a valid image_folder path before calling apply_sky_segmentation. Without source images, the system cannot generate segmentation masks.

How do I force recomputation of sky masks if the model updates?

Delete the cache directory (default: <image_folder>_sky_masks/) or increment the _SKYSEG_CACHE_VERSION constant in lingbot_map/vis/sky_segmentation.py. The code checks this version stamp on startup and regenerates masks if the cached version differs from the current code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}