What CUDA Extensions Are Used in the LINGBOT Map Render Pipeline?

The LINGBOT map render pipeline relies on a single optional CUDA extension called points_visibility that compiles just-in-time via torch.utils.cpp_extension to accelerate visibility filtering and frustum culling on depth maps.

The Robbyant/lingbot-map repository implements a hybrid rendering architecture that conditionally leverages GPU acceleration through a custom PyTorch extension. When a CUDA-capable device is present, the pipeline compiles and loads the points_visibility extension to perform high-performance per-pixel visibility checks and point cloud frustum culling, falling back to pure NumPy implementations when GPU resources are unavailable.

The Custom points_visibility Extension

The LINGBOT map render pipeline utilizes exactly one CUDA extension: points_visibility. This extension is not pre-compiled but rather built on-the-fly using PyTorch's JIT compilation system through torch.utils.cpp_extension.load().

Source Files and Architecture

The extension consists of three source files located in preprocess/points_visibility/:

• visibility.cpp – The C++ wrapper that binds CUDA kernels to Python via PyTorch's C++ API
• visibility_kernel.cu – The core CUDA kernel implementing per-pixel visibility checks for depth maps
• frustum_cull.cu – The CUDA kernel responsible for frustum culling of point clouds prior to visibility testing

These files work together to provide fast visibility-filtering functionality that is critical for accurate point-cloud visualization.

How the Extension is Loaded

The extension loading logic is centralized in preprocess/oxford.py within the ensure_visibility_extension() function. This utility manages the just-in-time compilation and optional fallback behavior.

def ensure_visibility_extension() -> bool:
    """Load the optional CUDA visibility extension on first use."""
    global _vis_ext, _HAS_VIS_EXT, _VIS_EXT_ATTEMPTED

    if _VIS_EXT_ATTEMPTED:
        return _HAS_VIS_EXT

    _VIS_EXT_ATTEMPTED = True
    try:
        from torch.utils.cpp_extension import load as _ext_load
        _EXT_DIR = Path(__file__).parent / "points_visibility"
        _vis_ext = _ext_load(
            name="points_visibility",
            sources=[
                str(_EXT_DIR / "visibility.cpp"),
                str(_EXT_DIR / "visibility_kernel.cu"),
                str(_EXT_DIR / "frustum_cull.cu"),
            ],
            build_directory=str(_EXT_DIR),
            verbose=True,
        )
        _HAS_VIS_EXT = torch.cuda.is_available()
    except Exception as _e:
        print(f"[visibility] CUDA extension unavailable ({_e}); NumPy fallback active.")
        _vis_ext = None
        _HAS_VIS_EXT = False

    return _HAS_VIS_EXT

The function returns a boolean indicating whether GPU acceleration is active, allowing downstream code to select the appropriate implementation path.

Using the Visibility Filter

Once loaded, the extension exposes CUDA-accelerated functions that can be invoked directly from Python. The primary entry point is the visibility_filter function, which processes depth maps on the GPU.

Checking Extension Availability

from preprocess.oxford import ensure_visibility_extension

if ensure_visibility_extension():
    print("CUDA visibility extension loaded – using GPU kernels.")
else:
    print("CUDA extension unavailable – falling back to NumPy.")

Applying the Filter to Depth Maps

The following example demonstrates how to call the CUDA kernel for visibility filtering:

import torch
from preprocess.oxford import _vis_ext, _HAS_VIS_EXT

def filter_depth(depth_np, fx, fy, cx, cy, threshold=0.01, radius=1):
    """
    Apply the CUDA visibility filter (if available) to a depth map.
    """
    if _HAS_VIS_EXT and _vis_ext is not None:
        # Convert inputs to CUDA tensors

        depth_t = torch.from_numpy(depth_np).float().cuda()
        fx_t = torch.tensor(fx, dtype=torch.float32, device='cuda')
        fy_t = torch.tensor(fy, dtype=torch.float32, device='cuda')
        cx_t = torch.tensor(cx, dtype=torch.float32, device='cuda')
        cy_t = torch.tensor(cy, dtype=torch.float32, device='cuda')

        # Call the CUDA kernel (exposed as `visibility_filter`)

        filtered = _vis_ext.visibility_filter(
            depth_t, fx_t, fy_t, cx_t, cy_t,
            torch.tensor(threshold, device='cuda'),
            torch.tensor(radius, device='cuda')
        )
        return filtered.cpu().numpy()
    else:
        # NumPy fallback (slower)

        return depth_np  # <-- replace with the actual NumPy implementation

Integration in the Rendering Pipeline

In the full LINGBOT pipeline, the visibility filter integrates with the point cloud viewer:

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

viewer = PointCloudViewer()
depth = load_depth_for_frame(frame_idx)          # (H, W) numpy array

fx, fy, cx, cy = get_camera_intrinsics()
depth_filtered = filter_depth(depth, fx, fy, cx, cy)

viewer.update_depth(depth_filtered)
viewer.render()

CPU Fallback Behavior

The extension is optional by design. If torch.cuda.is_available() returns False or if the compilation fails (missing nvcc, incompatible PyTorch version, etc.), the pipeline automatically degrades to a pure NumPy implementation. This ensures functional correctness across all hardware configurations while maximizing performance on systems with NVIDIA GPUs.

Summary

• The LINGBOT map render pipeline uses exactly one custom CUDA extension: points_visibility
• The extension is built just-in-time using torch.utils.cpp_extension from three source files in preprocess/points_visibility/
• Core components: visibility.cpp (Python binding), visibility_kernel.cu (visibility checks), and frustum_cull.cu (frustum culling)
• Loading is managed by ensure_visibility_extension() in preprocess/oxford.py, which returns a boolean indicating GPU availability
• The extension accelerates depth-map filtering through the visibility_filter CUDA kernel, with automatic fallback to NumPy when CUDA is unavailable

Frequently Asked Questions

What happens if I don't have a CUDA-capable GPU?

The pipeline automatically detects missing CUDA support and falls back to a pure NumPy implementation. When ensure_visibility_extension() fails to compile or load the extension, it sets _HAS_VIS_EXT to False and prints a warning message, allowing the rendering pipeline to continue using CPU-based algorithms.

Can I pre-compile the points_visibility extension instead of using JIT compilation?

While the current implementation uses PyTorch's just-in-time compilation via torch.utils.cpp_extension.load(), you could theoretically pre-compile the extension using setuptools with CppExtension. However, the repository is designed to compile on first use to avoid complex build dependencies during installation.

Which PyTorch version is required for the CUDA extension?

The extension requires PyTorch with CUDA support installed. The torch.utils.cpp_extension module handles the compilation against your specific PyTorch and CUDA versions, ensuring binary compatibility. The code checks torch.cuda.is_available() after loading to confirm the extension can execute on the GPU.

Where does the actual visibility algorithm run?

The core visibility algorithm runs in the visibility_kernel.cu file, which contains CUDA kernels that perform per-pixel visibility checks on depth maps. This is preceded by frustum culling in frustum_cull.cu to eliminate out-of-view points before the expensive visibility tests occur.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}