# What CUDA Extensions Are Used in the LINGBOT Map Render Pipeline?

> Discover how the LINGBOT map render pipeline uses the points_visibility CUDA extension to accelerate visibility filtering and frustum culling on depth maps. Learn about its C++ extension compilation.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: deep-dive
- Published: 2026-07-04

---

**The LINGBOT map render pipeline relies on a single optional CUDA extension called `points_visibility` that compiles just-in-time via `torch.utils.cpp_extension` to accelerate visibility filtering and frustum culling on depth maps.**

The Robbyant/lingbot-map repository implements a hybrid rendering architecture that conditionally leverages GPU acceleration through a custom PyTorch extension. When a CUDA-capable device is present, the pipeline compiles and loads the `points_visibility` extension to perform high-performance per-pixel visibility checks and point cloud frustum culling, falling back to pure NumPy implementations when GPU resources are unavailable.

## The Custom `points_visibility` Extension

The LINGBOT map render pipeline utilizes exactly one CUDA extension: **`points_visibility`**. This extension is not pre-compiled but rather built on-the-fly using PyTorch's JIT compilation system through `torch.utils.cpp_extension.load()`.

### Source Files and Architecture

The extension consists of three source files located in `preprocess/points_visibility/`:

- **[`visibility.cpp`](https://github.com/Robbyant/lingbot-map/blob/main/visibility.cpp)** – The C++ wrapper that binds CUDA kernels to Python via PyTorch's C++ API
- **`visibility_kernel.cu`** – The core CUDA kernel implementing per-pixel visibility checks for depth maps
- **`frustum_cull.cu`** – The CUDA kernel responsible for frustum culling of point clouds prior to visibility testing

These files work together to provide fast visibility-filtering functionality that is critical for accurate point-cloud visualization.

## How the Extension is Loaded

The extension loading logic is centralized in [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py) within the `ensure_visibility_extension()` function. This utility manages the just-in-time compilation and optional fallback behavior.

```python
def ensure_visibility_extension() -> bool:
    """Load the optional CUDA visibility extension on first use."""
    global _vis_ext, _HAS_VIS_EXT, _VIS_EXT_ATTEMPTED

    if _VIS_EXT_ATTEMPTED:
        return _HAS_VIS_EXT

    _VIS_EXT_ATTEMPTED = True
    try:
        from torch.utils.cpp_extension import load as _ext_load
        _EXT_DIR = Path(__file__).parent / "points_visibility"
        _vis_ext = _ext_load(
            name="points_visibility",
            sources=[
                str(_EXT_DIR / "visibility.cpp"),
                str(_EXT_DIR / "visibility_kernel.cu"),
                str(_EXT_DIR / "frustum_cull.cu"),
            ],
            build_directory=str(_EXT_DIR),
            verbose=True,
        )
        _HAS_VIS_EXT = torch.cuda.is_available()
    except Exception as _e:
        print(f"[visibility] CUDA extension unavailable ({_e}); NumPy fallback active.")
        _vis_ext = None
        _HAS_VIS_EXT = False

    return _HAS_VIS_EXT

```

The function returns a boolean indicating whether GPU acceleration is active, allowing downstream code to select the appropriate implementation path.

## Using the Visibility Filter

Once loaded, the extension exposes CUDA-accelerated functions that can be invoked directly from Python. The primary entry point is the `visibility_filter` function, which processes depth maps on the GPU.

### Checking Extension Availability

```python
from preprocess.oxford import ensure_visibility_extension

if ensure_visibility_extension():
    print("CUDA visibility extension loaded – using GPU kernels.")
else:
    print("CUDA extension unavailable – falling back to NumPy.")

```

### Applying the Filter to Depth Maps

The following example demonstrates how to call the CUDA kernel for visibility filtering:

```python
import torch
from preprocess.oxford import _vis_ext, _HAS_VIS_EXT

def filter_depth(depth_np, fx, fy, cx, cy, threshold=0.01, radius=1):
    """
    Apply the CUDA visibility filter (if available) to a depth map.
    """
    if _HAS_VIS_EXT and _vis_ext is not None:
        # Convert inputs to CUDA tensors

        depth_t = torch.from_numpy(depth_np).float().cuda()
        fx_t = torch.tensor(fx, dtype=torch.float32, device='cuda')
        fy_t = torch.tensor(fy, dtype=torch.float32, device='cuda')
        cx_t = torch.tensor(cx, dtype=torch.float32, device='cuda')
        cy_t = torch.tensor(cy, dtype=torch.float32, device='cuda')

        # Call the CUDA kernel (exposed as `visibility_filter`)

        filtered = _vis_ext.visibility_filter(
            depth_t, fx_t, fy_t, cx_t, cy_t,
            torch.tensor(threshold, device='cuda'),
            torch.tensor(radius, device='cuda')
        )
        return filtered.cpu().numpy()
    else:
        # NumPy fallback (slower)

        return depth_np  # <-- replace with the actual NumPy implementation

```

### Integration in the Rendering Pipeline

In the full LINGBOT pipeline, the visibility filter integrates with the point cloud viewer:

```python
from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

viewer = PointCloudViewer()
depth = load_depth_for_frame(frame_idx)          # (H, W) numpy array

fx, fy, cx, cy = get_camera_intrinsics()
depth_filtered = filter_depth(depth, fx, fy, cx, cy)

viewer.update_depth(depth_filtered)
viewer.render()

```

## CPU Fallback Behavior

The extension is **optional by design**. If `torch.cuda.is_available()` returns `False` or if the compilation fails (missing nvcc, incompatible PyTorch version, etc.), the pipeline automatically degrades to a pure NumPy implementation. This ensures functional correctness across all hardware configurations while maximizing performance on systems with NVIDIA GPUs.

## Summary

- The LINGBOT map render pipeline uses exactly one custom CUDA extension: **`points_visibility`**
- The extension is built just-in-time using `torch.utils.cpp_extension` from three source files in `preprocess/points_visibility/`
- **Core components**: [`visibility.cpp`](https://github.com/Robbyant/lingbot-map/blob/main/visibility.cpp) (Python binding), `visibility_kernel.cu` (visibility checks), and `frustum_cull.cu` (frustum culling)
- Loading is managed by `ensure_visibility_extension()` in [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py), which returns a boolean indicating GPU availability
- The extension accelerates depth-map filtering through the `visibility_filter` CUDA kernel, with automatic fallback to NumPy when CUDA is unavailable

## Frequently Asked Questions

### What happens if I don't have a CUDA-capable GPU?

The pipeline automatically detects missing CUDA support and falls back to a pure NumPy implementation. When `ensure_visibility_extension()` fails to compile or load the extension, it sets `_HAS_VIS_EXT` to `False` and prints a warning message, allowing the rendering pipeline to continue using CPU-based algorithms.

### Can I pre-compile the `points_visibility` extension instead of using JIT compilation?

While the current implementation uses PyTorch's just-in-time compilation via `torch.utils.cpp_extension.load()`, you could theoretically pre-compile the extension using `setuptools` with `CppExtension`. However, the repository is designed to compile on first use to avoid complex build dependencies during installation.

### Which PyTorch version is required for the CUDA extension?

The extension requires PyTorch with CUDA support installed. The `torch.utils.cpp_extension` module handles the compilation against your specific PyTorch and CUDA versions, ensuring binary compatibility. The code checks `torch.cuda.is_available()` after loading to confirm the extension can execute on the GPU.

### Where does the actual visibility algorithm run?

The core visibility algorithm runs in the **`visibility_kernel.cu`** file, which contains CUDA kernels that perform per-pixel visibility checks on depth maps. This is preceded by frustum culling in `frustum_cull.cu` to eliminate out-of-view points before the expensive visibility tests occur.