Common Failure Modes and Troubleshooting Steps for LingBot-Map Reconstruction

LingBot-Map reconstruction failures typically stem from COLMAP integration errors, missing camera intrinsics, GPU memory pressure, or incompatible model checkpoints, with specific diagnostic messages emitted in benchmark/datasets/general.py, demo.py, and lingbot_map/vis/point_cloud_viewer.py that enable targeted fixes.

LingBot-Map implements a streaming 3-D reconstruction pipeline that fuses vision-transformer feature extraction, monocular depth prediction, and COLMAP-based sparse structure-from-motion. When the reconstruction process fails, it does so at well-defined stages—from initial pose estimation to final point-cloud visualization—each generating specific log warnings that reference exact line numbers in the source code. Understanding these failure modes allows you to diagnose issues without tracing through the entire codebase.

COLMAP Integration Failures

The reconstruction pipeline optionally depends on COLMAP for camera pose initialization. Several failure modes originate in benchmark/datasets/general.py, where subprocess calls and file parsing occur.

Binary Not Found or Not on PATH

If the COLMAP executable is missing from the system PATH, the check at line 65–73 returns False.

• Symptom: Warning: “COLMAP binary ‘colmap’ not found” and reconstruction skips the sparse SfM stage.
• Fix: Install COLMAP (sudo apt-get install colmap or compile from source) and ensure the binary is accessible via $PATH. The code uses shutil.which(self._colmap_binary) to verify existence before execution.

Timeout or Crash During Feature Extraction

COLMAP execution is wrapped in a subprocess call capped at one hour (lines 71–85 and 82–88).

• Symptom: Warning: “COLMAP timed out (1 h limit)” or a non-zero exit code indicating crash.
• Fix: Reduce the dataset size or lower feature-extraction settings (e.g., --SiftExtraction.max_num_features). Run COLMAP manually in a terminal to inspect stderr for memory or corruption errors.

Empty or Unparseable Reconstruction Output

After the mapper runs, the code expects results in sparse/0 (lines 96–104). If the directory is missing or empty, or if images.txt and binary files are absent (lines 115–124), parsing fails.

• Symptom: “COLMAP mapper produced no output” or “No parseable COLMAP output found”.
• Fix: Verify that input images have sufficient overlap and texture. Increase feature limits or enable GPU extraction in COLMAP. Check file permissions and manually run model_converter if binary files exist but text files do not.

Camera Calibration and Pose Errors

Missing Intrinsics Fallback

When datasets lack camera intrinsics, the viewer falls back to defaults (lines 1160–1176 in benchmark/viewer.py).

• Symptom: Log line: “No intrinsics found … using default intrinsics”; potential scale or projection errors in the visualization.
• Fix: Supply proper camera parameters via the dataset CSV or adjust default_intrinsics in your configuration file to match your sensor.

Model Loading and GPU Memory Issues

Checkpoint Architecture Mismatch

Loading incompatible checkpoints triggers diagnostic prints in demo.py (lines 52–61).

• Symptom: Console output lists “Missing keys: X” or “Unexpected keys: Y”.
• Fix: Ensure the checkpoint matches the current model architecture (e.g., identical patch_size, enable_3d_rope). Re-export the checkpoint after code upgrades.

CUDA Allocation and Compilation Errors

The script sets PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to reduce pre-allocation (lines 28–35), but this conflicts with torch.compile.

• Symptom: RuntimeError during warm-up: “Expected curr_block->next == nullptr”.
• Fix: Remove the --compile flag to disable graph compilation, or unset the environment variable. The script automatically removes the CUDA allocation override when --compile is detected.

KV-Cache Cleaning Limitations

The streaming transformer in lingbot_map/models/gct_stream.py (lines 294–298) warns when components lack cache-clearing methods.

• Symptom: “Aggregator does not support KV cache cleaning”.
• Fix: This is usually benign. Only address it if you require explicit cache resets after long sequences.

Visualization and Data Quality Problems

Sky Segmentation Mask Failures

The point-cloud viewer calls apply_sky_segmentation (lines 73–80 in lingbot_map/vis/point_cloud_viewer.py) and expects pre-computed masks.

• Symptom: “Failed to generate sky mask” when computing on-the-fly for unsupported image sizes.
• Fix: Pre-compute masks using the provided segmentation script, or disable sky masking by setting mask_sky=False when initializing the viewer:

from lingbot_map.vis.point_cloud_viewer import PointCloudViewer

viewer = PointCloudViewer(
    pc_list=my_pointclouds,
    color_list=my_colors,
    conf_list=my_confidences,
    cam_dict=my_cam_dict,
    mask_sky=False,  # Skip sky segmentation to avoid mask errors

    device="cuda",
    port=8081,
)
viewer.run()

Depth Stride Mismatches

Using depth_stride > 1 (lines 98–104 in point_cloud_viewer.py) skips frames during point generation.

• Symptom: Empty point clouds for specific frames or apparent “gaps” in the reconstruction.
• Fix: Set depth_stride=1 for full density reconstruction, or accept the visual gaps as an intended performance trade-off.

Step-by-Step Diagnostic Workflow

Follow this sequence to isolate the root cause of a reconstruction failure:

1. Analyze the logs – Identify the first logger.warning or print statement that aborts the chain. The pipeline emits messages at each stage (COLMAP, pose parsing, sky segmentation).
2. Validate external dependencies – Confirm COLMAP, FFmpeg, and optional libraries like xformers are installed and on the system PATH.
3. Verify dataset integrity – Ensure image sequences are correctly ordered, have consistent resolution, and include a camera intrinsics file unless relying solely on COLMAP.
4. Run sub-steps manually – Execute COLMAP’s feature extractor and matcher separately to view full error logs. Test apply_sky_segmentation on a single image to confirm mask generation works.
5. Check fallback defaults – If intrinsics or masks are missing, verify that the default values (see viewer fallback logic) are appropriate for your camera; otherwise supply a custom intrinsics.yaml.
6. Disable compilation if needed – Remove --compile when encountering CUDA allocation errors; eager mode works reliably for most workloads.

Summary

• COLMAP failures (binary missing, timeout, empty output) originate in benchmark/datasets/general.py and require validating the installation, reducing dataset size, or checking image overlap.
• Camera intrinsics gaps trigger fallbacks in benchmark/viewer.py that you should override with accurate calibration data.
• Model loading errors in demo.py indicate architecture mismatches between checkpoints and code.
• GPU memory errors often result from incompatibility between torch.compile and the CUDA allocator configuration; disable compilation to resolve.
• Visualization artifacts like missing sky masks or sparse point clouds stem from point_cloud_viewer.py settings that you can adjust or disable.

Frequently Asked Questions

What should I do when COLMAP produces no sparse reconstruction?

Verify that your image sequence contains sufficient visual overlap and texture. In benchmark/datasets/general.py, the code checks for the existence of sparse/0 after mapping (lines 96–104). If this directory is empty, increase --SiftExtraction.max_num_features in your COLMAP settings or enable GPU-accelerated feature extraction to improve matching.

How do I resolve CUDA memory errors when using --compile?

The demo.py script sets PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to optimize memory usage (lines 28–35), but this conflicts with torch.compile’s graph optimization. If you see RuntimeError: Expected curr_block->next == nullptr, remove the --compile flag from your command line to run in eager mode, which eliminates the conflict.

Why does sky segmentation fail and how can I bypass it?

Sky segmentation fails when apply_sky_segmentation in lingbot_map/vis/point_cloud_viewer.py cannot find cached masks or cannot process the image size on-the-fly (lines 73–80). To bypass this, initialize PointCloudViewer with mask_sky=False, or pre-compute masks using the standalone segmentation script included in the repository.

What happens if my dataset lacks camera intrinsics?

The viewer in benchmark/viewer.py falls back to default focal length values when intrinsics are missing (lines 1160–1176), logging a warning. This may cause inaccurate scale or projection. Provide proper intrinsics via a dataset CSV file or modify the default_intrinsics configuration parameter to match your camera’s specifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}