How to Implement Custom Camera Modes in LingBot-Map's Render Pipeline

You implement custom camera modes in LingBot-Map by creating a factory function in demo_render/rgbd_render/camera.py that returns a CameraPath, registering it in the build_camera_path orchestrator, and exposing parameters through the CameraSegment dataclass in config.py.

LingBot-Map orchestrates camera trajectories through a config-driven pipeline that decouples motion planning from OpenGL rendering. The system processes CameraSegment definitions—loaded via PipelineConfig.from_yaml and load_config—to generate interpolated CameraPath objects, feeding per-frame Camera poses to the renderer. By extending the pipeline's factory methods, you can introduce entirely new camera behaviors without modifying the core rendering logic in demo_render/rgbd_render/renderer.py.

Understanding the Camera Pipeline Architecture

The architecture separates concerns across four key components:

• Camera and CameraPath (in demo_render/rgbd_render/camera.py): Represent a single pose and an interpolated sequence of poses, respectively.
• CameraSegment and CameraConfig (in demo_render/rgbd_render/config.py): Dataclasses that declare what mode to use, when (frame range), and how (mode-specific parameters).
• build_camera_path (in demo_render/rgbd_render/camera.py): Orchestrates segments, instantiates concrete paths per segment, and stitches them with optional transitions.
• Render loop (in demo_render/rgbd_render/renderer.py): Consumes the final CameraPath and feeds each Camera to the OpenGL renderer.

How Camera Modes Generate Motion

The pipeline processes segments through four distinct phases:

1. Configuration: You define a CameraSegment with a mode string (e.g., follow, birdeye, static, pivot) via YAML or CLI.
2. Instantiation: build_camera_path (lines 332-363) switches on seg.mode and calls a factory like make_follow_path or make_birdeye_path.
3. Keyframe Generation: Each factory returns a CameraPath capable of producing a Camera for any frame index via CameraPath.get_camera.
4. Segment Blending: The builder handles smooth transitions between adjacent segments using smoothstep interpolation.

Implementing a Custom Mode: Step-by-Step

To add a custom mode such as "orbit" (camera circles around the scene center), follow this implementation path.

Step 1: Extend the Configuration Dataclass

If your mode requires extra parameters, add fields to CameraSegment in demo_render/rgbd_render/config.py:

@dataclass
class CameraSegment:
    mode: str
    frames: List[int]
    # ... existing fields ...

    orbit_radius: float = 2.0
    orbit_speed: float = 1.0

Step 2: Implement the Factory Function

Create a factory function in demo_render/rgbd_render/camera.py that computes per-frame poses and returns a CameraPath:

def make_orbit_path(c2w_poses: np.ndarray,
                    total_frames: int,
                    radius: float = 2.0,
                    speed: float = 1.0,
                    fov_deg: float = 60.0) -> CameraPath:
    """Orbit camera around scene centre on a horizontal circle."""
    centre = c2w_poses[:, :3, 3].mean(axis=0).astype(np.float32)
    up = np.array([0, 1, 0], dtype=np.float32)

    keyframes = []
    for fi in range(total_frames):
        angle = 2 * np.pi * (fi / total_frames) * speed
        eye = centre + radius * np.array([np.cos(angle), 0, np.sin(angle)],
                                         dtype=np.float32)
        keyframes.append({
            'frame': fi,
            'eye': eye.tolist(),
            'center': centre.tolist(),
            'up': up.tolist(),
            'fov': fov_deg,
        })
    return CameraPath(keyframes, total_frames, interpolation='linear')

Step 3: Register in the Path Builder

Extend the mode cascade in build_camera_path (around line 690) to wire the factory to the configuration:

elif seg.mode == 'orbit':
    path = make_orbit_path(
        scene.c2w_poses,
        total_frames,
        radius=getattr(seg, 'orbit_radius', 2.0),
        speed=getattr(seg, 'orbit_speed', 1.0),
        fov_deg=fov,
    )
    seg_paths.append((s, e, path))

Step 4: Configure via YAML or CLI

Reference the new mode in your pipeline configuration:

camera:
  fov: 60
  transition: 30
  segments:
    - mode: orbit
      frames: [0, -1]
      orbit_radius: 2.0
      orbit_speed: 1.0

Alternatively, pass flat arguments via CLI (mapped automatically by _build_segments_from_flat_args):

python demo_render/demo.py \
    --config my_config.yaml \
    --orbit_radius 3.5 \
    --orbit_speed 0.8

Handling Transitions Between Segments

Transitions between adjacent segments occur in build_camera_path (lines 422-476). The system uses _smoothstep and _lerp_camera to blend camera poses across the transition frame range. If your mode requires a different blending strategy, modify the blend computation block (around line 546) or supply a custom interpolation value in the CameraPath constructor.

Summary

Implementing custom camera modes in LingBot-Map requires understanding the separation between configuration and execution:

• Configuration: Extend CameraSegment in config.py to declare mode-specific parameters.
• Factory: Implement a builder function in camera.py that generates CameraPath keyframes.
• Registration: Hook the factory into the build_camera_path orchestrator's mode selection cascade.
• Usage: Reference the mode in YAML configs or CLI arguments to trigger the new behavior.

Frequently Asked Questions

Where do I define the parameters for my custom camera mode?

Define parameters in the CameraSegment dataclass in demo_render/rgbd_render/config.py. Add typed fields with default values to ensure backward compatibility with existing configurations that may not specify your new parameters.

How does the render loop access my custom camera poses?

The render loop in demo_render/rgbd_render/renderer.py consumes the final CameraPath object produced by the pipeline. It calls CameraPath.get_camera(frame_idx) for each frame, which returns the interpolated Camera pose containing the eye, center, up, and fov values your factory function computed.

Can I combine multiple custom modes in a single render?

Yes. build_camera_path processes a list of CameraSegment objects sequentially. Define multiple segments with different modes in your YAML configuration, and the orchestrator automatically stitches them together using the smooth transition logic defined in the path builder.

What interpolation methods are available for camera paths?

The CameraPath constructor accepts an interpolation parameter, typically set to 'linear' in factory functions. The transition blending between segments uses _smoothstep and _lerp_camera as implemented in build_camera_path (lines 422-476), ensuring smooth handoffs between different camera motions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}