How to Contribute to the Development of Robbyant/lingbot-map's Architecture

You can contribute to the development of Robbyant/lingbot-map's architecture by extending modular components—including attention kernels in lingbot_map/layers/attention.py, output heads in lingbot_map/heads/, and aggregation strategies in lingbot_map/aggregator/—without modifying the core training loop.

The lingbot-map repository implements a neural-SLAM system built around a Global Context Transformer (GCT) with a deliberately modular design. This architecture separates functionality into six core packages, allowing developers to swap or extend specific components while maintaining the integrity of the forward pass. Whether you want to add a novel attention mechanism or a custom perception head, the codebase provides clear extension points that minimize disruption to existing functionality.

Understanding the Modular Architecture

The architecture organizes code into discrete packages that handle specific responsibilities in the pipeline. This separation enables targeted contributions to lingbot-map architecture development without requiring changes to unrelated systems.

Core Package Structure

• lingbot_map.models: Houses the model hierarchy including GCTBase and GCTStream classes that define forward passes, token handling, and prediction heads. Key files include lingbot_map/models/gct_base.py and lingbot_map/models/gct_stream_window_v2.py.
• lingbot_map.aggregator: Implements token-level aggregation strategies via AggregatorBase and AggregatorStream, managing how transformer outputs merge across timesteps.
• lingbot_map.layers: Contains low-level neural building blocks such as Attention, FlashInferAttention, Block, and VisionTransformer in lingbot_map/layers/attention.py, block.py, and vision_transformer.py.
• lingbot_map.heads: Provides output heads for specific tasks including CameraHead for pose estimation and DPTHead for depth prediction.
• lingbot_map.utils: Supplies geometry helpers, rotation utilities, and pose encoding functions used throughout the pipeline.
• lingbot_map.vis: Offers visualization tools including the PointCloudViewer for 3D reconstruction debugging.

Data Flow Through the Pipeline

Understanding the data flow helps identify where architectural contributions fit:

1. Input preprocessing: Raw RGB-D frames load and rectify via preprocess/oxford.py.
2. Patch embedding: The PatchEmbed class in lingbot_map/layers/vision_transformer.py splits images into tokens.
3. Transformer blocks: Blocks apply Attention or FlashInferAttention kernels, followed by LayerScale and DropPath for stability.
4. Aggregation: The AggregatorBase subclass (typically AggregatorStream) merges token histories using a fixed-size buffer.
5. Output heads: Task-specific heads like CameraHead or DPTHead decode tokens into depth maps, pose vectors, or masks.
6. Visualization: Utilities in lingbot_map/vis render outputs for debugging.

Extension Points for Contributors

The architecture offers four primary hooks for contributing to the development of Robbyant/lingbot-map's architecture without destabilizing existing code.

Implementing Custom Attention Kernels

To add a new attention mechanism, subclass Attention or FlashInferAttention in lingbot_map/layers/attention.py. For example, implement a Sparse-Local Attention module to reduce quadratic complexity for long sequences. Ensure your implementation maintains the interface expected by Block in lingbot_map/layers/block.py.

Creating New Output Heads

Add novel perception capabilities by inheriting from nn.Module in lingbot_map/heads/. Register your head in lingbot_map/models/gct_base.py by adding it to the self.heads dictionary. This enables the model to output additional predictions alongside depth and pose estimates.

Designing Alternative Aggregators

Replace the default streaming strategy by creating a new subclass of AggregatorBase in lingbot_map/aggregator/. Consider implementing a Hierarchical Memory aggregator that maintains long-term tokens in a separate buffer from recent windows.

Enhancing Geometry Utilities

Extend lingbot_map/utils/geometry.py or rotation.py with differentiable helpers such as an SE(3)-Lie algebra module to simplify pose updates in custom heads.

Step-by-Step: Adding a Semantic Segmentation Head

Below is a complete example demonstrating how to contribute to the lingbot-map architecture by adding a SemanticHead that outputs per-pixel class logits.

First, create lingbot_map/heads/semantic_head.py:

import torch
import torch.nn as nn
from .head_act import HeadAct

class SemanticHead(nn.Module):
    """Predict per-pixel class logits from transformer tokens."""
    def __init__(self, dim: int, num_classes: int, patch_size: int = 4):
        super().__init__()
        self.patch_size = patch_size
        self.classifier = nn.Sequential(
            nn.Conv2d(dim, dim // 2, kernel_size=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(dim // 2, num_classes, kernel_size=1),
        )
        self.act = HeadAct()

    def forward(self, tokens, H, W):
        # tokens: (B, N, C) where N = H*W/patch_size^2

        B, N, C = tokens.shape
        H_out, W_out = H // self.patch_size, W // self.patch_size
        x = tokens.permute(0, 2, 1).contiguous().view(B, C, H_out, W_out)
        logits = self.classifier(x)
        return self.act(logits)

Next, register the head in lingbot_map/models/gct_base.py (around line 70):

# Inside GCTBase.__init__

self.heads = {
    "camera": CameraHead(...),
    "depth": DPTHead(...),
    "semantic": SemanticHead(dim=self.embed_dim, num_classes=20),
}

The model now outputs outputs["semantic"] alongside existing keys.

Testing Your Implementation

Verify the new head with a minimal test script:

from lingbot_map.models.gct_base import GCTBase
import torch

model = GCTBase(...)
tokens = torch.randn(2, 1024, model.embed_dim)
H, W = 640, 480
out = model.heads["semantic"](tokens, H, W)
assert out.shape == (2, 20, H // 4, W // 4)

Add a unit test in tests/test_semantic_head.py and ensure it runs with the existing pytest configuration in pyproject.toml.

Contribution Workflow

Follow these steps to submit your architectural improvements:

1. Fork the repository and create a feature branch from main.
2. Establish a baseline by running python demo.py to verify your environment.
3. Implement your component in the appropriate package directory.
4. Write unit tests under tests/ covering forward passes and edge cases.
5. Update documentation including docstrings and README entries for new parameters.
6. Submit a Pull Request referencing the specific architectural area (e.g., "Adds Sparse-Local Attention to layers") and include test coverage.

The project requires PyTorch as specified in pyproject.toml. If your contribution uses optional performance kernels like FlashInfer, provide a pure-Python fallback for environments without the compiled extension.

Summary

• Modular design separates concerns into models, layers, aggregator, heads, utils, and vis packages.
• Extension points include lingbot_map/layers/attention.py for kernels, lingbot_map/heads/ for task outputs, and lingbot_map/aggregator/ for token merging strategies.
• Registration of new heads occurs in lingbot_map/models/gct_base.py via the self.heads dictionary.
• Testing requires unit tests in tests/ and verification against demo.py.
• Compatibility demands PyTorch compatibility and optional FlashInfer fallbacks for performance kernels.

Frequently Asked Questions

How do I add a new attention mechanism without breaking existing models?

Subclass Attention or FlashInferAttention in lingbot_map/layers/attention.py while maintaining the input-output tensor shapes expected by Block in lingbot_map/layers/block.py. The transformer blocks automatically use any attention module following the base interface, allowing seamless integration.

Can I replace the default patch embedding with a learned tokenizer?

Yes. Modify or replace PatchEmbed in lingbot_map/layers/patch_embed.py or adjust the embedding logic within lingbot_map/layers/vision_transformer.py. Ensure your implementation outputs tokens with dimensions compatible with the embed_dim parameter used in GCTBase.

What is the difference between GCTBase and GCTStream?

GCTBase in lingbot_map/models/gct_base.py provides the foundational transformer architecture and head management, while GCTStream in lingbot_map/models/gct_stream_window_v2.py implements a specific streaming variant using windowed aggregation. Contributors typically extend GCTBase for new heads or modify AggregatorStream for different temporal handling.

Where should I implement geometry utilities for custom pose estimation?

Add differentiable geometry helpers to lingbot_map/utils/geometry.py or lingbot_map/utils/rotation.py. These utilities import across heads and models, making them available for custom implementations like CameraHead or novel SLAM components.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}