# How to Contribute to the Development of Robbyant/lingbot-map's Architecture

> Contribute to Robbyant lingbot map architecture by extending modular components like attention kernels and output heads. Learn how to enhance the project without altering the core training loop.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: architecture
- Published: 2026-07-07

---

**You can contribute to the development of Robbyant/lingbot-map's architecture by extending modular components—including attention kernels in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), output heads in `lingbot_map/heads/`, and aggregation strategies in `lingbot_map/aggregator/`—without modifying the core training loop.**

The **lingbot-map** repository implements a neural-SLAM system built around a **Global Context Transformer (GCT)** with a deliberately modular design. This architecture separates functionality into six core packages, allowing developers to swap or extend specific components while maintaining the integrity of the forward pass. Whether you want to add a novel attention mechanism or a custom perception head, the codebase provides clear extension points that minimize disruption to existing functionality.

## Understanding the Modular Architecture

The architecture organizes code into discrete packages that handle specific responsibilities in the pipeline. This separation enables targeted contributions to **lingbot-map architecture development** without requiring changes to unrelated systems.

### Core Package Structure

- **lingbot_map.models**: Houses the model hierarchy including `GCTBase` and `GCTStream` classes that define forward passes, token handling, and prediction heads. Key files include [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) and [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py).
- **lingbot_map.aggregator**: Implements token-level aggregation strategies via `AggregatorBase` and `AggregatorStream`, managing how transformer outputs merge across timesteps.
- **lingbot_map.layers**: Contains low-level neural building blocks such as `Attention`, `FlashInferAttention`, `Block`, and `VisionTransformer` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py), [`block.py`](https://github.com/Robbyant/lingbot-map/blob/main/block.py), and [`vision_transformer.py`](https://github.com/Robbyant/lingbot-map/blob/main/vision_transformer.py).
- **lingbot_map.heads**: Provides output heads for specific tasks including `CameraHead` for pose estimation and `DPTHead` for depth prediction.
- **lingbot_map.utils**: Supplies geometry helpers, rotation utilities, and pose encoding functions used throughout the pipeline.
- **lingbot_map.vis**: Offers visualization tools including the `PointCloudViewer` for 3D reconstruction debugging.

### Data Flow Through the Pipeline

Understanding the data flow helps identify where architectural contributions fit:

1. **Input preprocessing**: Raw RGB-D frames load and rectify via [`preprocess/oxford.py`](https://github.com/Robbyant/lingbot-map/blob/main/preprocess/oxford.py).
2. **Patch embedding**: The `PatchEmbed` class in [`lingbot_map/layers/vision_transformer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/vision_transformer.py) splits images into tokens.
3. **Transformer blocks**: Blocks apply `Attention` or `FlashInferAttention` kernels, followed by `LayerScale` and `DropPath` for stability.
4. **Aggregation**: The `AggregatorBase` subclass (typically `AggregatorStream`) merges token histories using a fixed-size buffer.
5. **Output heads**: Task-specific heads like `CameraHead` or `DPTHead` decode tokens into depth maps, pose vectors, or masks.
6. **Visualization**: Utilities in `lingbot_map/vis` render outputs for debugging.

## Extension Points for Contributors

The architecture offers four primary hooks for **contributing to the development of Robbyant/lingbot-map's architecture** without destabilizing existing code.

### Implementing Custom Attention Kernels

To add a new attention mechanism, subclass `Attention` or `FlashInferAttention` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py). For example, implement a **Sparse-Local Attention** module to reduce quadratic complexity for long sequences. Ensure your implementation maintains the interface expected by `Block` in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py).

### Creating New Output Heads

Add novel perception capabilities by inheriting from `nn.Module` in `lingbot_map/heads/`. Register your head in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) by adding it to the `self.heads` dictionary. This enables the model to output additional predictions alongside depth and pose estimates.

### Designing Alternative Aggregators

Replace the default streaming strategy by creating a new subclass of `AggregatorBase` in `lingbot_map/aggregator/`. Consider implementing a **Hierarchical Memory** aggregator that maintains long-term tokens in a separate buffer from recent windows.

### Enhancing Geometry Utilities

Extend [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) or [`rotation.py`](https://github.com/Robbyant/lingbot-map/blob/main/rotation.py) with differentiable helpers such as an **SE(3)-Lie algebra** module to simplify pose updates in custom heads.

## Step-by-Step: Adding a Semantic Segmentation Head

Below is a complete example demonstrating how to **contribute to the lingbot-map architecture** by adding a `SemanticHead` that outputs per-pixel class logits.

First, create [`lingbot_map/heads/semantic_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/heads/semantic_head.py):

```python
import torch
import torch.nn as nn
from .head_act import HeadAct

class SemanticHead(nn.Module):
    """Predict per-pixel class logits from transformer tokens."""
    def __init__(self, dim: int, num_classes: int, patch_size: int = 4):
        super().__init__()
        self.patch_size = patch_size
        self.classifier = nn.Sequential(
            nn.Conv2d(dim, dim // 2, kernel_size=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(dim // 2, num_classes, kernel_size=1),
        )
        self.act = HeadAct()

    def forward(self, tokens, H, W):
        # tokens: (B, N, C) where N = H*W/patch_size^2

        B, N, C = tokens.shape
        H_out, W_out = H // self.patch_size, W // self.patch_size
        x = tokens.permute(0, 2, 1).contiguous().view(B, C, H_out, W_out)
        logits = self.classifier(x)
        return self.act(logits)

```

Next, register the head in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) (around line 70):

```python

# Inside GCTBase.__init__

self.heads = {
    "camera": CameraHead(...),
    "depth": DPTHead(...),
    "semantic": SemanticHead(dim=self.embed_dim, num_classes=20),
}

```

The model now outputs `outputs["semantic"]` alongside existing keys.

### Testing Your Implementation

Verify the new head with a minimal test script:

```python
from lingbot_map.models.gct_base import GCTBase
import torch

model = GCTBase(...)
tokens = torch.randn(2, 1024, model.embed_dim)
H, W = 640, 480
out = model.heads["semantic"](tokens, H, W)
assert out.shape == (2, 20, H // 4, W // 4)

```

Add a unit test in [`tests/test_semantic_head.py`](https://github.com/Robbyant/lingbot-map/blob/main/tests/test_semantic_head.py) and ensure it runs with the existing `pytest` configuration in [`pyproject.toml`](https://github.com/Robbyant/lingbot-map/blob/main/pyproject.toml).

## Contribution Workflow

Follow these steps to submit your **architectural improvements**:

1. **Fork the repository** and create a feature branch from `main`.
2. **Establish a baseline** by running `python demo.py` to verify your environment.
3. **Implement your component** in the appropriate package directory.
4. **Write unit tests** under `tests/` covering forward passes and edge cases.
5. **Update documentation** including docstrings and README entries for new parameters.
6. **Submit a Pull Request** referencing the specific architectural area (e.g., "Adds Sparse-Local Attention to layers") and include test coverage.

The project requires **PyTorch** as specified in [`pyproject.toml`](https://github.com/Robbyant/lingbot-map/blob/main/pyproject.toml). If your contribution uses optional performance kernels like **FlashInfer**, provide a pure-Python fallback for environments without the compiled extension.

## Summary

- **Modular design** separates concerns into `models`, `layers`, `aggregator`, `heads`, `utils`, and `vis` packages.
- **Extension points** include [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) for kernels, `lingbot_map/heads/` for task outputs, and `lingbot_map/aggregator/` for token merging strategies.
- **Registration** of new heads occurs in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) via the `self.heads` dictionary.
- **Testing** requires unit tests in `tests/` and verification against [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py).
- **Compatibility** demands PyTorch compatibility and optional FlashInfer fallbacks for performance kernels.

## Frequently Asked Questions

### How do I add a new attention mechanism without breaking existing models?

Subclass `Attention` or `FlashInferAttention` in [`lingbot_map/layers/attention.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/attention.py) while maintaining the input-output tensor shapes expected by `Block` in [`lingbot_map/layers/block.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/block.py). The transformer blocks automatically use any attention module following the base interface, allowing seamless integration.

### Can I replace the default patch embedding with a learned tokenizer?

Yes. Modify or replace `PatchEmbed` in [`lingbot_map/layers/patch_embed.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/patch_embed.py) or adjust the embedding logic within [`lingbot_map/layers/vision_transformer.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/layers/vision_transformer.py). Ensure your implementation outputs tokens with dimensions compatible with the `embed_dim` parameter used in `GCTBase`.

### What is the difference between `GCTBase` and `GCTStream`?

`GCTBase` in [`lingbot_map/models/gct_base.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_base.py) provides the foundational transformer architecture and head management, while `GCTStream` in [`lingbot_map/models/gct_stream_window_v2.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream_window_v2.py) implements a specific streaming variant using windowed aggregation. Contributors typically extend `GCTBase` for new heads or modify `AggregatorStream` for different temporal handling.

### Where should I implement geometry utilities for custom pose estimation?

Add differentiable geometry helpers to [`lingbot_map/utils/geometry.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/geometry.py) or [`lingbot_map/utils/rotation.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/utils/rotation.py). These utilities import across heads and models, making them available for custom implementations like `CameraHead` or novel SLAM components.