# How to Integrate lingbot-map with Other Services: A Complete Guide

> Learn to integrate lingbot-map with external services by instantiating GCTStream and StreamAggregator. Publish pose and mesh data via ROS, MQTT, or HTTP.

- Repository: [Robbyant/lingbot-map](https://github.com/Robbyant/lingbot-map)
- Tags: how-to-guide
- Published: 2026-07-19

---

**To integrate lingbot-map with external services, instantiate the `GCTStream` model and `StreamAggregator` to process RGB-D frames, then publish the resulting pose and mesh data through your preferred transport protocol (ROS, MQTT, or HTTP endpoints).**

The `lingbot-map` repository by Robbyant provides a modular Python library for real-time 3D reconstruction. Its clean separation between model inference, data aggregation, and visualization layers allows seamless integration with robotics middleware, cloud backends, and web services.

## Architecture Overview

The codebase organizes functionality into three distinct layers that communicate through well-defined Python classes.

### Model Layer

Located in `lingbot_map/models/`, specifically [`gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream.py) and [`gct_stream_window.py`](https://github.com/Robbyant/lingbot-map/blob/main/gct_stream_window.py), this layer encapsulates the neural network backend. The `GCTStream` class loads pretrained checkpoints and processes incoming RGB-D frames to generate live point clouds and meshes.

### Aggregation Layer

Implemented in [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py), the `StreamAggregator` class manages a sliding window of frames. It buffers incoming data, runs inference through the model, and fuses partial reconstructions into a global scene representation.

### Visualization Layer

Found under `lingbot_map/vis/`, this layer includes [`viser_wrapper.py`](https://github.com/Robbyant/lingbot-map/blob/main/viser_wrapper.py) for Open3D-based rendering, [`point_cloud_viewer.py`](https://github.com/Robbyant/lingbot-map/blob/main/point_cloud_viewer.py) for local debugging, and [`glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/glb_export.py) for generating web-compatible mesh files. These wrappers enable both real-time visualization and service-oriented export workflows.

## Integration Workflow

Connecting lingbot-map to external systems requires four specific steps. Each step leverages the isolated architecture to minimize coupling with your existing infrastructure.

### 1. Create a Data Source

Construct an iterator that yields `(timestamp, rgb_image, depth_image, intrinsics)` tuples. This abstraction allows the pipeline to consume data from video files, live camera drivers, or network sockets without modifying core library code.

### 2. Instantiate the Streaming Pipeline

Import the core classes and initialize them with your checkpoint path:

```python
from lingbot_map.models.gct_stream import GCTStream
from lingbot_map.aggregator.stream import StreamAggregator

model = GCTStream(checkpoint_path="/path/to/lingbot-map.pt")
aggregator = StreamAggregator(model=model, window_size=8)

```

The `window_size` parameter controls the temporal buffer for smoothing reconstructions.

### 3. Process Incoming Frames

Feed your data source into the aggregator within a processing loop:

```python
for ts, rgb, depth, intr in my_source():
    aggregator.process_frame(ts, rgb, depth, intr)
    result = aggregator.current_result()
    # result contains point_cloud, mesh, and pose (4×4 numpy array)

```

### 4. Export or Publish Results

Convert the `LingbotMapResult` object into your target format. For robotics systems, serialize `result.pose` as a transformation matrix. For web services, export the mesh to GLB format using `lingbot_map.vis.glb_export.export_glb()`.

## Integration Examples

The following patterns demonstrate concrete implementations for common service architectures.

### Processing Video Files

This example reads side-by-side RGB-D recordings using OpenCV:

```python
import cv2
from lingbot_map.models.gct_stream import GCTStream
from lingbot_map.aggregator.stream import StreamAggregator

# Initialize pipeline

model = GCTStream(checkpoint_path="lingbot-map.pt")
agg = StreamAggregator(model=model, window_size=8)

# Process video stream

cap = cv2.VideoCapture("my_dataset/video.mp4")
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    rgb = frame[:, :frame.shape[1] // 2, :]
    depth = frame[:, frame.shape[1] // 2:, :]
    
    agg.process_frame(
        ts=cap.get(cv2.CAP_PROP_POS_MSEC),
        rgb=rgb,
        depth=depth,
        intrinsics=my_intrinsics
    )
    
    result = agg.current_result()
    print("Current pose:", result.pose[:3, 3])
cap.release()

```

### Publishing to MQTT Brokers

Export poses as JSON and meshes as binary GLB for IoT platforms:

```python
import json
import paho.mqtt.client as mqtt
from lingbot_map.vis.glb_export import export_glb

client = mqtt.Client()
client.connect("broker.local", 1883)

def publish(result):
    # Serialize camera pose

    pose = result.pose.tolist()
    client.publish("lingbot/pose", json.dumps(pose))
    
    # Export and transmit mesh

    glb_path = "/tmp/scene.glb"
    export_glb(result.mesh, glb_path)
    with open(glb_path, "rb") as f:
        client.publish("lingbot/mesh", f.read(), qos=0)

# Call within your frame loop

publish(agg.current_result())

```

### Real-time Visualization

Use the built-in wrapper for debugging during development:

```python
from lingbot_map.vis.viser_wrapper import ViserWrapper

viewer = ViserWrapper()
while True:
    result = agg.current_result()
    viewer.update(result.point_cloud, result.mesh)
    if viewer.should_close():
        break

```

## Service Integration Patterns

Beyond the basic workflow, consider these architecture-specific patterns.

### ROS Robotics Integration

Subscribe to sensor topics and publish `geometry_msgs/Pose` and `sensor_msgs/PointCloud2` messages. Convert `result.pose` (the 4×4 numpy array) to ROS transform conventions. The [`benchmark/README.md`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/README.md) file documents how to register lingbot-map as a "method" within larger evaluation frameworks, which applies directly to ROS node packaging.

### Web Service Endpoints

Deploy lingbot-map behind a Flask or FastAPI server. Accept multipart uploads of RGB-D frames, process them through the aggregator, and return GLB files generated via `export_glb()`. Store checkpoints server-side to minimize per-request loading overhead.

## Summary

- **Use `GCTStream`** from [`lingbot_map/models/gct_stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/models/gct_stream.py) to load neural network checkpoints and run inference on RGB-D frames.
- **Configure `StreamAggregator`** from [`lingbot_map/aggregator/stream.py`](https://github.com/Robbyant/lingbot-map/blob/main/lingbot_map/aggregator/stream.py) with an appropriate window size to buffer and fuse reconstructions.
- **Implement data sources** as iterators yielding `(timestamp, rgb, depth, intrinsics)` tuples to abstract input origins.
- **Export results** using [`glb_export.py`](https://github.com/Robbyant/lingbot-map/blob/main/glb_export.py) for web viewers or serialize poses directly for robotics middleware.
- **Reference [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py)** for a complete end-to-end pipeline example.

## Frequently Asked Questions

### What data format does lingbot-map require?

The library expects iterators yielding tuples of `(timestamp, rgb_image, depth_image, intrinsics)`. Images should be numpy arrays with RGB channels, and intrinsics must provide camera calibration parameters. The `process_frame()` method handles the internal preprocessing.

### Can lingbot-map run on edge devices?

Yes. The `GCTStream` class supports ONNX Runtime backends, enabling deployment on edge devices with limited GPU resources. The aggregation layer operates independently of the model backend, allowing you to optimize each component separately for your hardware.

### How do I export reconstructions for web viewers?

Import `export_glb()` from `lingbot_map.vis.glb_export` to convert mesh objects into GLB format. This binary format loads directly in Three.js-based viewers and can be served via CDN or embedded in React applications.

### Where can I find the complete integration example?

The repository includes [`demo.py`](https://github.com/Robbyant/lingbot-map/blob/main/demo.py) at the root level, which demonstrates video file ingestion, model execution, and real-time visualization. For benchmarking integration patterns, consult [`benchmark/README.md`](https://github.com/Robbyant/lingbot-map/blob/main/benchmark/README.md), which explains how to wrap lingbot-map as a "method" in external evaluation pipelines.