How to Integrate lingbot-map with Other Services: A Complete Guide

To integrate lingbot-map with external services, instantiate the GCTStream model and StreamAggregator to process RGB-D frames, then publish the resulting pose and mesh data through your preferred transport protocol (ROS, MQTT, or HTTP endpoints).

The lingbot-map repository by Robbyant provides a modular Python library for real-time 3D reconstruction. Its clean separation between model inference, data aggregation, and visualization layers allows seamless integration with robotics middleware, cloud backends, and web services.

Architecture Overview

The codebase organizes functionality into three distinct layers that communicate through well-defined Python classes.

Model Layer

Located in lingbot_map/models/, specifically gct_stream.py and gct_stream_window.py, this layer encapsulates the neural network backend. The GCTStream class loads pretrained checkpoints and processes incoming RGB-D frames to generate live point clouds and meshes.

Aggregation Layer

Implemented in lingbot_map/aggregator/stream.py, the StreamAggregator class manages a sliding window of frames. It buffers incoming data, runs inference through the model, and fuses partial reconstructions into a global scene representation.

Visualization Layer

Found under lingbot_map/vis/, this layer includes viser_wrapper.py for Open3D-based rendering, point_cloud_viewer.py for local debugging, and glb_export.py for generating web-compatible mesh files. These wrappers enable both real-time visualization and service-oriented export workflows.

Integration Workflow

Connecting lingbot-map to external systems requires four specific steps. Each step leverages the isolated architecture to minimize coupling with your existing infrastructure.

1. Create a Data Source

Construct an iterator that yields (timestamp, rgb_image, depth_image, intrinsics) tuples. This abstraction allows the pipeline to consume data from video files, live camera drivers, or network sockets without modifying core library code.

2. Instantiate the Streaming Pipeline

Import the core classes and initialize them with your checkpoint path:

from lingbot_map.models.gct_stream import GCTStream
from lingbot_map.aggregator.stream import StreamAggregator

model = GCTStream(checkpoint_path="/path/to/lingbot-map.pt")
aggregator = StreamAggregator(model=model, window_size=8)

The window_size parameter controls the temporal buffer for smoothing reconstructions.

3. Process Incoming Frames

Feed your data source into the aggregator within a processing loop:

for ts, rgb, depth, intr in my_source():
    aggregator.process_frame(ts, rgb, depth, intr)
    result = aggregator.current_result()
    # result contains point_cloud, mesh, and pose (4×4 numpy array)

4. Export or Publish Results

Convert the LingbotMapResult object into your target format. For robotics systems, serialize result.pose as a transformation matrix. For web services, export the mesh to GLB format using lingbot_map.vis.glb_export.export_glb().

Integration Examples

The following patterns demonstrate concrete implementations for common service architectures.

Processing Video Files

This example reads side-by-side RGB-D recordings using OpenCV:

import cv2
from lingbot_map.models.gct_stream import GCTStream
from lingbot_map.aggregator.stream import StreamAggregator

# Initialize pipeline

model = GCTStream(checkpoint_path="lingbot-map.pt")
agg = StreamAggregator(model=model, window_size=8)

# Process video stream

cap = cv2.VideoCapture("my_dataset/video.mp4")
while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    rgb = frame[:, :frame.shape[1] // 2, :]
    depth = frame[:, frame.shape[1] // 2:, :]
    
    agg.process_frame(
        ts=cap.get(cv2.CAP_PROP_POS_MSEC),
        rgb=rgb,
        depth=depth,
        intrinsics=my_intrinsics
    )
    
    result = agg.current_result()
    print("Current pose:", result.pose[:3, 3])
cap.release()

Publishing to MQTT Brokers

Export poses as JSON and meshes as binary GLB for IoT platforms:

import json
import paho.mqtt.client as mqtt
from lingbot_map.vis.glb_export import export_glb

client = mqtt.Client()
client.connect("broker.local", 1883)

def publish(result):
    # Serialize camera pose

    pose = result.pose.tolist()
    client.publish("lingbot/pose", json.dumps(pose))
    
    # Export and transmit mesh

    glb_path = "/tmp/scene.glb"
    export_glb(result.mesh, glb_path)
    with open(glb_path, "rb") as f:
        client.publish("lingbot/mesh", f.read(), qos=0)

# Call within your frame loop

publish(agg.current_result())

Real-time Visualization

Use the built-in wrapper for debugging during development:

from lingbot_map.vis.viser_wrapper import ViserWrapper

viewer = ViserWrapper()
while True:
    result = agg.current_result()
    viewer.update(result.point_cloud, result.mesh)
    if viewer.should_close():
        break

Service Integration Patterns

Beyond the basic workflow, consider these architecture-specific patterns.

ROS Robotics Integration

Subscribe to sensor topics and publish geometry_msgs/Pose and sensor_msgs/PointCloud2 messages. Convert result.pose (the 4×4 numpy array) to ROS transform conventions. The benchmark/README.md file documents how to register lingbot-map as a "method" within larger evaluation frameworks, which applies directly to ROS node packaging.

Web Service Endpoints

Deploy lingbot-map behind a Flask or FastAPI server. Accept multipart uploads of RGB-D frames, process them through the aggregator, and return GLB files generated via export_glb(). Store checkpoints server-side to minimize per-request loading overhead.

Summary

• Use GCTStream from lingbot_map/models/gct_stream.py to load neural network checkpoints and run inference on RGB-D frames.
• Configure StreamAggregator from lingbot_map/aggregator/stream.py with an appropriate window size to buffer and fuse reconstructions.
• Implement data sources as iterators yielding (timestamp, rgb, depth, intrinsics) tuples to abstract input origins.
• Export results using glb_export.py for web viewers or serialize poses directly for robotics middleware.
• Reference demo.py for a complete end-to-end pipeline example.

Frequently Asked Questions

What data format does lingbot-map require?

The library expects iterators yielding tuples of (timestamp, rgb_image, depth_image, intrinsics). Images should be numpy arrays with RGB channels, and intrinsics must provide camera calibration parameters. The process_frame() method handles the internal preprocessing.

Can lingbot-map run on edge devices?

Yes. The GCTStream class supports ONNX Runtime backends, enabling deployment on edge devices with limited GPU resources. The aggregation layer operates independently of the model backend, allowing you to optimize each component separately for your hardware.

How do I export reconstructions for web viewers?

Import export_glb() from lingbot_map.vis.glb_export to convert mesh objects into GLB format. This binary format loads directly in Three.js-based viewers and can be served via CDN or embedded in React applications.

Where can I find the complete integration example?

The repository includes demo.py at the root level, which demonstrates video file ingestion, model execution, and real-time visualization. For benchmarking integration patterns, consult benchmark/README.md, which explains how to wrap lingbot-map as a "method" in external evaluation pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}