How to Create Spatial Memory for Persistent Robot Navigation Using RAG in DimoS

DimoS provides a plug-and-play SpatialMemory module that converts live video streams and odometry into a persistent semantic vector database, enabling Large Language Models to perform Retrieval-Augmented Generation (RAG) for spatial reasoning and navigation even after robot restarts.

The dimensionalOS (dimos) open-source framework implements a complete spatial memory stack that bridges computer vision, vector databases, and LLM agents. By storing CLIP embeddings of visual frames alongside precise pose metadata in ChromaDB, the system allows robots to answer natural language queries like "where is the kitchen?" or "go to the red chair" using semantic similarity search rather than hard-coded coordinates.

Core Architecture of the Spatial Memory System

The SpatialMemory Module

At the center of the stack is the SpatialMemory class defined in dimos/perception/spatial_perception.py. This module subscribes to a camera's color_image topic, processes incoming frames, and manages the entire data pipeline from raw pixels to queryable vectors.

The module initializes with configurable thresholds: min_distance_threshold (meters) and min_time_threshold (seconds) ensure the database only stores novel frames when the robot has physically moved or sufficient time has passed, preventing redundant entries.

Supporting Components

ImageEmbeddingProvider (dimos/agents_deprecated/memory/image_embedding.py) wraps the vision model (default CLIP) and returns fixed-size 512-dimensional embeddings for any np.ndarray frame passed to get_embedding().

SpatialVectorDB (dimos/agents_deprecated/memory/spatial_vector_db.py) provides a thin abstraction over ChromaDB, handling add_image_vector() for inserts and exposing query methods: query_by_text(), query_by_image(), and query_by_location(). It stores raw images, embeddings, and RobotLocation metadata.

VisualMemory (dimos/agents_deprecated/memory/visual_memory.py) serializes raw frames to disk as visual_memory.pkl every 100 frames or on shutdown, ensuring image data survives process restarts.

RobotLocation (dimos/types/robot_location.py) is a dataclass representing named poses with position (x, y, z), rotation (roll, pitch, yaw), and metadata tags, enabling human-readable location retrieval.

Data Flow: From Camera to Persistent Vector Store

The SpatialMemory.start() method initiates a streaming pipeline with seven distinct stages:

1. Capture: The module subscribes to color_image, converting each Image message to a NumPy array via cv2.cvtColor.

2. Pose Lookup: For every candidate frame, the system queries the transform tree: self.tf.get("world", "base_link") retrieves the current robot pose in the world frame.

3. Filtering: Frames are discarded unless the robot has traveled at least min_distance_threshold meters and min_time_threshold seconds has elapsed since the last stored frame.

4. Embedding: Valid frames pass through ImageEmbeddingProvider.get_embedding(), producing a 512-dim CLIP vector.

5. Metadata Assembly: The system constructs a primitive dictionary containing pose coordinates (x, y, z, roll, pitch, yaw), timestamp, and a unique frame_id.

6. Vector DB Insert: vector_db.add_image_vector() commits the embedding, raw image bytes, and metadata to ChromaDB at the persistent path _SPATIAL_MEMORY_DIR / "chromadb_data".

7. Visual Persistence: Raw frames accumulate in memory and flush to visual_memory.pkl via VisualMemory.save() every 100 frames or on graceful shutdown.

Implementing RAG-Based Navigation Queries

Semantic Query Methods

The SpatialVectorDB class exposes three primary retrieval modes that enable RAG pipelines:

• query_by_text(text): Encodes the query string through CLIP's text encoder and performs nearest-neighbor search against stored image embeddings, returning frames semantically matching descriptions like "kitchen counter" or "office door".

• query_by_image(image): Accepts a NumPy array, generates its embedding, and retrieves visually similar locations from the map.

• query_by_location(x, y, radius): Performs geometric nearest-neighbor search using stored pose metadata, returning all frames captured within radius meters of the specified coordinates.

Tagging and LLM Integration

add_named_location(name) and tag_location(RobotLocation) (exposed via RPC) allow the system or human operators to tag the robot's current pose with human-readable labels. These tagged locations are stored in self.robot_locations and retrievable via query_tagged_location(name).

The NavigationSkillContainer in dimos/agents/skills/navigation.py exposes these capabilities as @skill methods callable by LLMs. When processing a command like "go to the office", the skill executes a RAG sequence: first checking query_tagged_location("office"), then falling back to query_by_text("office") if no tag exists, and finally converting the retrieved frame's metadata into a PoseStamped goal for the navigation stack.

Code Examples

Deploying SpatialMemory in a Custom Blueprint

# blueprint.py

from dimos import spec
from dimos.core.blueprints import autoconnect
from dimos.perception.spatial_perception import SpatialMemory

def my_robot_blueprint(camera: spec.Camera):
    # Deploy with persistent storage path

    spatial_memory = autoconnect(
        SpatialMemory(
            collection_name="my_robot_memory",
            embedding_model="clip",
            db_path="/tmp/my_robot_spatial_db",
        )
    ).build()

    # Wire camera stream

    spatial_memory.color_image.connect(camera.color_image)
    spatial_memory.start()
    return spatial_memory

Pass db_path to enable persistence across restarts. The autoconnect utility handles dependency injection for the transform interface and other required components.

Tagging Locations for Named Navigation

# Interactive tagging or LLM tool call

memory = dimos.deploy(SpatialMemory)  # Assumes existing deployment

# Tag current pose as "kitchen"

success = memory.add_named_location(name="kitchen")
print(success)  # → True

Under the hood, add_named_location() fetches the latest TF pose, instantiates a RobotLocation dataclass, and appends it to the internal locations list.

Executing RAG Text Queries for Navigation

# LLM skill: navigate_with_text("go to the office")

memory = dimos.deploy(SpatialMemory)
hits = memory.query_by_text("office")

if hits:
    meta = hits[0]["metadata"][0]
    goal_pose = PoseStamped(
        position=make_vector3(meta["pos_x"], meta["pos_y"], meta["pos_z"]),
        orientation=Quaternion.from_euler(make_vector3(0, 0, meta["rot_z"])),
        frame_id="map",
    )
    navigation_interface.set_goal(goal_pose)

This mirrors the implementation in NavigationSkillContainer.navigate_using_semantic_map, which returns a human-readable status string to the LLM upon goal submission.

Recovering Memory After Restart

# Robot boot sequence

memory = dimos.deploy(
    SpatialMemory,
    db_path="/tmp/my_robot_spatial_db",  # Same path as previous run

    new_memory=False,                    # Load existing ChromaDB

)

# Verify persistence

stats = memory.get_stats()
print(f"Frames recovered: {stats['stored_frame_count']}")

Setting new_memory=False instructs the module to load the existing ChromaDB collection from disk and rehydrate the VisualMemory from visual_memory.pkl, eliminating the need to reprocess historical video.

Key Source Files and Implementation Details

File	Purpose
dimos/perception/spatial_perception.py	Core SpatialMemory module with frame filtering, embedding orchestration, and database integration.
dimos/agents_deprecated/memory/spatial_vector_db.py	ChromaDB wrapper for vector storage and multimodal queries (text, image, location).
dimos/agents_deprecated/memory/image_embedding.py	CLIP embedding provider for vision-language alignment.
dimos/agents_deprecated/memory/visual_memory.py	Disk serialization for raw frame persistence across restarts.
dimos/types/robot_location.py	Portable RobotLocation dataclass for named pose storage.
dimos/agents/skills/navigation.py	LLM-exposed navigation skills implementing RAG retrieval logic.
dimos/robot/unitree/go2/blueprints/smart/unitree_go2_spatial.py	Production blueprint wiring camera, SpatialMemory, and agent skills.

Summary

• DimoS implements spatial memory for persistent robot navigation using RAG through the SpatialMemory module in dimos/perception/spatial_perception.py.
• The system filters incoming video by distance and time thresholds, then stores 512-dim CLIP embeddings with pose metadata in a persistent ChromaDB instance.
• RAG queries support text, image, and geometric lookups via query_by_text(), query_by_image(), and query_by_location(), enabling LLMs to reason about space semantically.
• Named locations tagged via add_named_location() provide deterministic retrieval points for high-level navigation commands.
• Persistence is achieved through ChromaDB's native storage and VisualMemory's pickle serialization, allowing full map recovery after robot shutdown via new_memory=False.

Frequently Asked Questions

How does DimoS ensure spatial memory persists after robot shutdown?

The system uses two persistence mechanisms: ChromaDB stores vector embeddings and metadata at a configurable persistent path (db_path), while VisualMemory serializes raw frames to visual_memory.pkl on disk. On restart, initializing SpatialMemory with new_memory=False automatically reloads both the vector database and the image cache, restoring the complete semantic map without reprocessing video.

What embedding model does SpatialMemory use for RAG queries?

By default, SpatialMemory uses CLIP (Contrastive Language-Image Pre-training) through the ImageEmbeddingProvider class. This model generates 512-dimensional embeddings that align visual frames with text descriptions, enabling cross-modal retrieval where text queries match semantically relevant images from the robot's history.

Can I query the spatial memory using natural language commands?

Yes. The query_by_text() method in SpatialVectorDB accepts arbitrary natural language strings (e.g., "kitchen counter", "red chair"), encodes them using CLIP's text encoder, and returns the nearest-neighbor image frames from the database. The NavigationSkillContainer exposes this as an LLM skill, allowing language models to convert commands like "go to the office" into navigation goals via semantic retrieval.

How does the robot decide which frames to store in memory?

The SpatialMemory module applies dual filtering criteria: a frame is only stored if the robot has moved at least min_distance_threshold meters and at least min_time_threshold seconds has passed since the last capture. These configurable parameters prevent database bloat from redundant frames while ensuring sufficient spatial coverage for accurate RAG retrieval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}