# How to Create Spatial Memory for Persistent Robot Navigation Using RAG in DimoS

> Create persistent robot spatial memory for navigation using RAG. DimoS converts video and odometry into a semantic vector database empowering LLMs for robust robot restarts.

- Repository: [Dimensional/dimos](https://github.com/dimensionalOS/dimos)
- Tags: how-to-guide
- Published: 2026-03-15

---

**DimoS provides a plug-and-play `SpatialMemory` module that converts live video streams and odometry into a persistent semantic vector database, enabling Large Language Models to perform Retrieval-Augmented Generation (RAG) for spatial reasoning and navigation even after robot restarts.**

The dimensionalOS (`dimos`) open-source framework implements a complete spatial memory stack that bridges computer vision, vector databases, and LLM agents. By storing CLIP embeddings of visual frames alongside precise pose metadata in ChromaDB, the system allows robots to answer natural language queries like "where is the kitchen?" or "go to the red chair" using semantic similarity search rather than hard-coded coordinates.

## Core Architecture of the Spatial Memory System

### The SpatialMemory Module

At the center of the stack is the **`SpatialMemory`** class defined in [`dimos/perception/spatial_perception.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/spatial_perception.py). This module subscribes to a camera's `color_image` topic, processes incoming frames, and manages the entire data pipeline from raw pixels to queryable vectors.

The module initializes with configurable thresholds: **`min_distance_threshold`** (meters) and **`min_time_threshold`** (seconds) ensure the database only stores novel frames when the robot has physically moved or sufficient time has passed, preventing redundant entries.

### Supporting Components

**`ImageEmbeddingProvider`** ([`dimos/agents_deprecated/memory/image_embedding.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/image_embedding.py)) wraps the vision model (default **CLIP**) and returns fixed-size **512-dimensional embeddings** for any `np.ndarray` frame passed to `get_embedding()`.

**`SpatialVectorDB`** ([`dimos/agents_deprecated/memory/spatial_vector_db.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/spatial_vector_db.py)) provides a thin abstraction over ChromaDB, handling `add_image_vector()` for inserts and exposing query methods: `query_by_text()`, `query_by_image()`, and `query_by_location()`. It stores raw images, embeddings, and `RobotLocation` metadata.

**`VisualMemory`** ([`dimos/agents_deprecated/memory/visual_memory.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/visual_memory.py)) serializes raw frames to disk as `visual_memory.pkl` every 100 frames or on shutdown, ensuring image data survives process restarts.

**`RobotLocation`** ([`dimos/types/robot_location.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/types/robot_location.py)) is a dataclass representing named poses with position (x, y, z), rotation (roll, pitch, yaw), and metadata tags, enabling human-readable location retrieval.

## Data Flow: From Camera to Persistent Vector Store

The `SpatialMemory.start()` method initiates a streaming pipeline with seven distinct stages:

1. **Capture**: The module subscribes to `color_image`, converting each `Image` message to a NumPy array via `cv2.cvtColor`.

2. **Pose Lookup**: For every candidate frame, the system queries the transform tree: `self.tf.get("world", "base_link")` retrieves the current robot pose in the world frame.

3. **Filtering**: Frames are discarded unless the robot has traveled at least `min_distance_threshold` meters and `min_time_threshold` seconds has elapsed since the last stored frame.

4. **Embedding**: Valid frames pass through `ImageEmbeddingProvider.get_embedding()`, producing a 512-dim CLIP vector.

5. **Metadata Assembly**: The system constructs a primitive dictionary containing pose coordinates (x, y, z, roll, pitch, yaw), timestamp, and a unique `frame_id`.

6. **Vector DB Insert**: `vector_db.add_image_vector()` commits the embedding, raw image bytes, and metadata to ChromaDB at the persistent path `_SPATIAL_MEMORY_DIR / "chromadb_data"`.

7. **Visual Persistence**: Raw frames accumulate in memory and flush to `visual_memory.pkl` via `VisualMemory.save()` every 100 frames or on graceful shutdown.

## Implementing RAG-Based Navigation Queries

### Semantic Query Methods

The `SpatialVectorDB` class exposes three primary retrieval modes that enable RAG pipelines:

- **`query_by_text(text)`**: Encodes the query string through CLIP's text encoder and performs nearest-neighbor search against stored image embeddings, returning frames semantically matching descriptions like "kitchen counter" or "office door".

- **`query_by_image(image)`**: Accepts a NumPy array, generates its embedding, and retrieves visually similar locations from the map.

- **`query_by_location(x, y, radius)`**: Performs geometric nearest-neighbor search using stored pose metadata, returning all frames captured within `radius` meters of the specified coordinates.

### Tagging and LLM Integration

**`add_named_location(name)`** and **`tag_location(RobotLocation)`** (exposed via RPC) allow the system or human operators to tag the robot's current pose with human-readable labels. These tagged locations are stored in `self.robot_locations` and retrievable via `query_tagged_location(name)`.

The **`NavigationSkillContainer`** in [`dimos/agents/skills/navigation.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents/skills/navigation.py) exposes these capabilities as **`@skill`** methods callable by LLMs. When processing a command like "go to the office", the skill executes a RAG sequence: first checking `query_tagged_location("office")`, then falling back to `query_by_text("office")` if no tag exists, and finally converting the retrieved frame's metadata into a `PoseStamped` goal for the navigation stack.

## Code Examples

### Deploying SpatialMemory in a Custom Blueprint

```python

# blueprint.py

from dimos import spec
from dimos.core.blueprints import autoconnect
from dimos.perception.spatial_perception import SpatialMemory

def my_robot_blueprint(camera: spec.Camera):
    # Deploy with persistent storage path

    spatial_memory = autoconnect(
        SpatialMemory(
            collection_name="my_robot_memory",
            embedding_model="clip",
            db_path="/tmp/my_robot_spatial_db",
        )
    ).build()

    # Wire camera stream

    spatial_memory.color_image.connect(camera.color_image)
    spatial_memory.start()
    return spatial_memory

```

Pass `db_path` to enable persistence across restarts. The `autoconnect` utility handles dependency injection for the transform interface and other required components.

### Tagging Locations for Named Navigation

```python

# Interactive tagging or LLM tool call

memory = dimos.deploy(SpatialMemory)  # Assumes existing deployment

# Tag current pose as "kitchen"

success = memory.add_named_location(name="kitchen")
print(success)  # → True

```

Under the hood, `add_named_location()` fetches the latest TF pose, instantiates a `RobotLocation` dataclass, and appends it to the internal locations list.

### Executing RAG Text Queries for Navigation

```python

# LLM skill: navigate_with_text("go to the office")

memory = dimos.deploy(SpatialMemory)
hits = memory.query_by_text("office")

if hits:
    meta = hits[0]["metadata"][0]
    goal_pose = PoseStamped(
        position=make_vector3(meta["pos_x"], meta["pos_y"], meta["pos_z"]),
        orientation=Quaternion.from_euler(make_vector3(0, 0, meta["rot_z"])),
        frame_id="map",
    )
    navigation_interface.set_goal(goal_pose)

```

This mirrors the implementation in `NavigationSkillContainer.navigate_using_semantic_map`, which returns a human-readable status string to the LLM upon goal submission.

### Recovering Memory After Restart

```python

# Robot boot sequence

memory = dimos.deploy(
    SpatialMemory,
    db_path="/tmp/my_robot_spatial_db",  # Same path as previous run

    new_memory=False,                    # Load existing ChromaDB

)

# Verify persistence

stats = memory.get_stats()
print(f"Frames recovered: {stats['stored_frame_count']}")

```

Setting `new_memory=False` instructs the module to load the existing ChromaDB collection from disk and rehydrate the `VisualMemory` from `visual_memory.pkl`, eliminating the need to reprocess historical video.

## Key Source Files and Implementation Details

| File | Purpose |
|------|---------|
| [`dimos/perception/spatial_perception.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/spatial_perception.py) | Core **SpatialMemory** module with frame filtering, embedding orchestration, and database integration. |
| [`dimos/agents_deprecated/memory/spatial_vector_db.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/spatial_vector_db.py) | **ChromaDB** wrapper for vector storage and multimodal queries (text, image, location). |
| [`dimos/agents_deprecated/memory/image_embedding.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/image_embedding.py) | **CLIP** embedding provider for vision-language alignment. |
| [`dimos/agents_deprecated/memory/visual_memory.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents_deprecated/memory/visual_memory.py) | Disk serialization for raw frame persistence across restarts. |
| [`dimos/types/robot_location.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/types/robot_location.py) | Portable **RobotLocation** dataclass for named pose storage. |
| [`dimos/agents/skills/navigation.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/agents/skills/navigation.py) | LLM-exposed **navigation skills** implementing RAG retrieval logic. |
| [`dimos/robot/unitree/go2/blueprints/smart/unitree_go2_spatial.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/robot/unitree/go2/blueprints/smart/unitree_go2_spatial.py) | Production blueprint wiring camera, `SpatialMemory`, and agent skills. |

## Summary

- **DimoS** implements **spatial memory for persistent robot navigation using RAG** through the `SpatialMemory` module in [`dimos/perception/spatial_perception.py`](https://github.com/dimensionalOS/dimos/blob/main/dimos/perception/spatial_perception.py).
- The system filters incoming video by **distance and time thresholds**, then stores **512-dim CLIP embeddings** with pose metadata in a **persistent ChromaDB** instance.
- **RAG queries** support text, image, and geometric lookups via `query_by_text()`, `query_by_image()`, and `query_by_location()`, enabling LLMs to reason about space semantically.
- **Named locations** tagged via `add_named_location()` provide deterministic retrieval points for high-level navigation commands.
- **Persistence** is achieved through ChromaDB's native storage and `VisualMemory`'s pickle serialization, allowing full map recovery after robot shutdown via `new_memory=False`.

## Frequently Asked Questions

### How does DimoS ensure spatial memory persists after robot shutdown?

The system uses two persistence mechanisms: **ChromaDB** stores vector embeddings and metadata at a configurable persistent path (`db_path`), while **`VisualMemory`** serializes raw frames to `visual_memory.pkl` on disk. On restart, initializing `SpatialMemory` with `new_memory=False` automatically reloads both the vector database and the image cache, restoring the complete semantic map without reprocessing video.

### What embedding model does SpatialMemory use for RAG queries?

By default, **`SpatialMemory`** uses **CLIP** (Contrastive Language-Image Pre-training) through the `ImageEmbeddingProvider` class. This model generates **512-dimensional embeddings** that align visual frames with text descriptions, enabling cross-modal retrieval where text queries match semantically relevant images from the robot's history.

### Can I query the spatial memory using natural language commands?

Yes. The `query_by_text()` method in `SpatialVectorDB` accepts arbitrary natural language strings (e.g., "kitchen counter", "red chair"), encodes them using CLIP's text encoder, and returns the nearest-neighbor image frames from the database. The `NavigationSkillContainer` exposes this as an LLM skill, allowing language models to convert commands like "go to the office" into navigation goals via semantic retrieval.

### How does the robot decide which frames to store in memory?

The `SpatialMemory` module applies **dual filtering criteria**: a frame is only stored if the robot has moved at least **`min_distance_threshold`** meters **and** at least **`min_time_threshold`** seconds has passed since the last capture. These configurable parameters prevent database bloat from redundant frames while ensuring sufficient spatial coverage for accurate RAG retrieval.