How Robot Temporal States Are Managed Using Memory Modules in DimOS

DimOS manages robot temporal states through a TemporalMemory orchestrator that streams camera frames, extracts temporal windows for Vision-Language Model (VLM) analysis, and maintains a thread-safe TemporalState container with persistent graph storage, enabling the LLM agent to query historical context across runs.

DimOS (dimensionalOS/dimos) provides a sophisticated temporal memory system that allows robots to maintain coherent awareness of their environment over time. The architecture centers on a TemporalMemory module that processes visual input and a TemporalState container that safeguards the robot's evolving knowledge. Together, these components create a queryable history of entities, actions, and spatial relationships that persists across robot sessions.

Core Architecture Components

TemporalMemory Orchestrator

The TemporalMemory class, defined in dimos/perception/experimental/temporal_memory/temporal_memory.py starting at line 109, serves as the primary orchestrator. It manages the reactive data pipeline, coordinates VLM inference, handles persistence, and exposes the query skill to the agent. The module initializes a FrameWindowAccumulator for buffering frames, a TemporalState instance (self._state) for in-memory knowledge, and an EntityGraphDB for relational persistence.

TemporalState Container

Located in dimos/perception/experimental/temporal_memory/temporal_state.py at line 25, the TemporalState class provides a thread-safe, mutable representation of the robot's temporal knowledge. It maintains a roster of known entities, a rolling natural-language summary, a buffer of recent "chunks" (parsed window results), and timestamps for summary generation. All mutations are guarded by a lock, and the snapshot() method returns a deep-copy for safe concurrent access.

EntityGraphDB Persistence

The EntityGraphDB class in dimos/perception/experimental/temporal_memory/entity_graph_db.py persists entity-to-entity relationships and world positions using SQLite. This enables fast graph-based queries across runs, allowing the robot to recall spatial relationships like "where did I leave my keys?" even after rebooting.

Configuration Management

TemporalMemoryConfig, found in the same file as TemporalMemory at lines 66-98, centralizes tunable parameters including fps, window_s, stride_s, summary_interval_s, and persistence locations. This design allows the same module to be reused across different robots or simulations without code changes.

The Temporal Processing Pipeline

The system processes robot temporal states through a reactive pipeline that bridges perception and memory:

1. Configuration Initialization – A TemporalMemoryConfig object specifies frame rates, window sizes, and persistence options. The new_memory flag controls whether to wipe existing databases on startup.

2. Reactive Frame Ingestion – When TemporalMemory.start() (lines 9-24) is called, the module subscribes its color_image input to an RxPY Subject. A "sharpness barrier" drops blurry frames before forwarding clean data to the accumulator.

3. Window Extraction – Every stride_s seconds, an RxPY interval (line 45) triggers _analyze_window() (lines 86-102). The accumulator extracts contiguous frames respecting window_s, skipping stale scenes using temporal_utils.is_scene_stale.

4. VLM Analysis – Selected keyframes are sent to a Vision-Language Model via a lazily-instantiated WindowAnalyzer. The VLM returns structured JSON payloads describing entities and relationships.

5. State Mutation – self._state.update_from_window() (lines 72-98) merges new entities, auto-adds referenced IDs, updates last_present timestamps, and appends chunks to the buffer. It also determines whether a rolling summary is due (lines 143-146).

6. Rolling Summary Generation – If triggered, _update_rolling_summary() (lines 5-15) requests the VLM to compress recent chunks into a human-readable paragraph. self._state.apply_summary() (lines 49-62) stores this and advances next_summary_at_s.

7. Persistence and Graph Enrichment – Each VLM response appends to JSONL logs in both a per-run directory and a persistent location. When a graph database is present, self._graph_db.save_window_data (lines 85-95) stores the parsed window with the robot's world pose (_robot_x/_y/_z).

8. Agent Skill Exposure – The @skill-decorated query method (lines 30-61) pulls a consistent snapshot via self._state.snapshot(), merges recent windows, optionally builds graph context, and forwards questions to the VLM.

9. Lifecycle Management – TemporalMemory.stop() (lines 50-71) gracefully shuts down background threads, commits the graph database, clears buffers, and stops transport streams.

Integration Examples

Wiring TemporalMemory into a Robot Blueprint

Robot-specific blueprints wire the generic TemporalMemory into the dataflow. The following example from dimos/robot/unitree/go2/blueprints/agentic/unitree_go2_temporal_memory.py (lines 21-26) demonstrates integration with the Go2 quadruped:

from dimos.core.blueprints import autoconnect
from dimos.perception.experimental.temporal_memory import (
    TemporalMemoryConfig,
    temporal_memory,
)
from dimos.robot.unitree.go2.blueprints.agentic.unitree_go2_agentic import (
    unitree_go2_agentic,
)
from dimos.core.global_config import global_config

my_go2_with_memory = autoconnect(
    unitree_go2_agentic,
    temporal_memory(
        config=TemporalMemoryConfig(
            fps=2.0,
            window_s=6.0,
            stride_s=3.0,
            new_memory=global_config.new_memory,
        )
    ),
)

Querying Memory from the Agent

Once running, the LLM agent can invoke the temporal memory skill to query historical context:

# Start the robot with temporal memory

dimos run unitree_go2_temporal_memory

# Query the memory from CLI

dimos agent-send "What objects are currently on the table?"

The agent-send command forwards text to the running LLM agent, which invokes the query skill defined at lines 30-61 of temporal_memory.py.

Accessing State Programmatically

For testing or debugging, access the temporal state directly:

from dimos.perception.experimental.temporal_memory import temporal_memory

mem = temporal_memory()               # returns the blueprint (Module subclass)

mem.start()                           # start pipelines

# … after video ingestion …

state = mem.get_state()               # RPC call, returns dict with entity roster, summary, etc.

print(state["rolling_summary"])
mem.stop()

The get_state RPC accessor is defined at lines 36-44 of temporal_memory.py.

Inspecting Persisted Data

Query the SQLite graph database directly to inspect stored entities:

import sqlite3
from pathlib import Path

db_path = Path.home() / ".local" / "state" / "dimos" / "temporal_memory" / "entity_graph.db"
conn = sqlite3.connect(db_path)
cur = conn.cursor()
cur.execute("SELECT entity_id, descriptor FROM entities LIMIT 10")
for row in cur.fetchall():
    print(row)

The database location is defined in TemporalMemory.__init__ at lines 65-74.

Summary

• TemporalMemory orchestrates the reactive pipeline, VLM inference, and persistence layers in dimos/perception/experimental/temporal_memory/temporal_memory.py.
• TemporalState provides thread-safe, mutable storage for entity rosters and rolling summaries, with snapshot capabilities for concurrent access.
• EntityGraphDB persists spatial relationships and world poses across robot sessions using SQLite.
• The system uses RxPY (Subject, interval) for non-blocking frame ingestion and periodic window analysis.
• Configuration is centralized through TemporalMemoryConfig, enabling reuse across different robot platforms.
• The @skill decorator exposes the query method directly to the LLM agent, enabling natural language questions about historical context.

Frequently Asked Questions

How does DimOS ensure thread safety when updating temporal state?

The TemporalState class in temporal_state.py guards all mutations with a threading lock. When the VLM analysis completes and _analyze_window() calls update_from_window(), the state modifications are atomic. For read operations, the snapshot() method returns a deep copy of the current state, ensuring the LLM agent receives a consistent view even while new frames are being processed.

What persistence options are available for temporal memory?

DimOS implements a dual persistence strategy. Each run generates per-run JSONL logs stored in run_log_dir/temporal_memory/temporal_memory.jsonl for debugging specific sessions. Simultaneously, persistent storage accumulates across runs via entity_graph.db (SQLite) and a master temporal_memory.jsonl file. The new_memory configuration flag controls whether to wipe persistent storage on startup.

How can I configure the temporal memory module for different robots?

Use the TemporalMemoryConfig class to tune parameters without modifying source code. Key settings include fps (frame processing rate), window_s (duration of frame windows), stride_s (analysis frequency), and summary_interval_s (how often to generate rolling summaries). These configurations are passed through robot-specific blueprints, as demonstrated in the Go2 integration example.

What is the difference between the rolling summary and the chunk buffer?

The chunk buffer stores raw, structured JSON responses from recent VLM window analyses, providing granular access to specific time windows. The rolling summary is a compressed, natural-language paragraph generated periodically by asking the VLM to synthesize the chunk buffer contents. While the chunk buffer offers detailed evidence, the rolling summary provides immediate context for agent queries without overwhelming the LLM with raw JSON.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}