How to Implement Hierarchical Memory Systems: Episodic, Semantic, and Procedural Best Practices

Implement hierarchical memory systems using a three-layer architecture (working, long-term, external) with explicit promotion/demotion logic, organize fragments into four-level hierarchies (raw → semantic → temporal → conceptual), and tag entries by type (episodic, semantic, procedural) to enable adaptive retrieval and compression.

The davidkimai/context-engineering repository provides a production-ready framework for building hierarchical memory systems that mirror human cognitive architecture. This guide extracts implementation patterns from memory_management_lab.py and the cognitive architecture specifications to show you how to implement episodic, semantic, and procedural memory layers with proper promotion, compression, and retrieval mechanisms.

Architectural Overview of Hierarchical Memory Systems

Three-Layer Storage Architecture

The HierarchicalMemorySystem class in 00_COURSE/03_context_management/labs/memory_management_lab.py implements a tiered storage model:

• Working Memory: A fast, token-limited cache (default 5,000 tokens) for immediate conversational context. Implemented as WorkingMemory inside HierarchicalMemorySystem.
• Long-Term Memory: A persistent store (10M+ bytes) holding episodic, semantic, and procedural fragments. Implemented as LongTermMemory.
• External Retrieval: An optional plug-in hook (external_retriever) for vector databases or web search.

Memory Type Taxonomy

The repository distinguishes three fundamental memory types:

• Episodic: Time-stamped events (e.g., "User asked about hierarchical memory on 2024-02-28").
• Semantic: Extracted concepts and facts (e.g., "Hierarchical fragmentation uses four levels").
• Procedural: Learned procedures and policies (e.g., "If access_count > 3, promote to working memory").

Implementing the Core Memory Layers

Instantiating the Hierarchical Memory System

Create a system with explicit size constraints and optional external retrieval:

from memory_management_lab import HierarchicalMemorySystem

# Create a system with a 5k token working buffer

mem_sys = HierarchicalMemorySystem(
    working_memory_size=5000,         # tokens

    long_term_memory_size=10_000_000 # bytes

)

# Optional: plug in an external retriever (e.g., a vector DB)

def external_search(query, limit):
    # Replace with real vector search implementation

    return [("ext_1", "External result 1"), ("ext_2", "External result 2")]

mem_sys.external_retriever = external_search

Promotion and Demotion Logic

The repository implements explicit thresholds for moving data between layers:

• Promotion to Working Memory: Triggered when access_count > 3 or priority > 0.7.
• Demotion from Working Memory: Occurs when access_count < 2 and last_accessed > 6 hours.
• Long-Term Storage: Entries with priority > 0.5 or specific tags are written to LongTermMemory during the store() operation.

Periodically call optimize() to decay stale priorities (7-day half-life) and evict low-usage items.

Organizing Memory Fragments Hierarchically

The Four-Level Fragment Hierarchy

According to cognitive-tools/cognitive-architectures/reconstruction-memory-architecture.md, fragments organize into a four-level hierarchy:

1. Level 0 (Raw Fragments): Individual episodic events or data points.
2. Level 1 (Semantic Clusters): Grouped by conceptual similarity.
3. Level 2 (Temporal Sequences): Ordered series of clusters representing processes.
4. Level 3 (Conceptual Themes): High-level abstractions and schemas.

HierarchicalFragmentOrganizer Implementation

Use the HierarchicalFragmentOrganizer class to manage this structure:

from reconstruction_memory_architecture import HierarchicalFragmentOrganizer

organizer = HierarchicalFragmentOrganizer(max_levels=4)

# Assume `fragments` is a list of raw episodic snippets

fragments = [
    {"text": "User asked about hierarchical memory.", "timestamp": 1},
    {"text": "Explained three memory layers.", "timestamp": 2},
    # …

]

hierarchy = organizer.organize_fragments(fragments)

# Reconstruct using a cue (e.g., the word "memory")

cues = ["memory"]
reconstruction = organizer.reconstruct_with_hierarchy(cues, context={})
print(reconstruction)   # → high-level thematic summary + relevant details

Reconstruction proceeds top-down: activate themes → sequences → clusters → fragments.

Storing Episodic, Semantic, and Procedural Memory

Tag entries explicitly to distinguish memory types during retrieval:

# Episodic (event) entry

mem_sys.store(
    key="event_2024-02-28_meeting",
    content="Discussed hierarchical memory best practices with team.",
    tags=["episodic", "meeting"],
    priority=0.9
)

# Semantic (concept) entry

mem_sys.store(
    key="concept_hierarchical_fragmentation",
    content="Organize memory fragments into 4 hierarchical levels: raw → semantic → temporal → conceptual.",
    tags=["semantic", "memory"],
    priority=0.8
)

# Procedural (how-to) entry

mem_sys.store(
    key="procedure_promote_entry",
    content="If access_count > 3 or priority > 0.7, copy entry to working memory.",
    tags=["procedural", "policy"],
    priority=0.7
)

The retrieve() method automatically promotes frequently accessed entries to working memory based on the thresholds defined in the system configuration.

Multi-Modal Integration and Compression

Cross-Modal Memory Mapping

For systems handling diverse data types, implement multi-modal encoding as described in reconstruction-memory-architecture.md:

• Assign modality identifiers (['text'], ['visual'], ['auditory'], ['spatial'], ['temporal']).
• Use CrossModalMapper to discover correspondences between modalities.
• Build unified representations that enable procedural memories (e.g., "how to assemble a device") to coexist with textual facts.

Adaptive Compression Pipeline

The HierarchicalMemory class in 00_COURSE/02_context_processing/labs/long_context_lab.py implements a three-tier compression strategy:

from long_context_lab import HierarchicalMemory
import numpy as np

# d_model = dimensionality of the transformer representation

hm = HierarchicalMemory(d_model=256)

# Add a new context vector (e.g., a hidden state from a language model)

vector = np.random.randn(256)
hm.add_context(vector)

# Query with another vector; get the most relevant compressed representation

query_vec = np.random.randn(256)
relevant = hm.retrieve_relevant(query_vec, max_tokens=256)
print("Retrieved shape:", relevant.shape)

• Short-term: Raw vectors stored in working buffer.
• Medium-term: Compressed summaries using compress_medium (4× compression ratio).
• Long-term: Highly compressed gists using compress_long.

Tune the learned compression matrices on downstream tasks to preserve relevance while minimizing footprint.

Consolidated Best Practices Checklist

Follow these evidence-based guidelines derived from the Context-Engineering codebase:

• Size your layers: Configure working memory to your token budget (e.g., 5,000 tokens) and long-term memory to 10M+ bytes to keep inference cheap while preserving depth.
• Priority decay: Assign higher priority to frequently queried or critical facts, but decay priority after 7 days of inactivity to prevent stale entries from hogging space.
• Promotion thresholds: Promote entries to working memory when access_count > 3 or priority > 0.7; demote when access_count < 2 and last_accessed > 6 hours.
• Compression ratios: Use a 4× compression ratio when moving from medium-term to long-term storage, tuning the compress_medium and compress_long matrices on downstream tasks.
• Fragment indexing: Store each of the four hierarchy levels (raw, semantic clusters, temporal sequences, conceptual themes) in separate indexes to enable efficient top-down reconstruction.
• Modality tagging: Tag all entries with modality identifiers (text, visual, auditory) to ensure correct encoder selection and easier cross-modal mapping.
• Search order: Always search working memory first, then long-term with a fallback limit, then external retrievers, deduplicating keys before merging results.
• Monitoring: Expose get_statistics() to track hits, promotions, demotions, and utilization ratios, logging trends to data-drive layer resizing.
• Testing: Unit test store(), retrieve(), and optimize() methods, verifying that promotion/demotion respects the defined thresholds.

Summary

Implementing hierarchical memory systems requires balancing speed, capacity, and retrieval accuracy across three distinct layers. The key takeaways from the Context-Engineering repository include:

• Use a three-tier architecture (working, long-term, external) with explicit promotion thresholds (access_count > 3 or priority > 0.7) and demotion rules.
• Organize content into a four-level fragment hierarchy (raw → semantic clusters → temporal sequences → conceptual themes) using HierarchicalFragmentOrganizer.
• Tag entries by type (episodic, semantic, procedural) and modality to enable precise retrieval and cross-modal integration.
• Apply adaptive compression (4× ratio) when moving data to long-term storage, and decay priorities after 7 days to maintain relevance.
• Monitor system statistics via get_statistics() to optimize layer sizing and retrieval performance.

Frequently Asked Questions

What is the difference between episodic, semantic, and procedural memory in hierarchical systems?

Episodic memory stores time-stamped events and experiences (e.g., "User asked about hierarchical memory on 2024-02-28"), semantic memory stores facts and concepts (e.g., "Hierarchical fragmentation uses four levels"), and procedural memory stores learned skills and policies (e.g., "If access_count > 3, promote to working memory"). In the Context-Engineering repository, these are distinguished by tags passed to the store() method in memory_management_lab.py, enabling type-specific retrieval and reconstruction strategies.

How does promotion between working memory and long-term memory work?

Promotion occurs automatically when an entry's access_count exceeds 3 or its priority score exceeds 0.7, triggering a copy from LongTermMemory to WorkingMemory. Conversely, demotion occurs when access_count drops below 2 and last_accessed exceeds 6 hours. The optimize() method in HierarchicalMemorySystem handles these transitions while decaying priorities (7-day half-life) and evicting stale entries to maintain performance.

What compression strategy should I use for long-term storage?

Use a 4× compression ratio when transitioning from medium-term to long-term storage. The HierarchicalMemory class in long_context_lab.py implements learned compression matrices (compress_medium and compress_long) that should be fine-tuned on downstream tasks to preserve semantic relevance while minimizing footprint. This approach reduces memory usage without sacrificing the ability to reconstruct high-level themes during retrieval.

How do I organize memory fragments for efficient retrieval?

Implement a four-level hierarchy using HierarchicalFragmentOrganizer from reconstruction-memory-architecture.md: Level 0 (raw episodic fragments), Level 1 (semantic clusters), Level 2 (temporal sequences), and Level 3 (conceptual themes). Store each level in separate indexes and use top-down reconstruction (themes → sequences → clusters → fragments) when retrieving with cues. This structure enables efficient semantic lookup while preserving temporal and thematic context.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}