mem::consolidate vs mem::consolidate-pipeline: Key Differences in agentmemory

mem::consolidate transforms raw coding-session observations into long-term semantic memories immediately, while mem::consolidate-pipeline orchestrates a periodic multi-stage workflow that merges facts, extracts procedures, applies decay, and optionally exports to external systems like Obsidian.

The rohitg00/agentmemory repository provides two distinct consolidation mechanisms for AI agent knowledge bases. While both functions are exposed as MCP tools and REST endpoints, they serve complementary roles in the memory management lifecycle. Understanding when to use each ensures optimal performance for both real-time synthesis and long-term knowledge maintenance.

Core Architectural Distinctions

mem::consolidate: Observation-to-Memory Conversion

mem::consolidate performs a focused, single-pass conversion of raw observations into durable semantic memories. When triggered, it loads recent coding sessions from the KV.observations store, filters for high-importance entries with titles, and groups them by conceptual similarity.

The function calls the LLM compression endpoint using the CONSOLIDATION_SYSTEM prompt to synthesize observation groups. It then parses XML responses to either evolve existing memories or create new entries in the Memory collection. Each mutation generates an individual audit entry with "evolve" or "remember" actions.

mem::consolidate-pipeline: Multi-Tier Maintenance Orchestration

mem::consolidate-pipeline manages the broader health of the knowledge base through a configurable sequence of operations. Unlike the single-purpose consolidation function, this pipeline operates across semantic, procedural, and pattern memory tiers.

The pipeline executes sequentially:

1. Semantic merge: Pulls recent session summaries and merges facts using SEMANTIC_MERGE_SYSTEM, upserting into SemanticMemory
2. Reflect: Invokes mem::reflect to re-cluster memories
3. Procedural extraction: Identifies recurring patterns using PROCEDURAL_EXTRACTION_SYSTEM, storing in ProceduralMemory
4. Decay: Applies exponential strength decay based on CONSOLIDATION_DECAY_DAYS
5. Obsidian export: Optionally triggers mem::obsidian-export for external backup

A single audit entry records the entire pipeline execution with the tier configuration and stage results.

Source Code Implementation

The consolidation logic resides in specific function implementations within the repository.

src/functions/consolidate.ts (lines 70‑124 and 146‑216) contains the core observation processing loop. This implementation handles session fetching, concept grouping, and the up-to-10 LLM compress calls required to synthesize memories.

src/functions/consolidation-pipeline.ts (lines 45‑71, 115‑176, 190‑262, and 285‑311) orchestrates the multi-stage workflow. Each code block manages a distinct phase, from semantic merging through decay calculations.

Configuration checks reside in src/config.ts, which exposes isConsolidationEnabled() used by the pipeline to respect the CONSOLIDATION_ENABLED flag. Both functions register via src/triggers/api.ts (REST endpoints /agentmemory/consolidate and /agentmemory/consolidate-pipeline) and src/mcp/tools-registry.ts (MCP tool memory_consolidate).

API Signatures and Return Values

mem::consolidate accepts optional parameters for project filtering and minObservations thresholds. It returns a concise status object:

{
  consolidated: number,      // Count of memories created/updated
  totalObservations: number, // Total observations processed
  reason?: string            // Optional explanation
}

mem::consolidate-pipeline accepts a tier parameter (selecting specific stages) and a force boolean to bypass configuration checks. It returns a comprehensive results object:

{
  success: true,
  results: {
    semantic?: object,      // Semantic merge results
    reflect?: object,       // Reflection output
    procedural?: object,    // Extracted procedures
    decay?: object,          // Applied decay statistics
    obsidianExport?: object  // Export status
  }
}

Configuration and Execution Control

mem::consolidate executes immediately upon invocation without external configuration checks, making it suitable for on-demand synthesis after coding sessions.

mem::consolidate-pipeline respects the CONSOLIDATION_ENABLED environment flag. Unless force: true is passed in the payload, the pipeline skips execution when consolidation is disabled. This safety mechanism prevents accidental processing in environments where background maintenance is not desired.

Decay calculations use the CONSOLIDATION_DECAY_DAYS configuration value to determine the exponential decay rate applied to memory strength values during the pipeline's decay stage.

Practical Usage Examples

Trigger immediate observation consolidation after a development session:

// SDK invocation for single-pass consolidation
await sdk.trigger({
  function_id: "mem::consolidate",
  payload: { project: "my-app", minObservations: 15 }
});
// Returns: { consolidated: 3, totalObservations: 42 }

Run the full maintenance pipeline across all memory tiers:

// SDK invocation for comprehensive pipeline
await sdk.trigger({
  function_id: "mem::consolidate-pipeline",
  payload: { tier: "all", force: true }
});
// Returns detailed results for each processing stage

Equivalent REST endpoint calls using curl:

# Immediate consolidation

curl -X POST https://<host>/agentmemory/consolidate \
     -H "Content-Type: application/json" \
     -d '{"project":"my-app","minObservations":10}'

# Semantic tier only

curl -X POST https://<host>/agentmemory/consolidate-pipeline \
     -H "Content-Type: application/json" \
     -d '{"tier":"semantic"}'

Summary

• mem::consolidate performs immediate, single-purpose conversion of observations to semantic memories in src/functions/consolidate.ts, auditing each mutation individually.
• mem::consolidate-pipeline runs a configurable, multi-stage maintenance workflow in src/functions/consolidation-pipeline.ts, handling semantic merges, procedural extraction, decay, and external exports under a single audit entry.
• Execution control differs: consolidate runs on-demand unconditionally, while the pipeline checks CONSOLIDATION_ENABLED unless forced.
• Return structures vary from simple counts (consolidate) to detailed stage results (pipeline).

Frequently Asked Questions

When should I use mem::consolidate versus mem::consolidate-pipeline?

Use mem::consolidate when you need immediate synthesis of recent coding-session observations into long-term memories, such as right after completing a feature. Use mem::consolidate-pipeline for scheduled maintenance runs that refresh, merge, and prune the entire knowledge base across semantic and procedural tiers.

Can I run specific stages of the consolidation pipeline independently?

Yes. The mem::consolidate-pipeline accepts a tier parameter that lets you execute specific phases such as "semantic" for fact merging only, or "all" to run the complete sequence including reflection, procedural extraction, and decay management.

Why does the pipeline check CONSOLIDATION_ENABLED while consolidate does not?

The mem::consolidate-pipeline applies broad transformations across multiple memory stores and may trigger expensive LLM calls for reflection and procedural extraction. The configuration flag prevents accidental execution in production environments. mem::consolidate is typically triggered explicitly by user action after sessions, so it assumes immediate intent.

How do I force the pipeline to run when CONSOLIDATION_ENABLED is false?

Pass force: true in the payload when triggering mem::consolidate-pipeline. This boolean overrides the isConsolidationEnabled() check in src/config.ts and allows execution regardless of the environment configuration setting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}