Pipeline Configuration for L1 Memory Extraction in TencentDB Agent Memory

The L1 memory extraction pipeline in TencentDB Agent Memory is configured through the typed ExtractionConfig and PipelineTriggerConfig sections in MemoryCore/src/config.ts, controlling deduplication, model selection, and trigger intervals for converting raw L0 conversations into structured memories.

TencentDB Agent Memory implements a hierarchical memory system that processes raw conversation logs (L0) into extracted memories (L1), scene blocks (L2), and personas (L3). The configuration driving this pipeline is defined in MemoryCore/src/config.ts and parsed at runtime by the plugin using MemoryCore/src/utils/pipeline-manager.ts. Understanding these configuration options allows developers to tune extraction frequency, deduplication strategies, and model selection for their specific use cases.

Core Configuration Sections for L1 Extraction

The L1 memory extraction stage relies on two primary configuration objects that control both the extraction behavior and the scheduling mechanics.

ExtractionConfig Settings

The ExtractionConfig section in MemoryCore/src/config.ts defines how raw L0 logs are processed into structured L1 memory entries. Key fields include:

• enabled – Boolean toggle to activate L1 extraction (default: true)
• enableDedup – Activates deduplication to avoid storing duplicate memories (default: true)
• maxMemoriesPerSession – Caps the number of memories extracted per conversation session (default: 20)
• model – Specifies the LLM model identifier for extraction tasks (e.g., "openai/gpt-4o-mini")
• promptMode – Determines prompt style, either "chat" or "code" (default: "chat")

When enableDedup is active, the pipeline uses either dense vector embeddings or BM25 sparse vectors to identify duplicate memories before storage.

PipelineTriggerConfig Orchestration

The PipelineTriggerConfig section controls when the L1 extraction stage executes within the broader L0→L1→L2→L3 pipeline:

• everyNConversations – Triggers L1 extraction after every N conversation rounds (default: 5)
• l1IdleTimeoutSeconds – Forces L1 extraction after idle time expires (default: 30)
• enableWarmup – Allows pipeline warmup scheduling on startup (default: true)
• sessionActiveWindowHours – Excludes stale sessions from L2 polling after inactivity threshold

L1 extraction runs immediately when either the conversation count threshold or the idle timeout condition is met.

Pipeline Flow and Deduplication Strategy

The complete pipeline flow is orchestrated by the trigger configuration, with L1 serving as the critical transformation layer between raw logs and structured knowledge.

L1 Extraction Trigger Mechanics

According to the source implementation in MemoryCore/src/utils/pipeline-manager.ts, the L1 stage executes based on the following logic:

1. Conversation-based triggering – After accumulating everyNConversations rounds, the pipeline initiates L1 extraction
2. Idle-based triggering – If no new conversations arrive within l1IdleTimeoutSeconds, extraction runs automatically
3. L2 cascading – Once L1 completes, the system waits l2DelayAfterL1Seconds (default: 90) before initiating L2 scene generation

Sessions idle longer than sessionActiveWindowHours are excluded from subsequent L2 polling regardless of L1 completion status.

Deduplication and Embedding Configuration

The L1 deduplication step relies on the EmbeddingConfig section to determine similarity calculation methods:

• Dense embedding – When EmbeddingConfig.provider is set to "openai" or another provider, the pipeline uses vector similarity to detect duplicates
• BM25 fallback – When EmbeddingConfig.provider is "none" (default), the pipeline falls back to BM25Config for sparse vector scoring (default language: "zh")

The BM25Config section controls the sparse encoder with fields for enabled status and language settings, ensuring deduplication works even without external embedding services.

Configuration Examples

Minimal Zero-Config Setup

An empty JSON configuration relies on sensible defaults for immediate L1 extraction functionality:

{}

This configuration enables capture, L1 extraction with deduplication, automatic warmup scheduling, and BM25-only embedding (no external API required).

Production Configuration with Dense Embeddings

For production deployments requiring high-quality deduplication and cloud-based vector storage:

{
  "capture": {
    "enabled": true,
    "excludeAgents": ["bench-judge-*"],
    "l0l1RetentionDays": 30,
    "allowAggressiveCleanup": false
  },
  "extraction": {
    "enabled": true,
    "enableDedup": true,
    "maxMemoriesPerSession": 30,
    "model": "openai/gpt-4o-mini",
    "promptMode": "chat"
  },
  "pipelineTrigger": {
    "everyNConversations": 4,
    "enableWarmup": true,
    "l1IdleTimeoutSeconds": 20,
    "l2DelayAfterL1Seconds": 60,
    "l2MinIntervalSeconds": 600,
    "l2MaxIntervalSeconds": 1800,
    "sessionActiveWindowHours": 12
  },
  "embedding": {
    "enabled": true,
    "provider": "openai",
    "baseUrl": "https://api.openai.com/v1",
    "apiKey": "<YOUR_API_KEY>",
    "model": "text-embedding-3-large",
    "dimensions": 1536,
    "sendDimensions": true,
    "conflictRecallTopK": 7,
    "maxInputChars": 4000,
    "timeoutMs": 12000
  },
  "bm25": {
    "enabled": true,
    "language": "zh"
  }
}

This configuration enables aggressive L1 extraction (every 4 conversations) with OpenAI embeddings for semantic deduplication and hybrid recall.

Implementation Architecture

The configuration system uses strict TypeScript definitions in MemoryCore/src/config.ts (lines 14-180) to ensure type safety across the pipeline.

Configuration Parsing

The parseConfig utility in MemoryCore/src/utils/pipeline-manager.ts validates and transforms the raw JSON into runtime configuration objects. Concrete pipeline stages are instantiated by MemoryCore/src/utils/pipeline-factory.ts, which reads the config to create L1 extractors, L2 scene generators, and L3 persona synthesizers.

Prompt Handling

The promptMode field in ExtractionConfig maps to specific prompt templates implemented in MemoryCore/src/gateway/memory-prompt-handlers.ts, determining whether the extraction LLM receives chat-formatted or code-formatted instructions when processing L0 logs into L1 memories.

Summary

• The L1 memory extraction pipeline is controlled by ExtractionConfig and PipelineTriggerConfig in MemoryCore/src/config.ts
• Trigger conditions include conversation count (everyNConversations) and idle timeout (l1IdleTimeoutSeconds)
• Deduplication uses either dense embeddings (via EmbeddingConfig) or BM25 sparse vectors (via BM25Config) when embeddings are disabled
• The standalone LLM configuration allows dedicated model endpoints for memory extraction separate from the host runner
• Default settings enable immediate operation with BM25-only deduplication and conservative memory limits (20 memories per session)

Frequently Asked Questions

Where is the L1 memory extraction configuration defined?

The configuration is defined in MemoryCore/src/config.ts, which exports TypeScript interfaces and default values for ExtractionConfig, PipelineTriggerConfig, and related sections. The runtime parsing logic resides in MemoryCore/src/utils/pipeline-manager.ts.

How does the L1 extraction trigger mechanism work?

L1 extraction triggers on two conditions: after every everyNConversations conversation rounds (default: 5), or after l1IdleTimeoutSeconds seconds of inactivity (default: 30). The system evaluates these conditions continuously in the background pipeline manager.

What is the difference between BM25 and dense embedding deduplication?

BM25 uses sparse vector scoring based on term frequency and is controlled by BM25Config (enabled by default when EmbeddingConfig.provider is "none"). Dense embedding uses vector similarity through external providers like OpenAI and requires EmbeddingConfig.enabled set to true with valid API credentials. Dense embeddings generally provide better semantic matching but require external API access.

How do I configure a separate LLM for memory extraction?

Use the standaloneLlm configuration section to specify a dedicated endpoint for L1 extraction tasks. When defined with model, apiKey, and baseUrl fields, the pipeline bypasses the host runner and uses this isolated LLM connection for all memory extraction operations, preventing interference with primary conversational AI tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}