How the TDAI Core Pipeline Is Initialized and Configured: A Deep Dive into TencentDB-Agent-Memory

The TDAI Core initializes its multi-stage memory pipeline (L0 → L1 → L2 → L3) through a sequential four-step process in initialize(): directory setup, store creation, pipeline manager construction, and LLM runner wiring, with all timing thresholds defined in MemoryTdaiConfig.

The TencentDB-Agent-Memory repository implements a sophisticated memory extraction system for database AI agents. Understanding TDAI Core pipeline initialization and configuration is essential for developers who need to customize memory extraction behavior or debug pipeline execution flows. The core constructs its pipeline through a carefully orchestrated sequence of asynchronous operations during the bootstrap phase.

Core Construction and Dependency Injection

The pipeline lifecycle begins when the TdaiCore class is instantiated. The constructor in src/core/tdai-core.ts (lines 24-34) accepts a TdaiCoreOptions object and stores references to all external dependencies required for pipeline operation.

constructor(opts: TdaiCoreOptions) {
  this.hostAdapter = opts.hostAdapter;
  this.cfg = opts.config;
  this.logger = opts.hostAdapter.getLogger();
  this.dataDir = opts.hostAdapter.getRuntimeContext().dataDir;
  // ... additional dependency storage
}

The constructor captures the host adapter for runtime context, the parsed configuration object, the logger, and the data directory path. It also stores optional references to a session filter, storage adapter, and skill hooks that will be injected into the pipeline later. At this stage, no I/O operations occur—the constructor merely prepares the internal state for the subsequent initialization sequence.

The Initialization Sequence

The initialize() method in src/core/tdai-core.ts orchestrates the actual construction of the memory pipeline through three sequential actions. This method must be awaited before the core can process any conversation data.

Data Directory Setup

First, initDataDirectories(this.dataDir) creates the filesystem layout required for checkpoint persistence and COS (Cloud Object Storage) file operations (line 44). This ensures that the directory structure exists before any persistence operations attempt to write state to disk.

Store Initialization

Next, initStores() asynchronously creates the vector store and optional embedding service based on the configuration (lines 48-53). These storage backends serve as the foundation for L0 (raw message) buffering and semantic retrieval operations during later pipeline stages.

Pipeline Manager Construction

If memory extraction is enabled via cfg.extraction.enabled, the core constructs a MemoryPipelineManager through the factory function createPipelineManager:

if (this.cfg.extraction.enabled) {
  this.scheduler = createPipelineManager(this.cfg, this.logger, this.sessionFilter);
  // ...
}

Located in src/utils/pipeline-factory.ts, this factory assembles the pipeline manager with configuration thresholds, timers, and warm-up mode settings derived from MemoryTdaiConfig. The factory also prepares the L1, L2, and L3 runner slots and a persister that will handle checkpoint serialization.

Wiring the LLM Runners

After store initialization completes, wirePipelineRunners() determines whether to use the host-provided LLM runner or a stand-alone LLM runner based on cfg.llm.enabled or a provider=proxy setting (lines 69-75). This method creates and registers four critical components:

• L1 runner (createL1Runner) – Ingests buffered L0 messages and performs initial extraction batches.
• L2 runner (createL2Runner) – Extracts scene information from processed conversations.
• L3 runner (createL3Runner) – Generates personas based on accumulated context.
• Persister (createPersister) – Stores checkpoint data after each pipeline state change.

The wiring operation executes once during standard initialization, though it can be re-run after storage adapter injection if dynamic reconfiguration is required. All wiring operations are logged for debugging purposes.

Scheduler Activation and Checkpoint Recovery

The pipeline remains dormant until the first conversation turn is committed. When handleTurnCommitted() is invoked, ensureSchedulerStarted() performs lazy initialization:

1. The method reads existing checkpoint data via CheckpointManager from src/utils/checkpoint.ts.
2. It calls scheduler.start() to restore pending session states and re-arm timers.
3. Subsequent calls reuse the same in-flight promise, guaranteeing a single start sequence even under concurrency.

This lazy-start pattern ensures that pipeline resources are not consumed until actual memory extraction work is required, while still maintaining idempotency for the bootstrap process.

Configuration Schema and Timing Parameters

All pipeline behavior is governed by MemoryTdaiConfig, populated from MemoryCore/tdai-gateway.yaml. The configuration defines extraction enablement, model providers, and precise timing controls:

• everyNConversations – Triggers immediate L1 batch processing after N conversations.
• l1.idleTimeoutSeconds – Idle debounce duration before L1 runs on buffered messages.
• l2.delayAfterL1Seconds – Delay between L1 completion and L2 scene extraction.
• l2.minIntervalSeconds / l2.maxIntervalSeconds – bounds L2 scheduling frequency.
• enableWarmup – Activates exponential warm-up thresholds for new sessions.

The stand-alone LLM runner in src/adapters/standalone/llm-runner.ts uses these timing parameters when the core operates in proxy mode, decoupled from the host's LLM infrastructure.

Summary

• Four-stage bootstrap: The TDAI Core initializes through directory setup, store creation, pipeline manager construction, and LLM runner wiring.
• Factory pattern: src/utils/pipeline-factory.ts decouples pipeline assembly from configuration parsing.
• Lazy activation: The scheduler starts only when the first turn is committed, restoring state from src/utils/checkpoint.ts.
• YAML-driven: All timing thresholds and extraction settings originate from MemoryCore/tdai-gateway.yaml and populate MemoryTdaiConfig.
• Modular runners: L1, L2, and L3 runners are injected via wirePipelineRunners(), supporting both host-provided and stand-alone LLM adapters.

Frequently Asked Questions

What triggers the TDAI Core pipeline to start processing?

The pipeline scheduler starts lazily when handleTurnCommitted() is first called, invoking ensureSchedulerStarted(). This method reads the checkpoint file and calls scheduler.start() only once, even if multiple requests arrive simultaneously, by caching the start promise.

How does the pipeline handle restarts and state persistence?

The CheckpointManager class in src/utils/checkpoint.ts serializes per-session pipeline states to disk after each significant operation. When the core restarts, ensureSchedulerStarted() reads this checkpoint data and restores pending timers and warm-up thresholds, ensuring no conversation context is lost between process lifecycles.

Can the LLM runner be replaced with a custom implementation?

Yes. The wirePipelineRunners() method checks cfg.llm.enabled and provider=proxy to determine whether to use the host adapter's runner or instantiate the stand-alone runner from src/adapters/standalone/llm-runner.ts. Developers can inject custom runners by modifying the factory logic in src/utils/pipeline-factory.ts or by providing a compatible host adapter interface.

Where are the pipeline timing thresholds configured?

All timing parameters—including idleTimeoutSeconds, delayAfterL1Seconds, and everyNConversations—are defined in MemoryCore/tdai-gateway.yaml. These values populate the MemoryTdaiConfig interface, which is passed through the constructor to createPipelineManager and ultimately controls the timer logic in src/utils/pipeline-manager.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}