Phases of Wiki Incremental Ingestion: How TencentDB Agent Memory Processes Documents

The Wiki incremental ingestion pipeline operates in three sequential phases—extracting, merging, and indexing—that are tracked via the IngestProgress interface to provide real-time visibility into document processing.

The TencentCloud/TencentDB-Agent-Memory repository implements a robust incremental ingestion system designed to continuously enrich Wiki repositories with new knowledge while preserving existing content. This system processes source documents through a carefully orchestrated pipeline defined in MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts, ensuring atomic updates to both the filesystem and SQLite-backed search indexes.

The Three Phases of Wiki Incremental Ingestion

The ingestion engine divides document processing into three logical phases exposed through the IngestProgress payload. Each phase serves a distinct purpose in transforming raw source material into queryable knowledge.

Phase 1: Extracting (Parallel LLM Generation)

During the extracting phase, the engine performs parallel LLM calls to generate candidate Wiki pages from source material. This stage operates entirely in memory without persisting to disk, filtering out content that already exists in the current Wiki. According to the source code in ingest-v2/index.ts, this corresponds to stage 1, marked by the comment “阶段1：对单个源文件调 LLM 生成候选 wiki 页（纯内存，不落盘）”. The WikiSourceManager class defined in MemoryKnowledge/src/engines/wiki/manager.ts orchestrates these calls, ensuring only novel content proceeds to the next phase.

Phase 2: Merging (Serial Persistence)

The merging phase writes the generated pages to disk sequentially, handling conflict resolution, front-matter parsing, and page-level error tracking. Implemented as stage 2 in ingest-v2/index.ts (comment: “阶段2：串行落盘 + 收尾”), this phase uses the logic in MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts to persist pages to the filesystem. Unlike the parallel extraction stage, merging operates serially to prevent write conflicts and ensure data consistency when updating the Wiki repository.

Phase 3: Indexing (Atomic Search Rebuild)

In the final indexing phase, the system rebuilds the full-text search indexes (BM25/FTS5), regenerates graph edges, and updates metadata tables including page_meta and graph_edge. This occurs within a single SQLite transaction managed by MemoryKnowledge/src/engines/wiki/index-db.ts, ensuring that reads are never blocked during index construction. As noted in the source comments (“阶段3：overview（FTS 索引由上层 ingest 写事务完成）”), the FTS index completion happens atomically at the end of the ingest transaction, guaranteeing that search queries reflect the complete new state only after all pages are persisted.

Tracking Ingestion Progress with IngestProgress

The pipeline exposes granular progress information through the IngestProgress interface defined in manager.ts:

export interface IngestProgress {
  phase: "extracting" | "merging" | "indexing";
  total: number;
  completed: number;
  failed: number;
  skipped: number;
  percent: number;
}

The ProgressFn callback reports these metrics to callers, with phase switches emitted immediately to signal state transitions. To prevent UI flooding during high-volume ingests, the createThrottledProgressFn utility (also in manager.ts) throttles progress updates while preserving immediate phase change notifications.

Implementation Details and Source Files

The incremental ingestion pipeline relies on several key components:

• MemoryKnowledge/src/engines/wiki/manager.ts – Defines the IngestProgress interface, the throttling logic via createThrottledProgressFn, and the public ingest method that orchestrates the three phases.
• MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts – Contains the three-stage implementation (extract → merge → overview) and coordinates LLM calls with disk writes.
• MemoryKnowledge/src/engines/wiki/ingest-v2/prompts.ts – Builds the system and user prompts used during the extracting phase, supporting both single-stage and two-stage generation strategies.
• MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts – Implements the serial merge logic that writes pages to the filesystem during the merging phase.
• MemoryKnowledge/src/engines/wiki/index-db.ts – Creates the SQLite index.db, defines schema tables (source, page_meta, graph_edge), and executes the atomic transaction that finalizes the indexing phase.

Practical Code Examples

Running a Full Ingest with Progress Callbacks

Monitor all three phases in real-time by providing an onProgress handler to the WikiSourceManager.ingest method:

import { WikiSourceManager } from "./MemoryKnowledge/src/engines/wiki/manager.js";

const manager: WikiSourceManager = /* obtain manager instance */;
await manager.ingest(
  "my-wiki",
  { apiKey: "…", model: "gpt‑4o-mini" },
  {
    onProgress: (p) => {
      console.log(`[${p.phase}] ${p.percent.toFixed(1)}% – ${p.completed}/${p.total}`);
    },
  }
);

Listening for Phase Changes Only

Use createThrottledProgressFn with a 0ms throttle to receive updates only when the phase transitions, reducing overhead while tracking pipeline state:

import { createThrottledProgressFn } from "./MemoryKnowledge/src/engines/wiki/manager.js";

const throttled = createThrottledProgressFn((p) => {
  if (p.phase === "merging") {
    console.log("🛠️ Merging generated pages into the Wiki…");
  }
}, 0); // 0ms ⇒ emit every phase change

await manager.ingest("my-wiki", llmConfig, { onProgress: throttled });

Querying the Wiki After Ingestion

Once the indexing phase completes, the BM25 and graph indexes are immediately available for search operations:

const results = manager.search("my-wiki", "authentication token", 10, { hop: 2 });
results.forEach(r => console.log(`${r.title}: ${r.snippet}`));

Summary

• The Wiki incremental ingestion pipeline processes documents through three distinct phases: extracting (parallel LLM generation in memory), merging (serial disk writes with conflict resolution), and indexing (atomic SQLite transaction rebuilding BM25 and graph structures).
• Progress tracking is standardized via the IngestProgress interface, which reports phase names, completion counts, and percentages.
• The implementation spans multiple files in MemoryKnowledge/src/engines/wiki/, with core orchestration in manager.ts and phase-specific logic in ingest-v2/index.ts, merge.ts, and index-db.ts.
• The use of SQLite transactions during indexing ensures that search queries always return consistent results, even during active ingestion.

Frequently Asked Questions

What happens during the extracting phase of Wiki incremental ingestion?

During the extracting phase, the system makes parallel LLM calls to generate candidate Wiki pages from source documents. This process runs entirely in memory without writing to disk, and the engine filters out content that already exists in the current Wiki repository. As implemented in ingest-v2/index.ts, this phase corresponds to stage 1 (“阶段1：对单个源文件调 LLM 生成候选 wiki 页”).

How does the merging phase handle conflicts and errors?

The merging phase writes pages to the filesystem serially to prevent write conflicts, employing the logic in ingest-v2/merge.ts to handle front-matter parsing and track page-level errors. Because this phase operates sequentially rather than in parallel, it ensures data integrity when updating existing Wiki content and manages error isolation per page.

Why is the indexing phase wrapped in a single SQLite transaction?

The indexing phase rebuilds the BM25 full-text index, regenerates graph_edge relationships, and updates page_meta tables inside a single SQLite transaction managed by index-db.ts. This atomic approach ensures that read operations are never blocked during index construction and that search queries see a consistent snapshot of the Wiki only after all updates are complete.

How can I monitor the specific phase of an ongoing ingestion?

You can monitor the current phase by implementing the ProgressFn callback and examining the phase property of the IngestProgress object, which cycles through "extracting", "merging", and "indexing". For performance-sensitive applications, wrap your callback with createThrottledProgressFn from manager.ts to filter updates while still receiving immediate notifications when the phase changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}