# Phases of Wiki Incremental Ingestion: How TencentDB Agent Memory Processes Documents

> Explore the three phases of Wiki incremental ingestion: extracting, merging, and indexing. Learn how TencentDB Agent Memory processes documents with real-time visibility.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-08-27

---

**The Wiki incremental ingestion pipeline operates in three sequential phases—extracting, merging, and indexing—that are tracked via the `IngestProgress` interface to provide real-time visibility into document processing.**

The TencentCloud/TencentDB-Agent-Memory repository implements a robust incremental ingestion system designed to continuously enrich Wiki repositories with new knowledge while preserving existing content. This system processes source documents through a carefully orchestrated pipeline defined in [`MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts), ensuring atomic updates to both the filesystem and SQLite-backed search indexes.

## The Three Phases of Wiki Incremental Ingestion

The ingestion engine divides document processing into three logical phases exposed through the `IngestProgress` payload. Each phase serves a distinct purpose in transforming raw source material into queryable knowledge.

### Phase 1: Extracting (Parallel LLM Generation)

During the **extracting** phase, the engine performs parallel LLM calls to generate candidate Wiki pages from source material. This stage operates entirely in memory without persisting to disk, filtering out content that already exists in the current Wiki. According to the source code in [`ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/ingest-v2/index.ts), this corresponds to stage 1, marked by the comment *“阶段1：对单个源文件调 LLM 生成候选 wiki 页（纯内存，不落盘）”*. The `WikiSourceManager` class defined in [`MemoryKnowledge/src/engines/wiki/manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/manager.ts) orchestrates these calls, ensuring only novel content proceeds to the next phase.

### Phase 2: Merging (Serial Persistence)

The **merging** phase writes the generated pages to disk sequentially, handling conflict resolution, front-matter parsing, and page-level error tracking. Implemented as stage 2 in [`ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/ingest-v2/index.ts) (comment: *“阶段2：串行落盘 + 收尾”*), this phase uses the logic in [`MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts) to persist pages to the filesystem. Unlike the parallel extraction stage, merging operates serially to prevent write conflicts and ensure data consistency when updating the Wiki repository.

### Phase 3: Indexing (Atomic Search Rebuild)

In the final **indexing** phase, the system rebuilds the full-text search indexes (BM25/FTS5), regenerates graph edges, and updates metadata tables including `page_meta` and `graph_edge`. This occurs within a single SQLite transaction managed by [`MemoryKnowledge/src/engines/wiki/index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts), ensuring that reads are never blocked during index construction. As noted in the source comments (*“阶段3：overview（FTS 索引由上层 ingest 写事务完成）”*), the FTS index completion happens atomically at the end of the ingest transaction, guaranteeing that search queries reflect the complete new state only after all pages are persisted.

## Tracking Ingestion Progress with IngestProgress

The pipeline exposes granular progress information through the `IngestProgress` interface defined in [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts):

```typescript
export interface IngestProgress {
  phase: "extracting" | "merging" | "indexing";
  total: number;
  completed: number;
  failed: number;
  skipped: number;
  percent: number;
}

```

The `ProgressFn` callback reports these metrics to callers, with phase switches emitted immediately to signal state transitions. To prevent UI flooding during high-volume ingests, the `createThrottledProgressFn` utility (also in [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts)) throttles progress updates while preserving immediate phase change notifications.

## Implementation Details and Source Files

The incremental ingestion pipeline relies on several key components:

- **[`MemoryKnowledge/src/engines/wiki/manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/manager.ts)** – Defines the `IngestProgress` interface, the throttling logic via `createThrottledProgressFn`, and the public `ingest` method that orchestrates the three phases.
- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/index.ts)** – Contains the three-stage implementation (extract → merge → overview) and coordinates LLM calls with disk writes.
- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/prompts.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/prompts.ts)** – Builds the system and user prompts used during the extracting phase, supporting both single-stage and two-stage generation strategies.
- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/merge.ts)** – Implements the serial merge logic that writes pages to the filesystem during the merging phase.
- **[`MemoryKnowledge/src/engines/wiki/index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts)** – Creates the SQLite `index.db`, defines schema tables (`source`, `page_meta`, `graph_edge`), and executes the atomic transaction that finalizes the indexing phase.

## Practical Code Examples

### Running a Full Ingest with Progress Callbacks

Monitor all three phases in real-time by providing an `onProgress` handler to the `WikiSourceManager.ingest` method:

```typescript
import { WikiSourceManager } from "./MemoryKnowledge/src/engines/wiki/manager.js";

const manager: WikiSourceManager = /* obtain manager instance */;
await manager.ingest(
  "my-wiki",
  { apiKey: "…", model: "gpt‑4o-mini" },
  {
    onProgress: (p) => {
      console.log(`[${p.phase}] ${p.percent.toFixed(1)}% – ${p.completed}/${p.total}`);
    },
  }
);

```

### Listening for Phase Changes Only

Use `createThrottledProgressFn` with a 0ms throttle to receive updates only when the phase transitions, reducing overhead while tracking pipeline state:

```typescript
import { createThrottledProgressFn } from "./MemoryKnowledge/src/engines/wiki/manager.js";

const throttled = createThrottledProgressFn((p) => {
  if (p.phase === "merging") {
    console.log("🛠️ Merging generated pages into the Wiki…");
  }
}, 0); // 0ms ⇒ emit every phase change

await manager.ingest("my-wiki", llmConfig, { onProgress: throttled });

```

### Querying the Wiki After Ingestion

Once the indexing phase completes, the BM25 and graph indexes are immediately available for search operations:

```typescript
const results = manager.search("my-wiki", "authentication token", 10, { hop: 2 });
results.forEach(r => console.log(`${r.title}: ${r.snippet}`));

```

## Summary

- The Wiki incremental ingestion pipeline processes documents through three distinct **phases**: extracting (parallel LLM generation in memory), merging (serial disk writes with conflict resolution), and indexing (atomic SQLite transaction rebuilding BM25 and graph structures).
- Progress tracking is standardized via the **`IngestProgress`** interface, which reports phase names, completion counts, and percentages.
- The implementation spans multiple files in `MemoryKnowledge/src/engines/wiki/`, with core orchestration in [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts) and phase-specific logic in [`ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/ingest-v2/index.ts), [`merge.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/merge.ts), and [`index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/index-db.ts).
- The use of **SQLite transactions** during indexing ensures that search queries always return consistent results, even during active ingestion.

## Frequently Asked Questions

### What happens during the extracting phase of Wiki incremental ingestion?

During the extracting phase, the system makes parallel LLM calls to generate candidate Wiki pages from source documents. This process runs entirely in memory without writing to disk, and the engine filters out content that already exists in the current Wiki repository. As implemented in [`ingest-v2/index.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/ingest-v2/index.ts), this phase corresponds to stage 1 (*“阶段1：对单个源文件调 LLM 生成候选 wiki 页”*).

### How does the merging phase handle conflicts and errors?

The merging phase writes pages to the filesystem serially to prevent write conflicts, employing the logic in [`ingest-v2/merge.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/ingest-v2/merge.ts) to handle front-matter parsing and track page-level errors. Because this phase operates sequentially rather than in parallel, it ensures data integrity when updating existing Wiki content and manages error isolation per page.

### Why is the indexing phase wrapped in a single SQLite transaction?

The indexing phase rebuilds the BM25 full-text index, regenerates `graph_edge` relationships, and updates `page_meta` tables inside a single SQLite transaction managed by [`index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/index-db.ts). This atomic approach ensures that read operations are never blocked during index construction and that search queries see a consistent snapshot of the Wiki only after all updates are complete.

### How can I monitor the specific phase of an ongoing ingestion?

You can monitor the current phase by implementing the `ProgressFn` callback and examining the `phase` property of the `IngestProgress` object, which cycles through `"extracting"`, `"merging"`, and `"indexing"`. For performance-sensitive applications, wrap your callback with `createThrottledProgressFn` from [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts) to filter updates while still receiving immediate notifications when the phase changes.