# How the Wiki Engine in Memory Knowledge Service Ingests and Processes Documents

> Learn how the Wiki Engine in Memory Knowledge Service ingests and processes documents. Discover the three-stage pipeline that transforms markdown into searchable Wiki pages.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-09-04

---

**The Wiki Engine in the Memory Knowledge Service transforms raw markdown sources into searchable, interlinked Wiki pages through a three-stage pipeline—ingestion and LLM extraction, merging with overview generation, and SQLite index rebuilding—orchestrated primarily through `WikiSourceManager.ingest()` in [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts).**

The **Memory Knowledge Service** is a core component of the TencentCloud/TencentDB-Agent-Memory repository, responsible for converting unstructured documentation into queryable knowledge graphs. The Wiki Engine’s ingestion pipeline handles everything from SHA-256-based change detection to LLM-powered content extraction, ultimately building a full-text search index and directed graph structure for semantic retrieval.

## High-Level Ingestion Architecture

The ingestion workflow centers on the `WikiSourceManager` class defined in [`MemoryKnowledge/src/engines/wiki/manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/manager.ts). When an ingest request arrives via the API endpoint `POST /v3/wiki/ingest`, the system executes a coordinated sequence of operations across three logical stages.

### Initialization and Source Registration

Before processing begins, the engine initializes a dedicated project directory through `initWikiProject` (lines 719–726). This creates the necessary folder structure under the configured storage path and registers the Wiki in the manager’s in-memory map. The `init()` sequence establishes the `raw/sources` directory where incoming documents are staged prior to processing.

### The Three-Phase Pipeline

The core logic resides in `runIngestIncremental` (lines 611–791), which implements the incremental ingestion worker. This method classifies files, extracts content using LLM inference, merges results into Wiki pages, and rebuilds the search index. The pipeline distinguishes between *ingest* (new/changed files), *skip* (unchanged), and *delete* (removed) operations by comparing current SHA-256 hashes against the stored state in the SQLite `source` table.

## Stage 1: Extraction and Classification

The first stage focuses on identifying what needs processing and transforming raw text into structured page candidates using LLM inference.

### SHA-256-Based Source Classification

The `classifySources` function (invoked at lines 87–92) performs efficient change detection by comparing file hashes. For each document in `raw/sources`, the system calculates a SHA-256 checksum and queries the existing state via `readSourceStates` (lines 886–889). Files with matching hashes are skipped entirely, while new or modified files enter the extraction queue. Missing files trigger cleanup operations to maintain index integrity.

### LLM-Powered Content Extraction

For each source marked for ingestion, the engine executes `extractSource` (dynamically imported from [`engines/wiki/ingest-v2/index.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/engines/wiki/ingest-v2/index.js)). This function runs a parallel LLM chain that converts raw markdown into a `Map<string, string>` of candidate pages. The extraction process respects concurrency limits enforced by `getIngestConcurrency` (lines 1000–1002) and a global LLM semaphore (`globalLlmLimit`) to prevent rate-limiting issues.

Prior to LLM processing, the engine parses **front-matter metadata** via `extractFrontmatter` (lines 52–84) and identifies intra-Wiki links through `extractWikilinks` (lines 87–94). These extracted relationships populate the `WikiPage` type defined in [`types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/types.ts) and serve as the foundation for the graph construction phase.

### Progress Throttling and Concurrency

To prevent UI flooding during long-running extractions, the system implements `createThrottledProgressFn` (lines 27–46). This utility emits progress updates only when phases change, when completion percentage advances, or when extraction reaches 90% completion. The throttle interval defaults to `PROGRESS_THROTTLE_MS` (500ms), ensuring sustainable update frequencies for large document sets.

## Stage 2: Merging and Overview Generation

Once extraction completes, the pipeline transitions to merging candidates and generating high-level summaries.

### Committing Candidate Pages

The `commitCandidates` function (lines 711–728) merges extracted content into the Wiki directory structure, writing finalized markdown files to disk. This operation handles conflicts and collects errors in a `mergeErrors` array (lines 117–121) without aborting the entire ingest, allowing partial successes to proceed to indexing.

### Automatic Overview Creation

Following successful merges, the system optionally invokes `generateOverview` (lines 552–558) to create a summary page. This step requires a valid LLM client configuration; if unavailable, the pipeline skips overview generation but continues with index rebuilding. The overview synthesizes key themes from newly ingested content, providing navigational entry points for users.

## Stage 3: Index Rebuilding and Search

The final stage persists the processed content into queryable data structures and refreshes the in-memory read models.

### SQLite FTS5 and Graph Tables

The `writeIndex` function (lines 94–104) executes a transactional rebuild of three critical SQLite tables:
- **`wiki_fts`**: Full-text search index using FTS5
- **`page_meta`**: Page metadata with static snippets
- **`graph_edge`**: Directed wikilink relationships

Edge resolution occurs via `resolveEdges` (lines 200–224), which maps slugified page titles to internal IDs. This directed graph enables multi-hop traversal for semantic search expansion.

### Graph Construction and Multi-Hop Search

After index reconstruction, `loadReadModel` (lines 15–35) builds the in-memory `PageGraph` structure. The search API leverages this graph through `ftsSearch` (lines 81–90), which tokenizes queries using `tokenize` (lines 40–74) before executing SQLite FTS5 queries. For expanded results, the engine employs `graphMultiHopSearch` defined in [`graph-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/graph-search.ts) to perform BFS traversal across linked pages.

## Practical Implementation Examples

### Registering and Ingesting a Wiki Programmatically

```typescript
import { createWikiSourceManager } from "./src/engines/wiki/manager.js";

// Initialize manager with storage path
const wikiMgr = createWikiSourceManager("/var/wikis");

// Register new Wiki project
await wikiMgr.register({
  name: "product-wiki",
  path: "/var/wikis/product-wiki",
});

// Execute ingestion with LLM configuration
await wikiMgr.ingest("product-wiki", {
  provider: "openai",
  model: "gpt-4o-mini",
  apiKey: process.env.OPENAI_API_KEY,
});

```

*Key references*: `createWikiSourceManager` (lines 795–796), `register` (lines 42–46), `ingest` (lines 779–846).

### API Endpoint Usage

```http
POST /v3/wiki/ingest HTTP/1.1
Content-Type: application/json
Authorization: Bearer <token>

{
  "knowledge_id": "wiki-1",
  "service_url": "http://ks:8421/v3",
  "team_id": "team-1",
  "llm_config": {
    "provider": "openai",
    "model": "gpt-4o",
    "api_key": "***"
  }
}

```

The server handler in [`MemoryKnowledge/src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/wiki-service.ts) (lines 70–88) routes this request to `wikiService.ingest()`.

### Reading Processed Content

```typescript
const md = await wikiMgr.readPage(
  "product-wiki", 
  "wiki/concepts/architecture.md"
);
console.log(md?.slice(0, 200));

```

The `readPage` implementation (lines 71–99) resolves paths in both `raw/` and `wiki/` directories.

## Summary

- The **Wiki Engine** operates as a three-stage pipeline within the Memory Knowledge Service, orchestrated by `WikiSourceManager.ingest()` in [`manager.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/manager.ts).
- **Incremental processing** uses SHA-256 hashing to skip unchanged files, optimizing performance for large repositories.
- **LLM extraction** runs under strict concurrency controls (`globalLlmLimit`) and includes progress throttling to maintain UI responsiveness.
- **Content persistence** involves atomic SQLite transactions recreating FTS5 search indexes and directed graph edges for multi-hop queries.
- **Error handling** allows partial ingestions to complete, storing individual file errors while committing successful extractions to the index.

## Frequently Asked Questions

### How does the Wiki Engine handle duplicate or unchanged files during ingestion?

The engine implements change detection via SHA-256 hashing in `classifySources`. When `runIngestIncremental` executes, it compares each file’s current hash against entries in the SQLite `source` table (retrieved via `readSourceStates`). Files with identical hashes are categorized as *skipped*, bypassing expensive LLM extraction. Only new or modified files enter the *toIngest* queue, significantly reducing processing overhead for incremental updates.

### What happens if the LLM service is unavailable during the overview generation step?

The overview generation phase is fault-isolated from the core ingestion logic. When `generateOverview` is invoked (lines 552–558), the system checks for a valid LLM client. If the client cannot be initialized—due to network issues, authentication failures, or configuration errors—the pipeline catches this exception and continues with the remaining finalize steps. The Wiki remains fully functional with its pages and search index intact, simply lacking the auto-generated summary page until the next successful ingest.

### Which SQLite tables are rebuilt during the index write operation?

The `writeIndex` function (lines 94–104) transactionally recreates three tables: `wiki_fts` for full-text search using FTS5, `page_meta` for page metadata and static snippets, and `graph_edge` for storing directed wikilink relationships. These tables enable both keyword search via `ftsSearch` and graph traversal through `graphMultiHopSearch`. The operation occurs within a single write transaction to ensure consistency between the file system state and the searchable index.

### Where is the concurrency limit for LLM extraction defined?

Ingestion concurrency is controlled at two levels. Per-project limits are determined by `getIngestConcurrency` (lines 1000–1002), while a global semaphore named `globalLlmLimit` throttles total simultaneous LLM calls across all Wikis. This dual-layer approach prevents individual projects from monopolizing resources while protecting the LLM provider from rate-limit violations during bulk ingestion operations.