How the Wiki Engine in Memory Knowledge Service Ingests and Processes Documents

The Wiki Engine in the Memory Knowledge Service transforms raw markdown sources into searchable, interlinked Wiki pages through a three-stage pipeline—ingestion and LLM extraction, merging with overview generation, and SQLite index rebuilding—orchestrated primarily through WikiSourceManager.ingest() in manager.ts.

The Memory Knowledge Service is a core component of the TencentCloud/TencentDB-Agent-Memory repository, responsible for converting unstructured documentation into queryable knowledge graphs. The Wiki Engine’s ingestion pipeline handles everything from SHA-256-based change detection to LLM-powered content extraction, ultimately building a full-text search index and directed graph structure for semantic retrieval.

High-Level Ingestion Architecture

The ingestion workflow centers on the WikiSourceManager class defined in MemoryKnowledge/src/engines/wiki/manager.ts. When an ingest request arrives via the API endpoint POST /v3/wiki/ingest, the system executes a coordinated sequence of operations across three logical stages.

Initialization and Source Registration

Before processing begins, the engine initializes a dedicated project directory through initWikiProject (lines 719–726). This creates the necessary folder structure under the configured storage path and registers the Wiki in the manager’s in-memory map. The init() sequence establishes the raw/sources directory where incoming documents are staged prior to processing.

The Three-Phase Pipeline

The core logic resides in runIngestIncremental (lines 611–791), which implements the incremental ingestion worker. This method classifies files, extracts content using LLM inference, merges results into Wiki pages, and rebuilds the search index. The pipeline distinguishes between ingest (new/changed files), skip (unchanged), and delete (removed) operations by comparing current SHA-256 hashes against the stored state in the SQLite source table.

Stage 1: Extraction and Classification

The first stage focuses on identifying what needs processing and transforming raw text into structured page candidates using LLM inference.

SHA-256-Based Source Classification

The classifySources function (invoked at lines 87–92) performs efficient change detection by comparing file hashes. For each document in raw/sources, the system calculates a SHA-256 checksum and queries the existing state via readSourceStates (lines 886–889). Files with matching hashes are skipped entirely, while new or modified files enter the extraction queue. Missing files trigger cleanup operations to maintain index integrity.

LLM-Powered Content Extraction

For each source marked for ingestion, the engine executes extractSource (dynamically imported from engines/wiki/ingest-v2/index.js). This function runs a parallel LLM chain that converts raw markdown into a Map<string, string> of candidate pages. The extraction process respects concurrency limits enforced by getIngestConcurrency (lines 1000–1002) and a global LLM semaphore (globalLlmLimit) to prevent rate-limiting issues.

Prior to LLM processing, the engine parses front-matter metadata via extractFrontmatter (lines 52–84) and identifies intra-Wiki links through extractWikilinks (lines 87–94). These extracted relationships populate the WikiPage type defined in types.ts and serve as the foundation for the graph construction phase.

Progress Throttling and Concurrency

To prevent UI flooding during long-running extractions, the system implements createThrottledProgressFn (lines 27–46). This utility emits progress updates only when phases change, when completion percentage advances, or when extraction reaches 90% completion. The throttle interval defaults to PROGRESS_THROTTLE_MS (500ms), ensuring sustainable update frequencies for large document sets.

Stage 2: Merging and Overview Generation

Once extraction completes, the pipeline transitions to merging candidates and generating high-level summaries.

Committing Candidate Pages

The commitCandidates function (lines 711–728) merges extracted content into the Wiki directory structure, writing finalized markdown files to disk. This operation handles conflicts and collects errors in a mergeErrors array (lines 117–121) without aborting the entire ingest, allowing partial successes to proceed to indexing.

Automatic Overview Creation

Following successful merges, the system optionally invokes generateOverview (lines 552–558) to create a summary page. This step requires a valid LLM client configuration; if unavailable, the pipeline skips overview generation but continues with index rebuilding. The overview synthesizes key themes from newly ingested content, providing navigational entry points for users.

Stage 3: Index Rebuilding and Search

The final stage persists the processed content into queryable data structures and refreshes the in-memory read models.

SQLite FTS5 and Graph Tables

The writeIndex function (lines 94–104) executes a transactional rebuild of three critical SQLite tables:

• wiki_fts: Full-text search index using FTS5
• page_meta: Page metadata with static snippets
• graph_edge: Directed wikilink relationships

Edge resolution occurs via resolveEdges (lines 200–224), which maps slugified page titles to internal IDs. This directed graph enables multi-hop traversal for semantic search expansion.

Graph Construction and Multi-Hop Search

After index reconstruction, loadReadModel (lines 15–35) builds the in-memory PageGraph structure. The search API leverages this graph through ftsSearch (lines 81–90), which tokenizes queries using tokenize (lines 40–74) before executing SQLite FTS5 queries. For expanded results, the engine employs graphMultiHopSearch defined in graph-search.ts to perform BFS traversal across linked pages.

Practical Implementation Examples

Registering and Ingesting a Wiki Programmatically

import { createWikiSourceManager } from "./src/engines/wiki/manager.js";

// Initialize manager with storage path
const wikiMgr = createWikiSourceManager("/var/wikis");

// Register new Wiki project
await wikiMgr.register({
  name: "product-wiki",
  path: "/var/wikis/product-wiki",
});

// Execute ingestion with LLM configuration
await wikiMgr.ingest("product-wiki", {
  provider: "openai",
  model: "gpt-4o-mini",
  apiKey: process.env.OPENAI_API_KEY,
});

Key references: createWikiSourceManager (lines 795–796), register (lines 42–46), ingest (lines 779–846).

API Endpoint Usage

POST /v3/wiki/ingest HTTP/1.1
Content-Type: application/json
Authorization: Bearer <token>

{
  "knowledge_id": "wiki-1",
  "service_url": "http://ks:8421/v3",
  "team_id": "team-1",
  "llm_config": {
    "provider": "openai",
    "model": "gpt-4o",
    "api_key": "***"
  }
}

The server handler in MemoryKnowledge/src/store/wiki-service.ts (lines 70–88) routes this request to wikiService.ingest().

Reading Processed Content

const md = await wikiMgr.readPage(
  "product-wiki", 
  "wiki/concepts/architecture.md"
);
console.log(md?.slice(0, 200));

The readPage implementation (lines 71–99) resolves paths in both raw/ and wiki/ directories.

Summary

• The Wiki Engine operates as a three-stage pipeline within the Memory Knowledge Service, orchestrated by WikiSourceManager.ingest() in manager.ts.
• Incremental processing uses SHA-256 hashing to skip unchanged files, optimizing performance for large repositories.
• LLM extraction runs under strict concurrency controls (globalLlmLimit) and includes progress throttling to maintain UI responsiveness.
• Content persistence involves atomic SQLite transactions recreating FTS5 search indexes and directed graph edges for multi-hop queries.
• Error handling allows partial ingestions to complete, storing individual file errors while committing successful extractions to the index.

Frequently Asked Questions

How does the Wiki Engine handle duplicate or unchanged files during ingestion?

The engine implements change detection via SHA-256 hashing in classifySources. When runIngestIncremental executes, it compares each file’s current hash against entries in the SQLite source table (retrieved via readSourceStates). Files with identical hashes are categorized as skipped, bypassing expensive LLM extraction. Only new or modified files enter the toIngest queue, significantly reducing processing overhead for incremental updates.

What happens if the LLM service is unavailable during the overview generation step?

The overview generation phase is fault-isolated from the core ingestion logic. When generateOverview is invoked (lines 552–558), the system checks for a valid LLM client. If the client cannot be initialized—due to network issues, authentication failures, or configuration errors—the pipeline catches this exception and continues with the remaining finalize steps. The Wiki remains fully functional with its pages and search index intact, simply lacking the auto-generated summary page until the next successful ingest.

Which SQLite tables are rebuilt during the index write operation?

The writeIndex function (lines 94–104) transactionally recreates three tables: wiki_fts for full-text search using FTS5, page_meta for page metadata and static snippets, and graph_edge for storing directed wikilink relationships. These tables enable both keyword search via ftsSearch and graph traversal through graphMultiHopSearch. The operation occurs within a single write transaction to ensure consistency between the file system state and the searchable index.

Where is the concurrency limit for LLM extraction defined?

Ingestion concurrency is controlled at two levels. Per-project limits are determined by getIngestConcurrency (lines 1000–1002), while a global semaphore named globalLlmLimit throttles total simultaneous LLM calls across all Wikis. This dual-layer approach prevents individual projects from monopolizing resources while protecting the LLM provider from rate-limit violations during bulk ingestion operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}