# How LLM-Wiki Transforms Documentation into a Searchable Knowledge Base > Discover how LLM-Wiki transforms markdown documentation into a searchable knowledge base using a three-stage pipeline for efficient agent access to information. - Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory) - Tags: how-to-guide - Published: 2026-08-23 --- **LLM-Wiki converts raw markdown documentation into a searchable knowledge base through a three-stage pipeline that ingests files, generates LLM-powered summaries, and exposes lightweight metadata to agents at runtime.** LLM-Wiki is the core documentation engine in the TencentCloud/TencentDB-Agent-Memory repository that transforms static markdown files into structured, queryable knowledge bases for LLM agents. The system processes design documents offline to create compact, semantically rich summaries that enable efficient runtime retrieval without consuming excessive tokens. ## The Three-Stage Transformation Pipeline The transformation process operates through distinct ingest, processing, and runtime stages defined in [`MemoryKnowledge/src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/wiki-service.ts) and related modules. ### Stage 1: Ingestion and Raw File Management The pipeline begins when you create a wiki asset using `WikiService.create`, which initializes metadata only and returns a `wiki_id` for subsequent operations. Raw markdown files are then uploaded to the storage path `wiki/{service_id}/{team_id}/{wiki_id}/raw/sources` via `WikiService.rawWrite` or `WikiService.rawWriteStream`. When you explicitly trigger processing via `WikiService.ingest`, the service enqueues a build job that executes a configurable **WikiWorker**. This worker scans the raw sources directory, extracts markdown pages, and prepares them for transformation. The raw source tracking is maintained in an SQLite database (`index.db`) managed by `initIndexDb` and `upsertSource` in [`MemoryKnowledge/src/engines/wiki/index-db.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.js). ### Stage 2: LLM-Driven Processing and Summarization Once the worker processes the files, it performs two critical transformations. First, it injects a `locked: true` front-matter flag into each markdown page using the `injectLockedTrue` function, writing the processed output to `wiki/{service_id}/{team_id}/{wiki_id}/wiki`. This flag prevents accidental modifications to the generated content. When the build completes, `WikiService.onBuildComplete` triggers `generateWikiSummary` (defined in [`MemoryKnowledge/src/callback.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/callback.ts)). This function uses `createLlmClient` to prompt an LLM (OpenAI or Anthropic) to synthesize a **≤100-character Chinese summary** from the page titles and descriptions. The resulting summary is persisted back to the wiki row via `store.updateWikiStatus`, making it available for runtime queries. ### Stage 3: Runtime Knowledge Exposure At runtime, the `KnowledgeToolsInjector` (located in [`MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts)) injects a `` block into the system prompt. For each wiki resource, it adds an `about` attribute containing the LLM-generated summary via `summaryAttr`. Agents interact with the knowledge base through two primary operations: they first **search** the wiki using the `knowledge/search` endpoint (using the summary as a relevance cue), then **read_page** to fetch the full markdown content when needed. Because the summary is concise, the LLM can determine relevance without scanning entire page sets, making queries fast and token-efficient. ## End-to-End Implementation Example The following TypeScript implementation demonstrates the complete workflow from creation to runtime injection: ```typescript // 1. Create wiki metadata const { row } = wikiService.create({ service_id: "ks", team_id: "team-1", name: "Design Docs", source_type: "manual", }); // row.wiki_id serves as the persistent identifier // 2. Upload raw documentation await wikiService.rawWrite( "ks", "team-1", row.wiki_id, "architecture.md", "# Architecture\n\nDesign details …" ); // 3. Trigger asynchronous ingestion const result = wikiService.ingest("ks", "team-1", row.wiki_id); // result.kind === "ok" indicates build queued successfully // 4. Worker processes files (simplified implementation) async function myWikiWorker(ctx: WikiBuildContext) { // Parses raw files, injects locked:true front-matter, // and writes processed pages to wiki/.../wiki return { pageCount: 12 }; } ``` After processing completes, the knowledge base becomes available to agents: ```typescript // 5. Inject wiki into agent context const blocks = await knowledgeToolsInjector.execute(agentCtx); // Returns XML block like: // ``` ## Key Architectural Components The transformation relies on several specific modules working in concert: - **[`MemoryKnowledge/src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/wiki-service.ts)** – Orchestrates wiki assets through `create`, `rawWrite`, and `ingest` methods, plus handles build completion callbacks. - **[`MemoryKnowledge/src/callback.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/callback.ts)** – Contains `generateWikiSummary`, which prompts the LLM to create the concise Chinese abstract used for relevance matching. - **[`MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts)** – Implements `KnowledgeToolsInjector.execute` to insert wiki resources into system prompts via the `summaryAttr` attribute. - **[`MemoryKnowledge/src/engines/wiki/index-db.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.js)** – Manages the SQLite `index.db` for tracking raw source registration and state. - **[`MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js)** – Handles cascade deletion of processed pages when raw sources are removed, maintaining referential integrity. ## Summary - **Three-stage pipeline**: Ingest raw files → LLM processing and summarization → Runtime injection with lightweight metadata. - **Offline heavy lifting**: Document parsing, indexing, and summarization occur during the build phase, not at query time. - **Token-efficient retrieval**: The ≤100-character Chinese summary acts as a relevance filter, allowing agents to decide resource utility without loading full documents. - **Immutable processed content**: The `locked: true` front-matter flag ensures generated wiki pages remain stable after creation. ## Frequently Asked Questions ### What file formats does LLM-Wiki support for ingestion? LLM-Wiki primarily processes markdown files. The `WikiService.rawWrite` methods accept raw markdown content that the WikiWorker parses into structured pages during the ingest phase. ### How does the Chinese summary generation improve search performance? The `generateWikiSummary` function creates a concise ≤100-character Chinese abstract that captures the document's essence. When `KnowledgeToolsInjector` adds this to the system prompt via the `about` attribute, LLM agents can evaluate relevance using this compact descriptor rather than scanning full page content, reducing token consumption and latency. ### Where does LLM-Wiki store processed documentation? Raw files reside in `wiki/{service_id}/{team_id}/{wiki_id}/raw/sources`, while processed pages with injected `locked: true` front-matter are written to `wiki/{service_id}/{team_id}/{wiki_id}/wiki`. Metadata and source tracking are maintained in an SQLite database (`index.db`) managed through [`MemoryKnowledge/src/engines/wiki/index-db.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.js). ### Can the wiki ingestion process handle large documentation sets? Yes. The architecture uses an asynchronous worker pattern where `WikiService.ingest` enqueues build jobs rather than processing synchronously. The worker implementation can be customized, and the system handles cascade deletions through [`MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js) to manage updates efficiently.