How LLM-Wiki Transforms Documentation into a Searchable Knowledge Base

LLM-Wiki converts raw markdown documentation into a searchable knowledge base through a three-stage pipeline that ingests files, generates LLM-powered summaries, and exposes lightweight metadata to agents at runtime.

LLM-Wiki is the core documentation engine in the TencentCloud/TencentDB-Agent-Memory repository that transforms static markdown files into structured, queryable knowledge bases for LLM agents. The system processes design documents offline to create compact, semantically rich summaries that enable efficient runtime retrieval without consuming excessive tokens.

The Three-Stage Transformation Pipeline

The transformation process operates through distinct ingest, processing, and runtime stages defined in MemoryKnowledge/src/store/wiki-service.ts and related modules.

Stage 1: Ingestion and Raw File Management

The pipeline begins when you create a wiki asset using WikiService.create, which initializes metadata only and returns a wiki_id for subsequent operations. Raw markdown files are then uploaded to the storage path wiki/{service_id}/{team_id}/{wiki_id}/raw/sources via WikiService.rawWrite or WikiService.rawWriteStream.

When you explicitly trigger processing via WikiService.ingest, the service enqueues a build job that executes a configurable WikiWorker. This worker scans the raw sources directory, extracts markdown pages, and prepares them for transformation. The raw source tracking is maintained in an SQLite database (index.db) managed by initIndexDb and upsertSource in MemoryKnowledge/src/engines/wiki/index-db.js.

Stage 2: LLM-Driven Processing and Summarization

Once the worker processes the files, it performs two critical transformations. First, it injects a locked: true front-matter flag into each markdown page using the injectLockedTrue function, writing the processed output to wiki/{service_id}/{team_id}/{wiki_id}/wiki. This flag prevents accidental modifications to the generated content.

When the build completes, WikiService.onBuildComplete triggers generateWikiSummary (defined in MemoryKnowledge/src/callback.ts). This function uses createLlmClient to prompt an LLM (OpenAI or Anthropic) to synthesize a ≤100-character Chinese summary from the page titles and descriptions. The resulting summary is persisted back to the wiki row via store.updateWikiStatus, making it available for runtime queries.

Stage 3: Runtime Knowledge Exposure

At runtime, the KnowledgeToolsInjector (located in MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts) injects a <knowledge_tools> block into the system prompt. For each wiki resource, it adds an about attribute containing the LLM-generated summary via summaryAttr.

Agents interact with the knowledge base through two primary operations: they first search the wiki using the knowledge/search endpoint (using the summary as a relevance cue), then read_page to fetch the full markdown content when needed. Because the summary is concise, the LLM can determine relevance without scanning entire page sets, making queries fast and token-efficient.

End-to-End Implementation Example

The following TypeScript implementation demonstrates the complete workflow from creation to runtime injection:

// 1. Create wiki metadata
const { row } = wikiService.create({
  service_id: "ks",
  team_id: "team-1",
  name: "Design Docs",
  source_type: "manual",
});
// row.wiki_id serves as the persistent identifier

// 2. Upload raw documentation
await wikiService.rawWrite(
  "ks", 
  "team-1", 
  row.wiki_id,
  "architecture.md",
  "# Architecture\n\nDesign details …"

);

// 3. Trigger asynchronous ingestion
const result = wikiService.ingest("ks", "team-1", row.wiki_id);
// result.kind === "ok" indicates build queued successfully

// 4. Worker processes files (simplified implementation)
async function myWikiWorker(ctx: WikiBuildContext) {
  // Parses raw files, injects locked:true front-matter, 
  // and writes processed pages to wiki/.../wiki
  return { pageCount: 12 };
}

After processing completes, the knowledge base becomes available to agents:

// 5. Inject wiki into agent context
const blocks = await knowledgeToolsInjector.execute(agentCtx);
// Returns XML block like:
// <knowledge type="wiki" id="wiki-123" url="http://ks:8421/v3"
//   name="Design Docs" about="团队设计文档的概览 …" />

Key Architectural Components

The transformation relies on several specific modules working in concert:

• MemoryKnowledge/src/store/wiki-service.ts – Orchestrates wiki assets through create, rawWrite, and ingest methods, plus handles build completion callbacks.

• MemoryKnowledge/src/callback.ts – Contains generateWikiSummary, which prompts the LLM to create the concise Chinese abstract used for relevance matching.

• MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts – Implements KnowledgeToolsInjector.execute to insert wiki resources into system prompts via the summaryAttr attribute.

• MemoryKnowledge/src/engines/wiki/index-db.js – Manages the SQLite index.db for tracking raw source registration and state.

• MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js – Handles cascade deletion of processed pages when raw sources are removed, maintaining referential integrity.

Summary

• Three-stage pipeline: Ingest raw files → LLM processing and summarization → Runtime injection with lightweight metadata.

• Offline heavy lifting: Document parsing, indexing, and summarization occur during the build phase, not at query time.

• Token-efficient retrieval: The ≤100-character Chinese summary acts as a relevance filter, allowing agents to decide resource utility without loading full documents.

• Immutable processed content: The locked: true front-matter flag ensures generated wiki pages remain stable after creation.

Frequently Asked Questions

What file formats does LLM-Wiki support for ingestion?

LLM-Wiki primarily processes markdown files. The WikiService.rawWrite methods accept raw markdown content that the WikiWorker parses into structured pages during the ingest phase.

How does the Chinese summary generation improve search performance?

The generateWikiSummary function creates a concise ≤100-character Chinese abstract that captures the document's essence. When KnowledgeToolsInjector adds this to the system prompt via the about attribute, LLM agents can evaluate relevance using this compact descriptor rather than scanning full page content, reducing token consumption and latency.

Where does LLM-Wiki store processed documentation?

Raw files reside in wiki/{service_id}/{team_id}/{wiki_id}/raw/sources, while processed pages with injected locked: true front-matter are written to wiki/{service_id}/{team_id}/{wiki_id}/wiki. Metadata and source tracking are maintained in an SQLite database (index.db) managed through MemoryKnowledge/src/engines/wiki/index-db.js.

Can the wiki ingestion process handle large documentation sets?

Yes. The architecture uses an asynchronous worker pattern where WikiService.ingest enqueues build jobs rather than processing synchronously. The worker implementation can be customized, and the system handles cascade deletions through MemoryKnowledge/src/engines/wiki/ingest-v2/cascade.js to manage updates efficiently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}