How LLM-Wiki Transforms Documentation for Agents: A Technical Deep Dive

LLM-Wiki converts raw markdown documentation into a structured, vectorized knowledge base by chunking content, generating LLM-powered summaries and embeddings, and indexing them in SQLite for retrieval by downstream agents.

LLM-Wiki is the core ingestion engine in the TencentCloud/TencentDB-Agent-Memory repository that bridges static human-written docs and AI-ready knowledge graphs. It parses markdown files, extracts semantic meaning through large language models, and exposes the results via REST APIs that chatbots and RAG pipelines consume.

The 7-Step Documentation Transformation Pipeline

The transformation from raw markdown to agent-ready knowledge proceeds through a strictly ordered pipeline implemented in MemoryKnowledge/src/engines/wiki/ingest-v2/.

Step 1: Ingestion Request and Build Context

A client initiates transformation by calling the Knowledge API endpoint /wiki/ingest. This invokes WikiService.ingest() in MemoryKnowledge/src/store/wiki-service.ts (line 272), which creates a build context and enqueues an async task into the BuildQueue. The service returns an IngestResult—either ok, busy, or not_found—while the heavy processing happens in the background.

Step 2: Directory Scanning and Page Discovery

The worker process scans the wiki directory located at {dataRoot}/{service_id}/{team_id}/{wiki_id}/wiki. The function collectBriefs in MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts (lines 28-48) recursively walks the filesystem, collecting every .md file while excluding structural files like index.md. This produces a manifest of all documentation pages requiring processing.

Step 3: Intelligent Chunking and Front-Matter Extraction

Each markdown page undergoes preprocessing in MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts. The chunker splits content into roughly 1,000-character blocks to fit LLM context windows. Simultaneously, frontmatter.ts parses YAML front-matter to extract metadata fields—title, type, and description—that annotate the resulting chunks.

Step 4: LLM-Driven Summarization and Embedding

For every chunk, the system invokes the LLM client (LlmClient) defined in MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts. Depending on configuration, the LLM either generates a concise summary for knowledge graph construction or produces embedding vectors representing semantic meaning. These vectors enable similarity search for agent queries.

Step 5: Vector Index Construction

Generated records are persisted to a SQLite database (index.db) managed by MemoryKnowledge/src/engines/wiki/index-db.ts. The module provides upsertSource and listSources functions that populate both FTS5 full-text search indices and vector storage. This dual-indexing strategy supports both keyword-based and semantic retrieval patterns.

Step 6: Global Overview Synthesis

After all pages are processed, generateOverview() (in overview.ts) feeds collected page briefs to the LLM using the OVERVIEW_SYSTEM prompt template. The LLM synthesizes a global overview document (overview.md) that ties the knowledge base together using [[wikilinks]] syntax, creating a navigable graph structure for agents and humans alike.

Step 7: Publication and Agent Exposure

Finally, wiki-service.ts (lines 322-344) commits the overview.md and updated index.db to the wiki directory. Agents query this published knowledge via the OpenAPI-specified REST interface, retrieving page contents, embedding vectors, or the generated overview on demand.

Core Architectural Concepts

Asset Identification and Multi-Tenancy

Every wiki receives a globally unique identifier following the wiki-<8-char> pattern defined in MemoryCore/src/gateway/generated/types.ts (within the AssetMutateData schema). This naming convention enables multi-tenant isolation while maintaining a single namespace for agent queries.

State Machine and Lifecycle Management

The system tracks wiki state through IKnowledgeStore, moving entities through pending → processing → ready or failed. The BuildQueue respects these states to prevent concurrent modification conflicts during ingestion.

Soft-Delete and Cleanup Workflows

Deleting a wiki sets a cancellation flag rather than immediate removal. The system evicts the SQLite index from memory and removes the filesystem directory only after confirming all worker processes have finished, preventing corruption of active ingestion jobs.

Telemetry and Observability

Each major pipeline step (wiki-ingest, wiki-overview, etc.) is wrapped in an OpenTelemetry span via the tracer defined in MemoryKnowledge/src/telemetry.ts. This provides operational metrics including duration histograms and error rates for production monitoring.

Implementation Examples

Triggering Ingestion via Node.js

import { WikiService } from "./store/wiki-service.js";

const service = new WikiService({
  store: myKnowledgeStore,
  dataRoot: "/var/memory/data",
  worker: myWikiWorker,
});

await service.ingest("svc-001", "team-42", "wiki-9c1f2b");

This call instantiates the build context, enqueues the transformation task, and immediately returns a status enum without blocking the client.

Worker Pipeline Implementation

export const wikiWorker: WikiWorker = async (ctx) => {
  const { wikiId, dir } = ctx;
  // 1. Chunk pages
  const chunks = await chunkAllPages(dir);
  // 2. Summarise & embed via LLM
  const llm = getLlmClient();
  const summaries = await Promise.all(
    chunks.map((c) => llm.summarise(c.text))
  );
  // 3. Store in SQLite index
  await withWriteDb(wikiId, (db) => {
    summaries.forEach((s) => upsertSource(db, s));
  });
  // 4. Generate overview
  await generateOverview(dir, llm);
};

The worker implements the core transformation logic: chunking, LLM enrichment, vector persistence, and overview generation.

Querying via REST API

curl -X GET "https://memory.example.com/api/v1/wiki/wiki-9c1f2b/page?ref=Home"

The endpoint returns the rendered markdown, associated front-matter metadata, and optionally the embedding vector for the requested page.

Key Source Files

• MemoryKnowledge/src/store/wiki-service.ts – Orchestrates async ingestion, manages CRUD operations on raw and page files, and implements the wiki lifecycle state machine.

• MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts – Implements the text segmentation algorithm that splits markdown into LLM-friendly chunks while preserving semantic boundaries.

• MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts – Abstracts LLM provider interactions (OpenAI, Azure, etc.) for summarization and embedding generation, keeping the engine provider-agnostic.

• MemoryKnowledge/src/engines/wiki/index-db.ts – Manages the SQLite-based storage layer with FTS5 full-text search and vector similarity capabilities via upsertSource and query APIs.

• MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts – Contains the page discovery (collectBriefs) and global overview synthesis (generateOverview) logic.

• MemoryKnowledge/openapi.yaml – Describes the HTTP contract that agents use to retrieve transformed documentation, including LLM-Wiki specific endpoints.

Summary

• LLM-Wiki transforms static markdown into queryable vector knowledge through a seven-stage pipeline implemented in MemoryKnowledge/src/engines/wiki/ingest-v2/.
• The system uses 1,000-character chunking, LLM summarization, and SQLite FTS5 + vector storage to enable both lexical and semantic search.
• Each wiki receives a unique wiki-<8-char> ID and progresses through a pending → processing → ready state machine tracked in IKnowledgeStore.
• The LLM client abstraction in llm.ts allows swapping underlying providers without changing ingestion logic.
• Transformed documentation is exposed via OpenAPI-specified REST endpoints that agents query for content, embeddings, and auto-generated overview pages.

Frequently Asked Questions

What file formats does LLM-Wiki support for ingestion?

LLM-Wiki exclusively processes Markdown files (.md) stored in the wiki directory structure. The system specifically excludes structural files like index.md from chunking to prevent duplication, focusing instead on content pages that contain documentation.

How does LLM-Wiki handle large documentation repositories?

The system implements asynchronous job queuing via BuildQueue and processes pages in parallel using configurable workers. Large files are segmented into approximately 1,000-character chunks before LLM processing, ensuring each chunk fits within model context windows while the SQLite vector index (index.db) provides efficient storage regardless of repository size.

What database technology powers the vector search?

LLM-Wiki uses SQLite with FTS5 extensions for full-text search alongside custom vector storage tables. The index-db.ts module manages this hybrid index, providing functions like upsertSource and listSources that enable both exact keyword matching and embedding-based similarity search for agent queries.

How do agents access the transformed knowledge?

Agents interact with published wikis through REST APIs defined in MemoryKnowledge/openapi.yaml. They can request specific pages by reference, retrieve embedding vectors for semantic similarity operations, or fetch the auto-generated overview.md that provides a high-level map of the entire knowledge base via [[wikilinks]] navigation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}