# How LLM-Wiki Transforms Documentation for Agents: A Technical Deep Dive

> Discover how LLM-Wiki transforms documentation for agents. Learn technical details on chunking, LLM summaries, embeddings, and SQLite indexing for efficient knowledge retrieval.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: deep-dive
- Published: 2026-08-21

---

**LLM-Wiki converts raw markdown documentation into a structured, vectorized knowledge base by chunking content, generating LLM-powered summaries and embeddings, and indexing them in SQLite for retrieval by downstream agents.**

LLM-Wiki is the core ingestion engine in the **TencentCloud/TencentDB-Agent-Memory** repository that bridges static human-written docs and AI-ready knowledge graphs. It parses markdown files, extracts semantic meaning through large language models, and exposes the results via REST APIs that chatbots and RAG pipelines consume.

## The 7-Step Documentation Transformation Pipeline

The transformation from raw markdown to agent-ready knowledge proceeds through a strictly ordered pipeline implemented in `MemoryKnowledge/src/engines/wiki/ingest-v2/`.

### Step 1: Ingestion Request and Build Context

A client initiates transformation by calling the Knowledge API endpoint `/wiki/ingest`. This invokes `WikiService.ingest()` in [`MemoryKnowledge/src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/wiki-service.ts) (line 272), which creates a **build context** and enqueues an async task into the `BuildQueue`. The service returns an `IngestResult`—either `ok`, `busy`, or `not_found`—while the heavy processing happens in the background.

### Step 2: Directory Scanning and Page Discovery

The worker process scans the wiki directory located at `{dataRoot}/{service_id}/{team_id}/{wiki_id}/wiki`. The function `collectBriefs` in [`MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts) (lines 28-48) recursively walks the filesystem, collecting every `.md` file while excluding structural files like [`index.md`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/index.md). This produces a manifest of all documentation pages requiring processing.

### Step 3: Intelligent Chunking and Front-Matter Extraction

Each markdown page undergoes preprocessing in [`MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts). The **chunker** splits content into roughly **1,000-character blocks** to fit LLM context windows. Simultaneously, [`frontmatter.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/frontmatter.ts) parses YAML front-matter to extract metadata fields—`title`, `type`, and `description`—that annotate the resulting chunks.

### Step 4: LLM-Driven Summarization and Embedding

For every chunk, the system invokes the **LLM client** (`LlmClient`) defined in [`MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts). Depending on configuration, the LLM either generates a concise summary for knowledge graph construction or produces **embedding vectors** representing semantic meaning. These vectors enable similarity search for agent queries.

### Step 5: Vector Index Construction

Generated records are persisted to a SQLite database (`index.db`) managed by [`MemoryKnowledge/src/engines/wiki/index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts). The module provides `upsertSource` and `listSources` functions that populate both **FTS5 full-text search** indices and vector storage. This dual-indexing strategy supports both keyword-based and semantic retrieval patterns.

### Step 6: Global Overview Synthesis

After all pages are processed, `generateOverview()` (in [`overview.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/overview.ts)) feeds collected page briefs to the LLM using the `OVERVIEW_SYSTEM` prompt template. The LLM synthesizes a **global overview** document ([`overview.md`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/overview.md)) that ties the knowledge base together using `[[wikilinks]]` syntax, creating a navigable graph structure for agents and humans alike.

### Step 7: Publication and Agent Exposure

Finally, [`wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/wiki-service.ts) (lines 322-344) commits the [`overview.md`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/overview.md) and updated `index.db` to the wiki directory. Agents query this published knowledge via the OpenAPI-specified REST interface, retrieving page contents, embedding vectors, or the generated overview on demand.

## Core Architectural Concepts

### Asset Identification and Multi-Tenancy

Every wiki receives a globally unique identifier following the `wiki-<8-char>` pattern defined in [`MemoryCore/src/gateway/generated/types.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/gateway/generated/types.ts) (within the `AssetMutateData` schema). This naming convention enables multi-tenant isolation while maintaining a single namespace for agent queries.

### State Machine and Lifecycle Management

The system tracks wiki state through `IKnowledgeStore`, moving entities through `pending → processing → ready` or `failed`. The `BuildQueue` respects these states to prevent concurrent modification conflicts during ingestion.

### Soft-Delete and Cleanup Workflows

Deleting a wiki sets a cancellation flag rather than immediate removal. The system evicts the SQLite index from memory and removes the filesystem directory only after confirming all worker processes have finished, preventing corruption of active ingestion jobs.

### Telemetry and Observability

Each major pipeline step (`wiki-ingest`, `wiki-overview`, etc.) is wrapped in an OpenTelemetry span via the tracer defined in [`MemoryKnowledge/src/telemetry.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/telemetry.ts). This provides operational metrics including duration histograms and error rates for production monitoring.

## Implementation Examples

### Triggering Ingestion via Node.js

```typescript
import { WikiService } from "./store/wiki-service.js";

const service = new WikiService({
  store: myKnowledgeStore,
  dataRoot: "/var/memory/data",
  worker: myWikiWorker,
});

await service.ingest("svc-001", "team-42", "wiki-9c1f2b");

```

This call instantiates the build context, enqueues the transformation task, and immediately returns a status enum without blocking the client.

### Worker Pipeline Implementation

```typescript
export const wikiWorker: WikiWorker = async (ctx) => {
  const { wikiId, dir } = ctx;
  // 1. Chunk pages
  const chunks = await chunkAllPages(dir);
  // 2. Summarise & embed via LLM
  const llm = getLlmClient();
  const summaries = await Promise.all(
    chunks.map((c) => llm.summarise(c.text))
  );
  // 3. Store in SQLite index
  await withWriteDb(wikiId, (db) => {
    summaries.forEach((s) => upsertSource(db, s));
  });
  // 4. Generate overview
  await generateOverview(dir, llm);
};

```

The worker implements the core transformation logic: chunking, LLM enrichment, vector persistence, and overview generation.

### Querying via REST API

```bash
curl -X GET "https://memory.example.com/api/v1/wiki/wiki-9c1f2b/page?ref=Home"

```

The endpoint returns the rendered markdown, associated front-matter metadata, and optionally the embedding vector for the requested page.

## Key Source Files

- **[`MemoryKnowledge/src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/store/wiki-service.ts)** – Orchestrates async ingestion, manages CRUD operations on raw and page files, and implements the wiki lifecycle state machine.

- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/chunker.ts)** – Implements the text segmentation algorithm that splits markdown into LLM-friendly chunks while preserving semantic boundaries.

- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/llm.ts)** – Abstracts LLM provider interactions (OpenAI, Azure, etc.) for summarization and embedding generation, keeping the engine provider-agnostic.

- **[`MemoryKnowledge/src/engines/wiki/index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/index-db.ts)** – Manages the SQLite-based storage layer with FTS5 full-text search and vector similarity capabilities via `upsertSource` and query APIs.

- **[`MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/src/engines/wiki/ingest-v2/overview.ts)** – Contains the page discovery (`collectBriefs`) and global overview synthesis (`generateOverview`) logic.

- **[`MemoryKnowledge/openapi.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/openapi.yaml)** – Describes the HTTP contract that agents use to retrieve transformed documentation, including LLM-Wiki specific endpoints.

## Summary

- **LLM-Wiki** transforms static markdown into queryable vector knowledge through a seven-stage pipeline implemented in `MemoryKnowledge/src/engines/wiki/ingest-v2/`.
- The system uses **1,000-character chunking**, LLM summarization, and **SQLite FTS5 + vector storage** to enable both lexical and semantic search.
- Each wiki receives a unique `wiki-<8-char>` ID and progresses through a `pending → processing → ready` state machine tracked in `IKnowledgeStore`.
- The **LLM client abstraction** in [`llm.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/llm.ts) allows swapping underlying providers without changing ingestion logic.
- Transformed documentation is exposed via **OpenAPI-specified REST endpoints** that agents query for content, embeddings, and auto-generated overview pages.

## Frequently Asked Questions

### What file formats does LLM-Wiki support for ingestion?

LLM-Wiki exclusively processes **Markdown files** (`.md`) stored in the wiki directory structure. The system specifically excludes structural files like [`index.md`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/index.md) from chunking to prevent duplication, focusing instead on content pages that contain documentation.

### How does LLM-Wiki handle large documentation repositories?

The system implements **asynchronous job queuing** via `BuildQueue` and processes pages in parallel using configurable workers. Large files are segmented into approximately **1,000-character chunks** before LLM processing, ensuring each chunk fits within model context windows while the SQLite vector index (`index.db`) provides efficient storage regardless of repository size.

### What database technology powers the vector search?

LLM-Wiki uses **SQLite** with **FTS5** extensions for full-text search alongside custom vector storage tables. The [`index-db.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/index-db.ts) module manages this hybrid index, providing functions like `upsertSource` and `listSources` that enable both exact keyword matching and embedding-based similarity search for agent queries.

### How do agents access the transformed knowledge?

Agents interact with published wikis through **REST APIs** defined in [`MemoryKnowledge/openapi.yaml`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryKnowledge/openapi.yaml). They can request specific pages by reference, retrieve embedding vectors for semantic similarity operations, or fetch the auto-generated [`overview.md`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/overview.md) that provides a high-level map of the entire knowledge base via `[[wikilinks]]` navigation.