# How the LLM-Wiki Stores and Serves Documentation in TencentDB-Agent-Memory

> Discover how the LLM-Wiki stores and serves documentation using a three-layer architecture. Explore its multi-tenant storage, asynchronous ingestion pipeline, and REST API for efficient agent access.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: how-to-guide
- Published: 2026-08-29

---

**The LLM-Wiki uses a three-layer architecture—multi-tenant filesystem storage with SQLite metadata, an asynchronous ingestion pipeline that builds BM25 indexes and LLM summaries, and a REST API that exposes search and page-reading tools to agents.**

The LLM-Wiki component in the TencentDB-Agent-Memory repository provides team-level documentation storage and retrieval for AI agents. According to the source code, it combines a hierarchical filesystem layout, an asynchronous build queue, and a Hono-based REST API to manage the complete lifecycle of technical documentation. Understanding how the LLM-Wiki stores and serves documentation reveals how the system balances raw file storage with processed, searchable content.

## Storage Architecture and Metadata Layer

The storage layer persists wiki metadata, filesystem layout, and status flags using a combination of directory-based partitioning and SQLite indexing.

### Physical Directory Layout

Each wiki lives in a dedicated, multi-tenant directory structure defined by the `WikiService.dirFor` method in [`src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/store/wiki-service.ts). The path follows the pattern:

```text
{dataRoot}/{service_id}/{team_id}/{wiki_id}/

```

This layout enforces tenant isolation at the filesystem level, where `service_id` is extracted from the `x-tdai-service-id` HTTP header. The `WikiService` class manages this layout alongside an `IKnowledgeStore` SQLite interface that provides methods including `create()`, `getById()`, `list()`, `updateMeta()`, and `delete()`.

### Raw Files vs. Processed Pages

The wiki maintains two logical file groups within each directory:

- **`raw/*`** – Source files (Markdown, images) uploaded directly by clients. Operations on this layer do not trigger re-ingestion.
- **`page/*`** – Processed files generated by the ingestion pipeline. These contain automatically injected front-matter (specifically `locked: true`) and are indexed for search.

The service enforces the `PAGE_FORBIDDEN_REFS` whitelist to protect structural files like `index` and `schema` from modification.

## Asynchronous Ingestion Pipeline

When raw files are ready, clients trigger the `POST /v3/wiki/ingest` endpoint to build searchable indexes and generate documentation summaries.

### Build Queue and Context

The ingestion process is managed by the `BuildQueue` using a `WikiBuildContext` object that carries the wiki ID, service ID, team ID, target directory, and a callback ID (`ingestRunId`). The `WikiService.ingest()` method validates that the wiki is not already in a `processing` state before enqueuing work.

### WikiWorker Processing Stages

The **WikiWorker**, implemented in `src/engines/wiki/ingest-v2/*.ts`, executes a multi-stage pipeline:

1. **Source Scanning** – Reads raw files, computes SHA-256 hashes, and records `SourceStatus` (uploaded, ingested, failed).
2. **Front-Matter Injection** – Adds `locked: true` to page files to prevent manual editing of processed content.
3. **Index Building** – Feeds each page into a BM25 index managed by [`src/engines/wiki/graph-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/engines/wiki/graph-search.ts).
4. **LLM Summary Generation** – Optionally calls the LLM binding (configured via `callbackConfig`) to generate a short overview that agents use to decide whether to explore the wiki.

### Idempotency and Concurrency Control

The pipeline uses status flags stored in SQLite (`processing`, `ready`, `failed`) to ensure idempotency. Repeated ingest requests on a `ready` wiki are ignored, while concurrent requests receive an HTTP 409 `busy` response via the `maybeWriteError` helper in [`src/routes/wiki.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/wiki.ts).

## REST API and Serving Layer

All public interactions flow through the Hono router defined in [`src/routes/wiki.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/wiki.ts), which wraps responses in uniform envelopes (`wrapOk` / `wrapError`) for the Memory Panel.

### Multi-Tenant Endpoint Structure

Every endpoint requires the `x-tdai-service-id` header for tenant isolation. The API provides comprehensive CRUD operations:

- **`POST /v3/wiki/create`** – Generates a `wiki_id` and persists metadata via `wikiService.create()`.
- **`POST /v3/wiki/get`** – Retrieves metadata using `wikiService.getById()` with tenancy validation.
- **`POST /v3/wiki/list`** – Enumerates wikis with pagination via `wikiService.list()`.
- **`POST /v3/wiki/delete`** – Performs soft-deletion and disk cleanup via `wikiService.delete()` and `wikiMgr.remove()`.
- **`POST /v3/wiki/update-meta`** – Updates `name` or `summary` without affecting files.

### File and Page Operations

The API distinguishes between raw source management and processed page access:

**Raw File Layer** (`raw/ls`, `raw/read`, `raw/write`, `raw/rm`) provides direct CRUD on source files with size and path validation via `maybeWriteError`.

**Page Layer** (`page/ls`, `page/read`, `page/write`, `page/rm`) operates on processed content with enforcement of `PAGE_FORBIDDEN_REFS` to protect structural integrity.

### Search and Graph Endpoints

- **`POST /v3/wiki/search`** – Executes BM25 full-text search across all pages by delegating to `wikiMgr.search()`.
- **`POST /v3/wiki/graph`** – Returns a knowledge graph (nodes and edges) built from page headings and LLM-generated relations via `wikiMgr.graph()`.

## Agent Integration and Tool Registration

The **KnowledgeToolsInjector** in [`MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts) discovers wiki assets (`type === "wiki"`) and automatically registers two tools for each:

- **`search`** – Executes `POST /v3/wiki/search`.
- **`read_page`** – Executes `POST /v3/wiki/page/read`.

These tools are exposed to LLM agents as part of the system prompt, enabling autonomous documentation querying without explicit code changes.

## Practical Usage Example

Below is a complete lifecycle showing how a client interacts with the LLM-Wiki storage and serving layers:

```typescript
// 1. Create a new wiki for team "team-1"
await fetch("http://localhost:8421/v3/wiki/create", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ team_id: "team-1", name: "Project Design Wiki" })
});

// 2. Upload a raw markdown file
await fetch("http://localhost:8421/v3/wiki/raw/write", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({
    team_id: "team-1",
    wiki_id: "wiki-a1b2c3d4",
    files: [{ filename: "README.md", content: "# Overview\nDesign goals…" }]

  })
});

// 3. Trigger ingestion (builds index & LLM summary)
await fetch("http://localhost:8421/v3/wiki/ingest", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", user_id: "u-001" })
});

// 4. Search the wiki (BM25)
await fetch("http://localhost:8421/v3/wiki/search", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", query: "deployment process", limit: 5 })
});

// 5. Read a specific page (after ingest)
await fetch("http://localhost:8421/v3/wiki/page/read", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", refs: ["deployment"] })
});

```

## Summary

- **Multi-tenant storage** uses a filesystem hierarchy (`{dataRoot}/{service_id}/{team_id}/{wiki_id}`) managed by `WikiService` in [`src/store/wiki-service.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/store/wiki-service.ts), backed by SQLite metadata.
- **Asynchronous ingestion** runs via `BuildQueue` and `WikiWorker` in `src/engines/wiki/ingest-v2/`, producing BM25 indexes and LLM-generated summaries while enforcing idempotency through status flags.
- **REST API** in [`src/routes/wiki.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/wiki.ts) serves raw files, processed pages, search results, and knowledge graphs, all wrapped in uniform response envelopes and isolated via the `x-tdai-service-id` header.
- **Agent integration** occurs through `KnowledgeToolsInjector`, which registers `search` and `read_page` tools pointing to `/v3/wiki/search` and `/v3/wiki/page/read`.

## Frequently Asked Questions

### What is the difference between raw files and pages in LLM-Wiki?

Raw files are source uploads (Markdown, images) stored under `raw/` without processing or indexing. Pages are processed outputs stored under `page/` that contain injected front-matter (`locked: true`) and are indexed for BM25 search. The `PAGE_FORBIDDEN_REFS` whitelist protects structural files like `index` and `schema` from modification at the page layer.

### How does the ingestion pipeline handle concurrent requests?

The pipeline uses SQLite status flags (`processing`, `ready`, `failed`) to maintain state. If a wiki is already `processing`, concurrent ingest requests receive an HTTP 409 `busy` response. Re-ingesting a wiki that is already `ready` is ignored, making the operation idempotent according to the implementation in [`src/routes/wiki.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/routes/wiki.ts).

### How do agents discover and query wiki documentation?

The `KnowledgeToolsInjector` in [`MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts) automatically discovers wiki assets (`type === "wiki"`) and registers two tools in the agent's system prompt: `search` (calling `POST /v3/wiki/search`) and `read_page` (calling `POST /v3/wiki/page/read`). This allows agents to query documentation without explicit API integration.

### What search algorithm does LLM-Wiki use for full-text search?

The system uses **BM25** full-text search implemented in [`src/engines/wiki/graph-search.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/src/engines/wiki/graph-search.ts). The search endpoint (`POST /v3/wiki/search`) delegates to `wikiMgr.search()`, which queries the index built during the ingestion pipeline. Additionally, the graph endpoint (`POST /v3/wiki/graph`) provides knowledge graph navigation based on page headings and LLM-generated relations.