How the LLM-Wiki Stores and Serves Documentation in TencentDB-Agent-Memory

The LLM-Wiki uses a three-layer architecture—multi-tenant filesystem storage with SQLite metadata, an asynchronous ingestion pipeline that builds BM25 indexes and LLM summaries, and a REST API that exposes search and page-reading tools to agents.

The LLM-Wiki component in the TencentDB-Agent-Memory repository provides team-level documentation storage and retrieval for AI agents. According to the source code, it combines a hierarchical filesystem layout, an asynchronous build queue, and a Hono-based REST API to manage the complete lifecycle of technical documentation. Understanding how the LLM-Wiki stores and serves documentation reveals how the system balances raw file storage with processed, searchable content.

Storage Architecture and Metadata Layer

The storage layer persists wiki metadata, filesystem layout, and status flags using a combination of directory-based partitioning and SQLite indexing.

Physical Directory Layout

Each wiki lives in a dedicated, multi-tenant directory structure defined by the WikiService.dirFor method in src/store/wiki-service.ts. The path follows the pattern:

{dataRoot}/{service_id}/{team_id}/{wiki_id}/

This layout enforces tenant isolation at the filesystem level, where service_id is extracted from the x-tdai-service-id HTTP header. The WikiService class manages this layout alongside an IKnowledgeStore SQLite interface that provides methods including create(), getById(), list(), updateMeta(), and delete().

Raw Files vs. Processed Pages

The wiki maintains two logical file groups within each directory:

• raw/* – Source files (Markdown, images) uploaded directly by clients. Operations on this layer do not trigger re-ingestion.
• page/* – Processed files generated by the ingestion pipeline. These contain automatically injected front-matter (specifically locked: true) and are indexed for search.

The service enforces the PAGE_FORBIDDEN_REFS whitelist to protect structural files like index and schema from modification.

Asynchronous Ingestion Pipeline

When raw files are ready, clients trigger the POST /v3/wiki/ingest endpoint to build searchable indexes and generate documentation summaries.

Build Queue and Context

The ingestion process is managed by the BuildQueue using a WikiBuildContext object that carries the wiki ID, service ID, team ID, target directory, and a callback ID (ingestRunId). The WikiService.ingest() method validates that the wiki is not already in a processing state before enqueuing work.

WikiWorker Processing Stages

The WikiWorker, implemented in src/engines/wiki/ingest-v2/*.ts, executes a multi-stage pipeline:

1. Source Scanning – Reads raw files, computes SHA-256 hashes, and records SourceStatus (uploaded, ingested, failed).
2. Front-Matter Injection – Adds locked: true to page files to prevent manual editing of processed content.
3. Index Building – Feeds each page into a BM25 index managed by src/engines/wiki/graph-search.ts.
4. LLM Summary Generation – Optionally calls the LLM binding (configured via callbackConfig) to generate a short overview that agents use to decide whether to explore the wiki.

Idempotency and Concurrency Control

The pipeline uses status flags stored in SQLite (processing, ready, failed) to ensure idempotency. Repeated ingest requests on a ready wiki are ignored, while concurrent requests receive an HTTP 409 busy response via the maybeWriteError helper in src/routes/wiki.ts.

REST API and Serving Layer

All public interactions flow through the Hono router defined in src/routes/wiki.ts, which wraps responses in uniform envelopes (wrapOk / wrapError) for the Memory Panel.

Multi-Tenant Endpoint Structure

Every endpoint requires the x-tdai-service-id header for tenant isolation. The API provides comprehensive CRUD operations:

• POST /v3/wiki/create – Generates a wiki_id and persists metadata via wikiService.create().
• POST /v3/wiki/get – Retrieves metadata using wikiService.getById() with tenancy validation.
• POST /v3/wiki/list – Enumerates wikis with pagination via wikiService.list().
• POST /v3/wiki/delete – Performs soft-deletion and disk cleanup via wikiService.delete() and wikiMgr.remove().
• POST /v3/wiki/update-meta – Updates name or summary without affecting files.

File and Page Operations

The API distinguishes between raw source management and processed page access:

Raw File Layer (raw/ls, raw/read, raw/write, raw/rm) provides direct CRUD on source files with size and path validation via maybeWriteError.

Page Layer (page/ls, page/read, page/write, page/rm) operates on processed content with enforcement of PAGE_FORBIDDEN_REFS to protect structural integrity.

Search and Graph Endpoints

• POST /v3/wiki/search – Executes BM25 full-text search across all pages by delegating to wikiMgr.search().
• POST /v3/wiki/graph – Returns a knowledge graph (nodes and edges) built from page headings and LLM-generated relations via wikiMgr.graph().

Agent Integration and Tool Registration

The KnowledgeToolsInjector in MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts discovers wiki assets (type === "wiki") and automatically registers two tools for each:

• search – Executes POST /v3/wiki/search.
• read_page – Executes POST /v3/wiki/page/read.

These tools are exposed to LLM agents as part of the system prompt, enabling autonomous documentation querying without explicit code changes.

Practical Usage Example

Below is a complete lifecycle showing how a client interacts with the LLM-Wiki storage and serving layers:

// 1. Create a new wiki for team "team-1"
await fetch("http://localhost:8421/v3/wiki/create", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ team_id: "team-1", name: "Project Design Wiki" })
});

// 2. Upload a raw markdown file
await fetch("http://localhost:8421/v3/wiki/raw/write", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({
    team_id: "team-1",
    wiki_id: "wiki-a1b2c3d4",
    files: [{ filename: "README.md", content: "# Overview\nDesign goals…" }]

  })
});

// 3. Trigger ingestion (builds index & LLM summary)
await fetch("http://localhost:8421/v3/wiki/ingest", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", user_id: "u-001" })
});

// 4. Search the wiki (BM25)
await fetch("http://localhost:8421/v3/wiki/search", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", query: "deployment process", limit: 5 })
});

// 5. Read a specific page (after ingest)
await fetch("http://localhost:8421/v3/wiki/page/read", {
  method: "POST",
  headers: { "x-tdai-service-id": "svc-12345", "Content-Type": "application/json" },
  body: JSON.stringify({ wiki_id: "wiki-a1b2c3d4", refs: ["deployment"] })
});

Summary

• Multi-tenant storage uses a filesystem hierarchy ({dataRoot}/{service_id}/{team_id}/{wiki_id}) managed by WikiService in src/store/wiki-service.ts, backed by SQLite metadata.
• Asynchronous ingestion runs via BuildQueue and WikiWorker in src/engines/wiki/ingest-v2/, producing BM25 indexes and LLM-generated summaries while enforcing idempotency through status flags.
• REST API in src/routes/wiki.ts serves raw files, processed pages, search results, and knowledge graphs, all wrapped in uniform response envelopes and isolated via the x-tdai-service-id header.
• Agent integration occurs through KnowledgeToolsInjector, which registers search and read_page tools pointing to /v3/wiki/search and /v3/wiki/page/read.

Frequently Asked Questions

What is the difference between raw files and pages in LLM-Wiki?

Raw files are source uploads (Markdown, images) stored under raw/ without processing or indexing. Pages are processed outputs stored under page/ that contain injected front-matter (locked: true) and are indexed for BM25 search. The PAGE_FORBIDDEN_REFS whitelist protects structural files like index and schema from modification at the page layer.

How does the ingestion pipeline handle concurrent requests?

The pipeline uses SQLite status flags (processing, ready, failed) to maintain state. If a wiki is already processing, concurrent ingest requests receive an HTTP 409 busy response. Re-ingesting a wiki that is already ready is ignored, making the operation idempotent according to the implementation in src/routes/wiki.ts.

How do agents discover and query wiki documentation?

The KnowledgeToolsInjector in MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts automatically discovers wiki assets (type === "wiki") and registers two tools in the agent's system prompt: search (calling POST /v3/wiki/search) and read_page (calling POST /v3/wiki/page/read). This allows agents to query documentation without explicit API integration.

What search algorithm does LLM-Wiki use for full-text search?

The system uses BM25 full-text search implemented in src/engines/wiki/graph-search.ts. The search endpoint (POST /v3/wiki/search) delegates to wikiMgr.search(), which queries the index built during the ingestion pipeline. Additionally, the graph endpoint (POST /v3/wiki/graph) provides knowledge graph navigation based on page headings and LLM-generated relations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}