How to Set Up Wiki Ingestion with LLM-Powered Document Processing in TencentDB-Agent-Memory

Setting up wiki ingestion requires registering the wiki source via MetadataClient, triggering the /wiki/ingest endpoint, and configuring LLM_MODE=proxy to enable LLM-powered summarization through the Knowledge Service on port :8421.

TencentDB-Agent-Memory provides a comprehensive knowledge service that ingests wiki pages, generates LLM-powered summaries, and exposes them to agents as searchable tools. This guide walks through the complete pipeline—from registering a wiki source to injecting processed summaries into agent prompts—using the official Python SDK and TypeScript APIs. The implementation relies on three distinct layers: the management-plane for metadata, the data-plane for LLM processing, and the injection-pipeline for agent integration.

Architecture Overview

The wiki ingestion pipeline operates across three specialized layers that handle distinct responsibilities:

Layer	Responsibility	Key Component
Management-plane	CRUD operations for knowledge metadata	MetadataClient (sdk/memory-core/python/tencentdb_agent_memory/v3/metadata_client.py)
Data-plane	Page retrieval, LLM summarization, and storage	Knowledge Service (:8421)
Injection-pipeline	Exposure of summaries to agents	KnowledgeToolsInjector (MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts)

When a wiki source is registered, the Knowledge Service stores raw pages under /data/knowledge and processes them through an LLM configured via the context proxy. The generated summaries populate a SQLite database within the tdai-panel-data volume, which agents later query for context-aware responses.

Prerequisites and Configuration

Before triggering ingestion, verify that your deployment includes the following required settings:

• LLM_MODE=proxy (line 82 in deploy/panel-knowledge-combined/start-combined.sh): Enables the LLM-powered ingest path through the context proxy.
• KNOWLEDGE_DATA_DIR=/data/knowledge (defined in deploy/panel-knowledge-combined/README.md): Specifies the directory for raw wiki files, SQLite databases, and processing logs.
• tdai-panel-data volume: Mounted into the Knowledge Service container to persist wiki data across restarts (documented in deploy/global-images/README.md).
• llm_wiki.enabled flag in MemoryProxy/config.example.yaml: Controls whether wiki assets are exposed to agents through the injection system.

Step-by-Step Implementation

Register the Wiki Source

Use the Python SDK's MetadataClient to register your wiki with the knowledge service. This creates the metadata record that the ingestion pipeline references.

from tencentdb_agent_memory.v3 import MetadataClient

meta = MetadataClient(base_url="http://memory:8080/v3")
meta.create_knowledge(
    {
        "knowledge_id": "wiki-docs",
        "type": "wiki",
        "service_url": "http://ks:8421/v3",
        "name": "Project Wiki",
        "team_id": "team-1",
    }
)

This call targets the create_knowledge method implemented in sdk/memory-core/python/tencentdb_agent_memory/v3/metadata_client.py, which validates the payload and registers the wiki in the system metadata store.

Trigger Ingestion

Initiate the LLM processing by calling the wiki ingest endpoint. The front-end wrapper in MemoryPanel/web/src/lib/api/knowledge-api.ts provides a clean abstraction:

import { knowledgeApi } from '@/lib/api/knowledge-api';

async function startWikiIngest(wikiId: string) {
  await knowledgeApi.ingest(wikiId);
}

The underlying implementation posts to /wiki/ingest with the wiki identifier (lines 318-322 in knowledge-api.ts). The Knowledge Service then queues the pages for background processing.

Monitor Ingestion Status

Poll the status endpoint until internal_status transitions from ingesting to ready. The SDK provides a helper method that implements this polling logic:

async function ingestWithPolling(wikiId: string) {
  await knowledgeApi.ingest(wikiId);
  while (true) {
    const status = await knowledgeApi.get(wikiId);
    if (status.internal_status !== 'ingesting') break;
    await new Promise(r => setTimeout(r, 2000));
  }
}

This pattern appears in knowledge-api.ts (lines 389-425), where the ingestWithPolling method orchestrates the status check loop with a two-second interval.

Inject Summaries into Agent Context

Once the status reaches ready, the KnowledgeToolsInjector (MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts) discovers assets of type llm_wiki and injects their summaries into the system prompt. The injector assembles a knowledge block containing the title and summary for each processed page:

{
  "knowledge": [
    {
      "type": "wiki",
      "id": "wiki-docs",
      "title": "Design Overview",
      "summary": "This wiki page describes the overall design of the system..."
    }
  ]
}

The discovery flow (lines 3-15) identifies eligible wiki assets, while the summary handling (lines 93-101) populates the about field used in the prompt injection.

Code Reference Guide

Complete Ingestion Workflow (Python)

from tencentdb_agent_memory.v3 import MetadataClient
import time

meta = MetadataClient(base_url="http://memory:8080/v3")

# 1. Register

meta.create_knowledge({
    "knowledge_id": "wiki-docs",
    "type": "wiki",
    "service_url": "http://ks:8421/v3",
    "name": "Project Wiki",
    "team_id": "team-1"
})

# 2. Trigger via HTTP or SDK methods

# 3. Poll until ready

Agent Consumption Pattern

Agents access processed wiki content through two primary methods exposed by the Knowledge Service: search for semantic retrieval across summaries, and read_page for direct access to specific documents. The LLM-summarized content serves as the single source of truth for agent decision-making, keeping the agent loop lightweight by avoiding full document retrieval during inference.

Summary

• Three-layer architecture: Management-plane (MetadataClient), data-plane (Knowledge Service on :8421), and injection-pipeline (KnowledgeToolsInjector) work sequentially to process wiki content.
• Configuration requirements: Set LLM_MODE=proxy in the startup script and ensure KNOWLEDGE_DATA_DIR=/data/knowledge exists with the tdai-panel-data volume mounted.
• Implementation flow: Register via Python SDK (metadata_client.py), trigger via TypeScript API (knowledge-api.ts), monitor status through polling, and inject into prompts via the knowledge tools injector.
• Storage: Processed summaries persist in SQLite within the mounted volume, with raw pages stored under /data/knowledge.

Frequently Asked Questions

What LLM configuration is required for wiki ingestion?

Set LLM_MODE=proxy in deploy/panel-knowledge-combined/start-combined.sh (line 82) to route wiki content through the context proxy for summarization. This enables the LLM-powered processing pipeline that generates the about summaries and title-level metadata stored in the knowledge database.

How do I monitor the status of a wiki ingestion job?

Poll the /wiki/get endpoint using the ingestWithPolling method in MemoryPanel/web/src/lib/api/knowledge-api.ts (lines 389-425). The endpoint returns an internal_status field that transitions from ingesting to ready once the LLM has processed all pages and stored the summaries in the SQLite database.

Where are the processed wiki summaries stored?

The Knowledge Service stores raw wiki files and generated summaries in /data/knowledge (configured via KNOWLEDGE_DATA_DIR), which maps to the tdai-panel-data Docker volume. The LLM-generated summaries specifically reside in a SQLite database within this directory, ensuring persistence across container restarts.

How do agents access the ingested wiki content?

The KnowledgeToolsInjector (MemoryProxy/src/injection/injectors/knowledge-tools-injector.ts) automatically discovers wiki assets with llm_wiki.enabled and injects their summaries into the agent's system prompt under the knowledge block. Agents can then call search or read_page capabilities against these summaries to retrieve context without processing raw documents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}