Which Database and Search Engine Does the TencentDB Agent Memory Wiki Use?

The Wiki component uses SQLite as its persistent database and SQLite FTS5 as its full-text search engine, implementing BM25 ranking for hybrid English-Chinese tokenization.

The TencentDB Agent Memory project includes a Wiki component designed for LLM-assisted knowledge retrieval. Understanding which database and search engine the Wiki uses is essential for developers optimizing search performance or extending the knowledge base architecture. According to the source code in the TencentDB-Agent-Memory repository, the system relies on a lightweight, self-contained SQLite architecture with native full-text search capabilities.

SQLite as the Persistent Database

The Wiki stores all persistent data—including pages, metadata, raw files, and the BM25 search index—in a SQLite database file. By default, this file is named knowledge.db and is managed through the Drizzle ORM with the SQLite dialect configured in MemoryKnowledge/drizzle.config.ts.

The Drizzle configuration explicitly sets dialect: "sqlite", enabling schema migrations and type-safe queries against the SQLite backend. This approach eliminates external database dependencies, making the Wiki component portable and easy to deploy across different environments without requiring separate database servers.

Key files managing the database layer include:

• MemoryKnowledge/drizzle.config.ts – Defines the SQLite dialect and connection settings
• MemoryKnowledge/src/store/wiki-service.ts – Handles Wiki lifecycle management, metadata persistence, and coordinates index creation in the SQLite database
• MemoryKnowledge/src/engines/wiki/index-db.ts – Defines the schema for the FTS5 virtual table and related storage structures

SQLite FTS5 as the Search Engine

For full-text retrieval, the Wiki leverages SQLite FTS5 (Full-Text Search version 5), a virtual table module built into SQLite. The search engine is implemented in MemoryKnowledge/src/engines/wiki/manager.ts, where the code explicitly references "Search Engine (SQLite FTS5)" in its architecture comments.

The FTS5 implementation provides several critical capabilities:

• BM25 Ranking – The virtual table calculates relevance scores using the BM25 algorithm, ranking results by term frequency and inverse document frequency
• Hybrid Tokenization – The engine tokenizes both English and Chinese text using tokenize() functions, storing token lists in the wiki_fts virtual table to ensure consistent query processing
• MATCH Query Syntax – Search operations execute SQL MATCH queries against the wiki_fts table, returning snippet previews and relevance-ranked results

This architecture delivers fast, token-aware retrieval suitable for natural language queries without requiring external search services like Elasticsearch or OpenSearch.

How the FTS5 Search Pipeline Works

When a user submits a search query, the Wiki manager processes the request through a structured pipeline defined in MemoryKnowledge/src/engines/wiki/manager.ts:

1. Query Tokenization – The input string is processed through the tokenize() function using the same tokenizer configuration applied during index creation
2. Virtual Table Query – The tokenized query executes against the wiki_fts FTS5 virtual table using SQLite's MATCH operator
3. Result Ranking – SQLite's built-in BM25 ranking automatically scores documents, returning the most relevant pages first
4. Snippet Generation – The query extracts contextual snippets using FTS5's snippet() function for preview display

This tokenization consistency—using identical tokenizers for indexing and querying—guarantees that search results match user expectations across multilingual content.

Practical Usage Examples

Creating and Ingesting Wiki Content

Create a new Wiki knowledge base via the Knowledge Service API:

# Create a new Wiki metadata entry

curl -X POST http://localhost:8421/v3/knowledge/wiki/create \
  -H "Content-Type: application/json" \
  -d '{
        "knowledge_id": "wiki-docs",
        "type": "wiki",
        "service_url": "http://localhost:8421/v3",
        "name": "Team Wiki",
        "team_id": "team-1"
      }'

Trigger content ingestion to build the SQLite index and FTS5 table:

# Ingest files from KNOWLEDGE_DATA_DIR into SQLite/FTS5

curl -X POST http://localhost:8421/v3/knowledge/wiki/ingest \
  -H "Content-Type: application/json" \
  -d '{"wiki_id":"wiki-docs"}'

Searching via the REST API

Execute BM25-ranked searches through the service endpoint:

# Search with BM25 ranking via FTS5

curl -X POST http://localhost:8421/v3/knowledge/wiki/search \
  -H "Content-Type: application/json" \
  -d '{
        "wiki_id": "wiki-docs",
        "query": "memory architecture",
        "limit": 5
      }'

Direct SQLite Debugging Queries

For debugging or advanced analytics, query the FTS5 virtual table directly:

-- Connect to the Wiki database (path specified in KNOWLEDGE_DB_PATH)
sqlite3 ./data/knowledge.db

-- Execute BM25 search with snippet highlighting
SELECT 
    rowid, 
    title, 
    snippet(wiki_fts, -1, '<b>', '</b>', '…', 10) AS snippet
FROM wiki_fts
WHERE wiki_fts MATCH 'memory architecture';

Summary

• SQLite serves as the sole persistent database for the Wiki component, storing all content and metadata in a single file (knowledge.db)
• SQLite FTS5 functions as the native full-text search engine through the wiki_fts virtual table, eliminating external search infrastructure dependencies
• BM25 ranking provides relevance scoring for search results without requiring additional ranking algorithms
• Drizzle ORM manages the SQLite schema and migrations using the sqlite dialect configuration
• Hybrid tokenization supports both English and Chinese text processing within the same search index
• The implementation resides primarily in MemoryKnowledge/src/engines/wiki/manager.ts and MemoryKnowledge/src/engines/wiki/index-db.ts

Frequently Asked Questions

Does the Wiki support other databases besides SQLite?

No, according to the source code analysis, the Wiki component is architected specifically for SQLite. The Drizzle configuration in MemoryKnowledge/drizzle.config.ts explicitly sets dialect: "sqlite", and the FTS5 search engine is a SQLite-specific extension. There are no abstraction layers or configuration options for alternative databases like PostgreSQL or MySQL in the current implementation.

What tokenization strategy does the Wiki FTS5 engine use?

The Wiki implements hybrid tokenization supporting both English and Chinese text. The tokenize() function in MemoryKnowledge/src/engines/wiki/manager.ts processes content during indexing and applies identical tokenization during query execution. This ensures that the wiki_fts virtual table stores tokenized representations that match the query tokens, enabling accurate retrieval across multilingual documents.

How is the BM25 ranking implemented in the SQLite FTS5 search?

BM25 ranking is provided natively by SQLite's FTS5 extension. When executing MATCH queries against the wiki_fts virtual table, SQLite automatically calculates BM25 scores based on term frequency and document frequency statistics maintained within the FTS5 index. The Wiki manager leverages this built-in functionality rather than implementing custom ranking logic, returning results ordered by the SQLite-computed relevance scores.

Can I query the Wiki search index directly via SQL?

Yes, the FTS5 virtual table wiki_fts can be queried directly using standard SQLite SQL syntax. You can execute MATCH operations, retrieve snippets using the snippet() function, and inspect the raw tokenized data within the SQLite database file. Direct SQL access is useful for debugging search relevance issues or performing bulk analytics operations outside the REST API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}