How the Wiki Custom Tokenizer Handles English and Chinese Text in TencentDB-Agent-Memory

The Wiki component processes multilingual content through a dual-stage pipeline—leveraging @node-rs/jieba for Chinese word segmentation during indexing (tokenizeForFts) while extracting English tokens via Unicode patterns during query construction (buildFtsQuery).

The TencentDB-Agent-Memory repository implements a sophisticated full-text search system for Wiki documents using SQLite FTS5. Understanding how the Wiki custom tokenizer handles English and Chinese text reveals the architecture behind its multilingual search capabilities and ensures optimal indexing for mixed-language datasets.

Indexing Stage: tokenizeForFts

The indexing pipeline converts raw Wiki content into searchable tokens stored in the FTS5 content column. According to the source code in MemoryCore/src/core/store/sqlite.ts (lines 78-90), the tokenizeForFts function implements language-aware segmentation strategies.

Chinese Segmentation with Jieba

When the optional @node-rs/jieba library is available, the tokenizer invokes jieba.cutForSearch(raw, true). This search-engine mode segments Chinese sentences into both full words and their constituent sub-words. For example, the term "人工智能" generates three tokens: "人工", "智能", and "人工智能". This approach ensures matches occur whether users search for partial terms or complete phrases.

Fallback Behavior and SQLite Integration

If the jieba library cannot be loaded, the system returns the raw string unchanged. The resulting token list is joined with single spaces and stored in the FTS5 table, allowing SQLite's built-in unicode61 tokenizer to further split content on whitespace and punctuation boundaries.

Query Construction Stage: buildFtsQuery

The search pipeline processes user queries through the buildFtsQuery function (lines 102-124 in MemoryCore/src/core/store/sqlite.ts), transforming natural language input into valid FTS5 MATCH expressions.

Chinese Query Processing

For Chinese text, the tokenizer applies the same cutForSearch method used during indexing. It then filters the results to remove empty tokens, pure punctuation characters, and entries matching the ZH_STOP_WORDS list. Duplicate tokens are removed before being quoted and OR-joined into the final query string.

English and Unicode Token Extraction

When jieba is unavailable, the tokenizer falls back to the Unicode regex /[\p{L}\p{N}_]+/gu to extract sequences of letters, numbers, or underscores. This pattern captures English words, numeric identifiers, and mixed alphanumeric tokens (such as "TypeScript" or "API_v2") as discrete units suitable for full-text matching.

FTS5 MATCH String Generation

Processed tokens are formatted as quoted strings and combined with the OR operator to construct the final MATCH expression. A query like "用户旅行 planning" becomes "用户" OR "旅行" OR "planning" when jieba is present.

Implementation Examples

// Example: indexing a Wiki page
const rawContent = "用户五月去日本旅行 and planning a trip to Tokyo.";
const indexed = tokenizeForFts(rawContent);
// With jieba: "用户 五月 去 日本 旅行 and planning a trip to Tokyo."
// Without jieba: returns original string unchanged

// Example: building a search query
const query = "用户旅行 planning";
const ftsQuery = buildFtsQuery(query);
// With jieba: `"用户" OR "旅行" OR "planning"`
// Fallback: extracts tokens via Unicode regex

Summary

• tokenizeForFts (lines 78-90 in MemoryCore/src/core/store/sqlite.ts): Indexes Chinese content using jieba's search-mode segmentation or returns raw text as fallback, preparing content for SQLite's unicode61 tokenizer.
• buildFtsQuery (lines 102-124 in MemoryCore/src/core/store/sqlite.ts): Constructs FTS5 MATCH strings through Chinese word segmentation (with stop-word filtering) or Unicode regex extraction for English content.
• Multilingual support: Handles mixed Chinese-English queries by combining jieba segmentation for CJK characters with Unicode word boundary detection for Latin scripts.
• Graceful degradation: Operates without the @node-rs/jieba dependency by relying on Unicode patterns and raw string storage.

Frequently Asked Questions

What happens if the @node-rs/jieba library is not installed?

If jieba is unavailable, tokenizeForFts returns the raw input string unchanged, allowing SQLite's native tokenizer to handle whitespace splitting. During query construction, buildFtsQuery falls back to the Unicode regex /[\p{L}\p{N}_]+/gu to extract word tokens from English and mixed-alphanumeric text.

How does the tokenizer handle mixed Chinese-English queries?

The pipeline processes the entire query string through the same segmentation logic. Chinese portions are split by jieba into constituent words while English segments are preserved as whole tokens or extracted via Unicode patterns. The resulting token sets are combined into a single OR-joined FTS5 MATCH expression that matches any relevant term across both languages.

What Chinese stop words are filtered during query construction?

The buildFtsQuery implementation filters tokens against the ZH_STOP_WORDS constant, which contains common Chinese particles and high-frequency words that provide little search value. The exact list includes functional words like "的", "了", and "是" that would otherwise dilute search precision.

Why does the indexer join tokens with spaces before storing?

The tokenizeForFts function joins segmented Chinese tokens with single spaces to create a delimited format compatible with SQLite's unicode61 tokenizer. This design allows the FTS5 engine to process both jieba-segmented Chinese text and natural English whitespace using the same underlying tokenizer configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}