# How the Wiki Custom Tokenizer Handles English and Chinese Text in TencentDB-Agent-Memory

> Discover how the Wiki custom tokenizer in TencentDB-Agent-Memory processes English and Chinese text using a dual-stage pipeline for efficient indexing and query construction.

- Repository: [Tencent Cloud/TencentDB-Agent-Memory](https://github.com/TencentCloud/TencentDB-Agent-Memory)
- Tags: deep-dive
- Published: 2026-08-27

---

**The Wiki component processes multilingual content through a dual-stage pipeline—leveraging @node-rs/jieba for Chinese word segmentation during indexing (`tokenizeForFts`) while extracting English tokens via Unicode patterns during query construction (`buildFtsQuery`).**

The TencentDB-Agent-Memory repository implements a sophisticated full-text search system for Wiki documents using SQLite FTS5. Understanding how the Wiki custom tokenizer handles English and Chinese text reveals the architecture behind its multilingual search capabilities and ensures optimal indexing for mixed-language datasets.

## Indexing Stage: `tokenizeForFts`

The indexing pipeline converts raw Wiki content into searchable tokens stored in the FTS5 `content` column. According to the source code in [`MemoryCore/src/core/store/sqlite.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/sqlite.ts) (lines 78-90), the `tokenizeForFts` function implements language-aware segmentation strategies.

### Chinese Segmentation with Jieba

When the optional **@node-rs/jieba** library is available, the tokenizer invokes `jieba.cutForSearch(raw, true)`. This search-engine mode segments Chinese sentences into both full words and their constituent sub-words. For example, the term "人工智能" generates three tokens: "人工", "智能", and "人工智能". This approach ensures matches occur whether users search for partial terms or complete phrases.

### Fallback Behavior and SQLite Integration

If the jieba library cannot be loaded, the system returns the raw string unchanged. The resulting token list is joined with single spaces and stored in the FTS5 table, allowing SQLite's built-in `unicode61` tokenizer to further split content on whitespace and punctuation boundaries.

## Query Construction Stage: `buildFtsQuery`

The search pipeline processes user queries through the `buildFtsQuery` function (lines 102-124 in [`MemoryCore/src/core/store/sqlite.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/sqlite.ts)), transforming natural language input into valid FTS5 MATCH expressions.

### Chinese Query Processing

For Chinese text, the tokenizer applies the same `cutForSearch` method used during indexing. It then filters the results to remove empty tokens, pure punctuation characters, and entries matching the `ZH_STOP_WORDS` list. Duplicate tokens are removed before being quoted and OR-joined into the final query string.

### English and Unicode Token Extraction

When jieba is unavailable, the tokenizer falls back to the Unicode regex `/[\p{L}\p{N}_]+/gu` to extract sequences of letters, numbers, or underscores. This pattern captures English words, numeric identifiers, and mixed alphanumeric tokens (such as "TypeScript" or "API_v2") as discrete units suitable for full-text matching.

### FTS5 MATCH String Generation

Processed tokens are formatted as quoted strings and combined with the `OR` operator to construct the final MATCH expression. A query like "用户旅行 planning" becomes `"用户" OR "旅行" OR "planning"` when jieba is present.

## Implementation Examples

```typescript
// Example: indexing a Wiki page
const rawContent = "用户五月去日本旅行 and planning a trip to Tokyo.";
const indexed = tokenizeForFts(rawContent);
// With jieba: "用户 五月 去 日本 旅行 and planning a trip to Tokyo."
// Without jieba: returns original string unchanged

// Example: building a search query
const query = "用户旅行 planning";
const ftsQuery = buildFtsQuery(query);
// With jieba: `"用户" OR "旅行" OR "planning"`
// Fallback: extracts tokens via Unicode regex

```

## Summary

- **`tokenizeForFts`** (lines 78-90 in [`MemoryCore/src/core/store/sqlite.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/sqlite.ts)): Indexes Chinese content using jieba's search-mode segmentation or returns raw text as fallback, preparing content for SQLite's `unicode61` tokenizer.
- **`buildFtsQuery`** (lines 102-124 in [`MemoryCore/src/core/store/sqlite.ts`](https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/main/MemoryCore/src/core/store/sqlite.ts)): Constructs FTS5 MATCH strings through Chinese word segmentation (with stop-word filtering) or Unicode regex extraction for English content.
- **Multilingual support**: Handles mixed Chinese-English queries by combining jieba segmentation for CJK characters with Unicode word boundary detection for Latin scripts.
- **Graceful degradation**: Operates without the @node-rs/jieba dependency by relying on Unicode patterns and raw string storage.

## Frequently Asked Questions

### What happens if the @node-rs/jieba library is not installed?

If jieba is unavailable, `tokenizeForFts` returns the raw input string unchanged, allowing SQLite's native tokenizer to handle whitespace splitting. During query construction, `buildFtsQuery` falls back to the Unicode regex `/[\p{L}\p{N}_]+/gu` to extract word tokens from English and mixed-alphanumeric text.

### How does the tokenizer handle mixed Chinese-English queries?

The pipeline processes the entire query string through the same segmentation logic. Chinese portions are split by jieba into constituent words while English segments are preserved as whole tokens or extracted via Unicode patterns. The resulting token sets are combined into a single OR-joined FTS5 MATCH expression that matches any relevant term across both languages.

### What Chinese stop words are filtered during query construction?

The `buildFtsQuery` implementation filters tokens against the `ZH_STOP_WORDS` constant, which contains common Chinese particles and high-frequency words that provide little search value. The exact list includes functional words like "的", "了", and "是" that would otherwise dilute search precision.

### Why does the indexer join tokens with spaces before storing?

The `tokenizeForFts` function joins segmented Chinese tokens with single spaces to create a delimited format compatible with SQLite's `unicode61` tokenizer. This design allows the FTS5 engine to process both jieba-segmented Chinese text and natural English whitespace using the same underlying tokenizer configuration.