# Character Set Filtering (字符集过滤) in Rime Wanxiang: Technical Implementation Guide

> **Rime Wanxiang's character set filtering feature restricts Chinese conversion candidates to a curated subset of 8,105 common characters by default, while allowing users to toggle between "small" (小字集) and "large" (大字集) sets vi...

- Repository: [amzxyz/rime_wanxiang](https://github.com/amzxyz/rime_wanxiang)
- Tags: how-to-guide
- Published: 2026-02-24

---

**Rime Wanxiang's character set filtering feature restricts Chinese conversion candidates to a curated subset of 8,105 common characters by default, while allowing users to toggle between "small" (小字集) and "large" (大字集) sets via Ctrl+G or fine-tune visibility through whitelist and blacklist rules defined in [`super_filter.lua`](https://github.com/amzxyz/rime_wanxiang/blob/main/super_filter.lua).**

The character set filtering mechanism in the amzxyz/rime_wanxiang repository provides granular control over which Unicode characters appear in the input method's candidate list. Implemented as a Lua filter that processes each candidate after the replacer step, this system combines a binary character database with configurable schema rules to dynamically hide or show specific glyphs based on user preferences and toggle states.

## How the Filter Architecture Works

### The Toggle Switch Definition

The filtering behavior is controlled by a schema switch named `charset_filter` defined in [`wanxiang.schema.yaml`](https://github.com/amzxyz/rime_wanxiang/blob/main/wanxiang.schema.yaml). This switch exposes two states: **大字集** (large/full set) and **小字集** (small/limited set). By default, the schema resets to state `0` (小字集), restricting output to common characters only. The switch binds to the keyboard shortcut `Control+g`, allowing real-time toggling during text input.

```yaml

# wanxiang.schema.yaml, lines 45-48

switches:
  - name: charset_filter
    states: [ 大字集, 小字集 ]
    reset: 0

```

### Lua Filter Registration and Initialization

The filter registers in the schema's `filters` section at line 98, ensuring [`super_filter.lua`](https://github.com/amzxyz/rime_wanxiang/blob/main/super_filter.lua) executes on every candidate after other processing steps. During initialization, the `init_charset_filter` function (lines 408-423) loads the binary database `lua/data/charset.reverse.bin`, which maps each Unicode code point to attribute strings like `"8105"` indicating membership in the common character set.

```lua
-- super_filter.lua, simplified from lines 408-423
local function init_charset_filter(env)
  local path = "lua/data/charset.reverse.bin"
  local file = io.open(path, "rb")
  -- Binary database loaded into env.charset_db
  -- Maps code points to attribute strings (e.g., "8105")
end

```

### Rule Parsing and Configuration Schema

Filter rules parse from the `charset` section of the schema (lines 59-90). Each rule specifies an `option` name linking it to a toggle switch, a `base` attribute string for set membership testing, and optional `addlist` (whitelist) and `blacklist` arrays containing explicit characters to force-show or force-hide.

```lua
-- super_filter.lua, lines 59-90 (conceptual)
env.filters = {
  {
    option = "charset_filter",  -- Links to switch name
    base = "8105",              -- Requires attribute "8105" in DB
    addlist = { "𰻝", "𰻞" },   -- Force show these
    blacklist = {}               -- Force hide these
  }
}

```

## Candidate Evaluation Logic

### The Precedence Hierarchy

For every candidate character, the `in_charset` function (lines 501-548) evaluates visibility through a strict three-tier precedence in `codepoint_in_charset`:

1. **Blacklist check** – If the character exists in the rule's `blacklist`, the filter immediately returns `false` (hide candidate).
2. **Addlist check** – If the character exists in the `addlist`, the filter returns `true` (show candidate) regardless of base attributes.
3. **Base attribute intersection** – The filter checks if the character's database attribute (e.g., `"8105"`) intersects with the rule's `base` string. If no intersection, the candidate hides.

When **no rules are active**, the filter passes all candidates through unchanged (line 617 returns `true`).

### The Toggle Mechanism Explained

The **小字集** (small set) mode activates the filter with `base = "8105"`, showing only characters whose database attributes contain that string. When the user presses **Ctrl+G** to switch to **大字集** (large set), the active rule's `base` attribute becomes empty. Since an empty base cannot intersect with any database attribute, the evaluation falls through. However, because the filter treats "no active matching rule" as *show all*, toggling to 大字集 effectively disables filtering and exposes the full Unicode repertoire.

## Practical Configuration Examples

### Enabling the Default Character Set Filter

Add the switch definition to your custom schema file to enable the default 8,105-character limitation with toggle support.

```yaml

# wanxiang.custom.yaml

switches:
  - name: charset_filter
    states: [ 大字集, 小字集 ]
    reset: 0  # Default to 小字集 (limited set)

    # Toggle with Control+g as defined in main schema

```

### Whitelisting Rare Characters with Addlist

Use the `addlist` array to force specific rare characters to appear even when the filter restricts the base set to common characters.

```yaml

# wanxiang.custom.yaml

charset:
  - option: charset_filter
    base: "8105"
    addlist:
      - "𰻝"  # Biángbiángmiàn character

      - "𰻞"
    blacklist: []

```

### Blacklisting Specific Characters

The `blacklist` array permanently hides designated characters regardless of toggle state or database attributes.

```yaml

# wanxiang.custom.yaml

charset:
  - option: charset_filter
    base: "8105"
    addlist: []
    blacklist:
      - "𰻞"  # This character will never appear in candidates

```

### Internal Filter Check (Lua Implementation)

The filter only processes single Chinese characters, passing through punctuation and non-Chinese text unchanged.

```lua
-- super_filter.lua, candidate evaluation logic (simplified)
local function in_charset(env, ctx, text)
  if not text or text == "" then return true end
  local cp = utf8.codes(text)()  -- Extract first code point
  
  -- Skip non-Chinese characters
  if not wanxiang.IsChineseCharacter(utf8.char(cp)) then 
    return true 
  end
  
  return codepoint_in_charset(env, ctx, cp, utf8.char(cp))
end

```

## Summary

- **Default Behavior**: Wanxiang restricts candidates to **8,105 common characters** (小字集) using the binary database at `lua/data/charset.reverse.bin`.
- **Quick Toggle**: Press **Ctrl+G** to switch between 小字集 (filtered) and 大字集 (unfiltered) modes; this toggles the `charset_filter` switch defined in [`wanxiang.schema.yaml`](https://github.com/amzxyz/rime_wanxiang/blob/main/wanxiang.schema.yaml).
- **Fine-Grained Control**: Use `addlist` (whitelist) and `blacklist` arrays in schema configuration to override the default set for specific Unicode characters.
- **Implementation**: All logic resides in [`lua/super_filter.lua`](https://github.com/amzxyz/rime_wanxiang/blob/main/lua/super_filter.lua), specifically in `init_charset_filter` (database loading) and `codepoint_in_charset` (evaluation logic), with rules parsing at lines 59-90 and candidate checking at lines 501-548.

## Frequently Asked Questions

### How do I switch between the small and large character sets?

Press **Ctrl+G** while composing text. This hotkey toggles the `charset_filter` switch between **大字集** (show all characters) and **小字集** (show only the 8,105 common characters). You can also toggle this through the Rime switcher menu if your frontend supports it.

### Can I display specific rare characters while keeping the small character set active?

Yes. Add the specific characters to an `addlist` array in your schema's `charset` configuration section. Characters listed in `addlist` bypass the base attribute check and always appear in candidates, even when the filter restricts the main set to common characters.

### Where does Wanxiang store the character classification data?

The filter references a binary database file located at `lua/data/charset.reverse.bin` in the repository. This file maps Unicode code points to attribute strings (such as `"8105"`). The `init_charset_filter` function in [`super_filter.lua`](https://github.com/amzxyz/rime_wanxiang/blob/main/super_filter.lua) loads this database into memory during filter initialization.

### What happens if I disable all character set filter rules?

When no `charset` rules are active or when the current switch state matches no configured rule, the filter returns `true` for all candidates (line 617 in [`super_filter.lua`](https://github.com/amzxyz/rime_wanxiang/blob/main/super_filter.lua)). This effectively disables filtering and exposes the entire available Unicode repertoire, equivalent to selecting the **大字集** mode.