Character Set Filtering (字符集过滤) in Rime Wanxiang: Technical Implementation Guide

Rime Wanxiang's character set filtering feature restricts Chinese conversion candidates to a curated subset of 8,105 common characters by default, while allowing users to toggle between "small" (小字集) and "large" (大字集) sets via Ctrl+G or fine-tune visibility through whitelist and blacklist rules defined in super_filter.lua.

The character set filtering mechanism in the amzxyz/rime_wanxiang repository provides granular control over which Unicode characters appear in the input method's candidate list. Implemented as a Lua filter that processes each candidate after the replacer step, this system combines a binary character database with configurable schema rules to dynamically hide or show specific glyphs based on user preferences and toggle states.

How the Filter Architecture Works

The Toggle Switch Definition

The filtering behavior is controlled by a schema switch named charset_filter defined in wanxiang.schema.yaml. This switch exposes two states: 大字集 (large/full set) and 小字集 (small/limited set). By default, the schema resets to state 0 (小字集), restricting output to common characters only. The switch binds to the keyboard shortcut Control+g, allowing real-time toggling during text input.

# wanxiang.schema.yaml, lines 45-48

switches:
  - name: charset_filter
    states: [ 大字集, 小字集 ]
    reset: 0

Lua Filter Registration and Initialization

The filter registers in the schema's filters section at line 98, ensuring super_filter.lua executes on every candidate after other processing steps. During initialization, the init_charset_filter function (lines 408-423) loads the binary database lua/data/charset.reverse.bin, which maps each Unicode code point to attribute strings like "8105" indicating membership in the common character set.

-- super_filter.lua, simplified from lines 408-423
local function init_charset_filter(env)
  local path = "lua/data/charset.reverse.bin"
  local file = io.open(path, "rb")
  -- Binary database loaded into env.charset_db
  -- Maps code points to attribute strings (e.g., "8105")
end

Rule Parsing and Configuration Schema

Filter rules parse from the charset section of the schema (lines 59-90). Each rule specifies an option name linking it to a toggle switch, a base attribute string for set membership testing, and optional addlist (whitelist) and blacklist arrays containing explicit characters to force-show or force-hide.

-- super_filter.lua, lines 59-90 (conceptual)
env.filters = {
  {
    option = "charset_filter",  -- Links to switch name
    base = "8105",              -- Requires attribute "8105" in DB
    addlist = { "𰻝", "𰻞" },   -- Force show these
    blacklist = {}               -- Force hide these
  }
}

Candidate Evaluation Logic

The Precedence Hierarchy

For every candidate character, the in_charset function (lines 501-548) evaluates visibility through a strict three-tier precedence in codepoint_in_charset:

1. Blacklist check – If the character exists in the rule's blacklist, the filter immediately returns false (hide candidate).
2. Addlist check – If the character exists in the addlist, the filter returns true (show candidate) regardless of base attributes.
3. Base attribute intersection – The filter checks if the character's database attribute (e.g., "8105") intersects with the rule's base string. If no intersection, the candidate hides.

When no rules are active, the filter passes all candidates through unchanged (line 617 returns true).

The Toggle Mechanism Explained

The 小字集 (small set) mode activates the filter with base = "8105", showing only characters whose database attributes contain that string. When the user presses Ctrl+G to switch to 大字集 (large set), the active rule's base attribute becomes empty. Since an empty base cannot intersect with any database attribute, the evaluation falls through. However, because the filter treats "no active matching rule" as show all, toggling to 大字集 effectively disables filtering and exposes the full Unicode repertoire.

Practical Configuration Examples

Enabling the Default Character Set Filter

Add the switch definition to your custom schema file to enable the default 8,105-character limitation with toggle support.

# wanxiang.custom.yaml

switches:
  - name: charset_filter
    states: [ 大字集, 小字集 ]
    reset: 0  # Default to 小字集 (limited set)

    # Toggle with Control+g as defined in main schema

Whitelisting Rare Characters with Addlist

Use the addlist array to force specific rare characters to appear even when the filter restricts the base set to common characters.

# wanxiang.custom.yaml

charset:
  - option: charset_filter
    base: "8105"
    addlist:
      - "𰻝"  # Biángbiángmiàn character

      - "𰻞"
    blacklist: []

Blacklisting Specific Characters

The blacklist array permanently hides designated characters regardless of toggle state or database attributes.

# wanxiang.custom.yaml

charset:
  - option: charset_filter
    base: "8105"
    addlist: []
    blacklist:
      - "𰻞"  # This character will never appear in candidates

Internal Filter Check (Lua Implementation)

The filter only processes single Chinese characters, passing through punctuation and non-Chinese text unchanged.

-- super_filter.lua, candidate evaluation logic (simplified)
local function in_charset(env, ctx, text)
  if not text or text == "" then return true end
  local cp = utf8.codes(text)()  -- Extract first code point
  
  -- Skip non-Chinese characters
  if not wanxiang.IsChineseCharacter(utf8.char(cp)) then 
    return true 
  end
  
  return codepoint_in_charset(env, ctx, cp, utf8.char(cp))
end

Summary

• Default Behavior: Wanxiang restricts candidates to 8,105 common characters (小字集) using the binary database at lua/data/charset.reverse.bin.
• Quick Toggle: Press Ctrl+G to switch between 小字集 (filtered) and 大字集 (unfiltered) modes; this toggles the charset_filter switch defined in wanxiang.schema.yaml.
• Fine-Grained Control: Use addlist (whitelist) and blacklist arrays in schema configuration to override the default set for specific Unicode characters.
• Implementation: All logic resides in lua/super_filter.lua, specifically in init_charset_filter (database loading) and codepoint_in_charset (evaluation logic), with rules parsing at lines 59-90 and candidate checking at lines 501-548.

Frequently Asked Questions

How do I switch between the small and large character sets?

Press Ctrl+G while composing text. This hotkey toggles the charset_filter switch between 大字集 (show all characters) and 小字集 (show only the 8,105 common characters). You can also toggle this through the Rime switcher menu if your frontend supports it.

Can I display specific rare characters while keeping the small character set active?

Yes. Add the specific characters to an addlist array in your schema's charset configuration section. Characters listed in addlist bypass the base attribute check and always appear in candidates, even when the filter restricts the main set to common characters.

Where does Wanxiang store the character classification data?

The filter references a binary database file located at lua/data/charset.reverse.bin in the repository. This file maps Unicode code points to attribute strings (such as "8105"). The init_charset_filter function in super_filter.lua loads this database into memory during filter initialization.

What happens if I disable all character set filter rules?

When no charset rules are active or when the current switch state matches no configured rule, the filter returns true for all candidates (line 617 in super_filter.lua). This effectively disables filtering and exposes the entire available Unicode repertoire, equivalent to selecting the 大字集 mode.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}