How ConversionEngine Implements Offline Dictionary Lookup Using marisa-trie

The ConversionEngine uses a static marisa-trie binary loaded at startup and queries it via marisa::Agent's predictive_search to provide instant offline Chinese word lookups without network dependencies.

The hallelujahim open-source input method engine stores its entire dictionary locally to ensure privacy and responsiveness. At its core, the ConversionEngine leverages the marisa-trie library—a compact static trie structure—to compress roughly 227,000 Chinese words into a memory-efficient format that supports fast prefix matching.

Loading the Binary marisa-trie at Startup

The ConversionEngine initializes a global marisa::Trie instance declared in src/ConversionEngine.mm at line 12. During the loadPreparedData sequence, the loadTrie method mounts a pre-built binary file into memory:

- (void)loadTrie {
    NSString *path = [[NSBundle mainBundle] pathForResource:@"google_227800_words" ofType:@"bin"];
    const char *path2 = [path cStringUsingEncoding:[NSString defaultCStringEncoding]];
    trie.load(path2);
}

This operation executes once during application launch. The google_227800_words.bin file contains approximately 227,800 dictionary entries in a highly compressed static trie format. The binary is generated from the plain-text source dictionary/google_227800_words.txt using the build script dictionary/build-binary-trie.sh, which invokes marisa-builder to create the immutable trie structure.

Performing Prefix Queries with marisa::Agent

When the user types a prefix, the ConversionEngine's wordsStartsWith: method creates a marisa::Agent to traverse the trie. The agent holds iterator state while walking nodes that match the input buffer:

marisa::Agent agent;
const char *query = [buffer cStringUsingEncoding:[NSString defaultCStringEncoding]];
agent.set_query(query);

while (trie.predictive_search(agent)) {
    const marisa::Key key = agent.key();
    NSString *word = [[NSString alloc] initWithBytes:key.ptr() length:key.length() encoding:NSASCIIStringEncoding];
    [filtered addObject:word];
}

The trie.predictive_search(agent) call, implemented in src/ConversionEngine.mm between lines 75 and 86, enumerates all keys that start with the given prefix in O(number of matches) time. Each retrieved marisa::Key is converted back to an NSString and collected into the filtered array for further processing.

Integrating and Ranking Lookup Results

Raw trie results flow into the getCandidates: method (lines 91-98 of src/ConversionEngine.mm), where they undergo additional refinement. The engine sorts the retrieved words by frequency using sortWordsByFrequency: and merges them with suggestions from auxiliary sources such as the spell-checker and phonex modules.

This architecture ensures that the marisa-trie serves as the primary offline data source, while higher-level logic handles ranking and context-specific filtering. Because the entire dictionary resides in the application bundle as a static binary, the lookup process requires zero network connectivity.

Summary

• Static binary storage: The ConversionEngine loads google_227800_words.bin via marisa::Trie::load() at startup, keeping the entire 227,000-word dictionary in memory.
• Fast prefix enumeration: The predictive_search method of marisa::Agent walks the trie to find all keys matching a user-supplied prefix with minimal overhead.
• Complete offline operation: All dictionary data is packaged inside the app bundle; no external requests are made during lookup or result integration.
• Build pipeline: The dictionary/build-binary-trie.sh script compiles plain-text word lists into the compact marisa-trie binary format used by the engine.

Frequently Asked Questions

How does marisa-trie enable offline dictionary lookups?

marisa-trie is a static trie library that compresses string sets into immutable binary files. The ConversionEngine loads this binary into a global marisa::Trie object at startup, allowing the app to perform prefix searches entirely from memory without internet access, as all 227,000 words are embedded in the google_227800_words.bin file within the app bundle.

What is the performance characteristic of the predictive_search method?

The predictive_search operation traverses only the nodes necessary to enumerate matches, operating in time proportional to the number of results rather than the total dictionary size. This makes prefix lookups extremely fast even on the full 227,800-word dataset stored in src/ConversionEngine.mm.

How is the binary dictionary file generated?

The repository includes dictionary/build-binary-trie.sh, a shell script that processes dictionary/google_227800_words.txt using the marisa-builder tool. This converts the plain-text word list—containing one word per line—into the compact binary format that the ConversionEngine's loadTrie method reads during initialization.

Can the dictionary be modified at runtime?

No. marisa-trie is designed for static, read-only datasets. The trie is loaded once via trie.load() and remains immutable throughout the application lifecycle. To update the dictionary, you must rebuild the binary using the provided shell script and redistribute the application with the new google_227800_words.bin file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}