How Hallelujah IM Compares Word Frequency Between Multiple Candidate Matches

The Hallelujah input method ranks candidates by looking up each word's frequency in a pre-loaded JSON dictionary and sorting them in descending order using a custom comparator in sortWordsByFrequency:.

The Hallelujah input method (dongyuwei/hallelujahim) implements a deterministic two-stage pipeline to ensure the most commonly used words appear first when multiple candidates match the user's input buffer. The system leverages a static frequency table bundled with the application to perform numeric comparisons between candidate words in real time.

Two-Stage Pipeline for Frequency-Based Ranking

Stage 1: Loading the Frequency Dictionary

When the conversion engine initializes, it loads the bundled JSON file words_with_frequency_and_translation_and_ipa.json into memory. The engine stores this data in the wordsWithFrequencyAndTranslation property, which maps each word to a dictionary containing its frequency value and other metadata.

This occurs in src/ConversionEngine.mm during engine startup (lines 46-48), ensuring the frequency data is immediately available for all subsequent lookups without file I/O overhead during typing.

Stage 2: Gathering and Sorting Candidates

For a given input buffer, the engine first queries the MARISA trie using wordsStartsWith: (lines 75-86) to retrieve all words matching the current prefix. If the trie returns results, the raw array passes directly to the sorting phase; otherwise, the system falls back to spell-checking mechanisms.

The method sortWordsByFrequency: receives this raw candidate array and performs the actual comparison logic implemented in src/ConversionEngine.mm (lines 89-101).

The Frequency Comparison Algorithm

The sorting mechanism uses a custom comparator that subtracts frequency values to determine ordering. For any two candidate words, the engine retrieves their frequency fields from the pre-loaded dictionary and computes the difference:

int64_t n = [dict1[@"frequency"] longLongValue] -
            [dict2[@"frequency"] longLongValue];
if (n > 0)   return NSOrderedAscending;   // word1 higher freq
if (n < 0)   return NSOrderedDescending;  // word2 higher freq

Key logic details:

• Higher frequency results in NSOrderedAscending – When freq1 exceeds freq2, the positive difference triggers NSOrderedAscending, placing the higher-frequency word earlier in the array.
• 64-bit integer precision – The code uses int64_t and longLongValue to handle large frequency counts without overflow.
• Stable secondary ordering – If frequencies are equal (n == 0), the method returns NSOrderedSame, preserving the original trie retrieval order.

Assembling the Final Candidate List

The getCandidates: method (lines 182-197) constructs the complete suggestion list by concatenating three distinct sources in priority order:

1. User-defined substitutions – Custom shortcuts take absolute precedence
2. Frequency-sorted candidates – The output from sortWordsByFrequency: providing the primary ranking
3. Fallback suggestions – Spell-check corrections or pinyin matches when primary candidates are exhausted

The UI layer in src/InputController.mm consumes this pre-ordered array directly, displaying candidates exactly as ranked by the engine without additional client-side sorting.

Code Examples

Getting Ranked Candidates for User Input

// buffer contains the current typed text
NSArray *candidates = [[ConversionEngine sharedEngine] getCandidates:buffer];
// candidates[0] represents the highest-frequency match

Direct Frequency Sorting

NSArray *raw = @[@"hello", @"world", @"apple"];
NSArray *sorted = [[ConversionEngine sharedEngine] sortWordsByFrequency:raw];
// sorted array is ordered by descending frequency according to the JSON table

Manual Frequency Lookup

NSDictionary *freqTable = [[ConversionEngine sharedEngine] wordsWithFrequencyAndTranslation];
int64_t helloFreq = [freqTable[@"hello"][@"frequency"] longLongValue];
int64_t worldFreq = [freqTable[@"world"][@"frequency"] longLongValue];

Summary

• The engine loads frequency data from words_with_frequency_and_translation_and_ipa.json at startup into wordsWithFrequencyAndTranslation.
• Raw candidates come from the MARISA trie via wordsStartsWith:.
• sortWordsByFrequency: compares words by subtracting their frequency values, returning NSOrderedAscending for higher frequencies.
• Final candidate lists prioritize user substitutions first, then frequency-sorted matches, then fallbacks.
• The UI receives fully ordered arrays from getCandidates: without additional processing.

Frequently Asked Questions

How does the input method handle words with identical frequencies?

When two candidates share the same frequency value, the comparator returns NSOrderedSame (the n == 0 case), which preserves their original relative order as returned by the MARISA trie. This ensures deterministic, stable sorting even for equally common words.

Where is the word frequency data stored in the repository?

The frequency table resides in dictionary/words_with_frequency_and_translation_and_ipa.json. The ConversionEngine reads this file during initialization and caches it in memory to avoid disk access during active typing sessions.

What happens if a candidate word is not found in the frequency table?

The sortWordsByFrequency: method looks up each word in wordsWithFrequencyAndTranslation. If a word lacks a frequency entry, the lookup returns nil, and the longLongValue conversion yields 0. Such words effectively receive zero frequency and sort to the bottom of the candidate list unless other ordering mechanisms (like user substitutions) override the ranking.

Does the frequency comparison support real-time learning or user-specific frequency adjustments?

According to the source code in src/ConversionEngine.mm, the system uses a static JSON file bundled with the application. The ranking relies on pre-computed global frequencies rather than personal usage statistics, ensuring consistent behavior across sessions and users.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}