How ConversionEngine Sorts Candidates by Word Frequency in hallelujahim

The ConversionEngine sorts candidate words by subtracting frequency values from a JSON dictionary, placing higher-frequency words first in the results array.

The dongyuwei/hallelujahim repository implements an intelligent input method engine that prioritizes dictionary suggestions based on real-world usage statistics. Understanding how the ConversionEngine sorts candidates by word frequency reveals the mechanics behind its predictive text accuracy and suggestion ranking system.

Understanding the ConversionEngine Sorting Pipeline

The sorting process operates across three distinct phases, moving from raw data loading to final candidate presentation.

Loading Frequency Data from JSON

When the engine initializes, it reads words_with_frequency_and_translation_and_ipa.json through the -getWordsWithFrequencyAndTranslation method in src/ConversionEngine.mm. Each JSON entry contains a "frequency" field representing corpus occurrence statistics. This data loads into memory as an NSDictionary where word strings map to their frequency metadata.

Collecting Raw Matches from the Trie

The -wordsStartsWith: method queries the MARISA trie structure for all dictionary entries sharing the user's typed prefix. This returns an NSMutableArray of plain NSString objects without any frequency information attached. At this stage, the candidates exist as unsorted raw matches from the lexical database.

Executing the Frequency-Based Sort

The -sortWordsByFrequency: method receives the raw match array and reorders it using the frequency table loaded during initialization. The implementation creates a comparator block that subtracts the longLongValue of the second word's frequency from the first word's frequency. If the result is positive, the first word (higher frequency) appears before the second. This produces a descending order where the most common words appear at the top of the candidate list.

Deep Dive into the sortWordsByFrequency: Implementation

The sorting logic resides in src/ConversionEngine.mm between lines 89-102. The method signature accepts an NSArray of word strings and returns a sorted NSArray.

// Simplified representation of the comparator logic from ConversionEngine.mm
- (NSArray *)sortWordsByFrequency:(NSArray *)words {
    return [words sortedArrayUsingComparator:^NSComparisonResult(NSString *word1, NSString *word2) {
        NSDictionary *dict1 = self.wordsWithFrequency[word1];
        NSDictionary *dict2 = self.wordsWithFrequency[word2];
        
        long long freq1 = [dict1[@"frequency"] longLongValue];
        long long freq2 = [dict2[@"frequency"] longLongValue];
        
        if (freq1 > freq2) return NSOrderedAscending;    // Higher frequency first
        if (freq1 < freq2) return NSOrderedDescending;
        return NSOrderedSame;
    }];
}

The -getCandidates: method invokes this sorting routine at lines 91-95, immediately after retrieving raw matches from the trie and before applying additional filtering or substitution logic.

Practical Code Examples

Sorting a Custom Word List

You can leverage the engine's frequency sorting for arbitrary word arrays:

NSArray *userWords = @[@"application", @"app", @"apple", @"apply"];
ConversionEngine *engine = [ConversionEngine sharedEngine];
NSArray *prioritized = [engine sortWordsByFrequency:userWords];

// Result: Words ordered by corpus frequency (most common first)
NSLog(@"%@", prioritized);

Full Candidate Generation Workflow

The complete pipeline from user input to sorted candidates:

NSString *input = @"techn";
ConversionEngine *engine = [ConversionEngine sharedEngine];

// Returns array sorted by frequency, with most common words first
NSArray *candidates = [engine getCandidates:input];

/* Internal steps:
   1. wordsStartsWith:@"techn" queries the MARISA trie
   2. sortWordsByFrequency: orders results by JSON frequency data
   3. Additional processing (pinyin, spell-check) appends to list */

Summary

• The ConversionEngine loads frequency statistics from words_with_frequency_and_translation_and_ipa.json during initialization via -getWordsWithFrequencyAndTranslation in src/ConversionEngine.mm.
• Raw candidate matches come from the MARISA trie through -wordsStartsWith:, returning unsorted strings.
• The -sortWordsByFrequency: method implements a comparator that subtracts frequency values, placing higher-frequency words earlier in the array (descending order).
• This sorting occurs at lines 89-102 of ConversionEngine.mm and is invoked from -getCandidates: before final candidate presentation.

Frequently Asked Questions

Where does ConversionEngine store word frequency data?

The engine references frequency statistics from words_with_frequency_and_translation_and_ipa.json, which the -getWordsWithFrequencyAndTranslation method loads into an NSDictionary during engine initialization. Each entry maps word strings to metadata dictionaries containing "frequency" keys with long long values representing corpus occurrence counts.

What sorting algorithm does sortWordsByFrequency: use?

The method utilizes NSArray's sortedArrayUsingComparator: with a custom block comparator. This implements a stable sort where the comparison logic subtracts the second word's frequency from the first word's frequency. Positive results place the first word before the second, creating a descending order without implementing a specific algorithm like quicksort or mergesort directly—the underlying Foundation framework handles the sorting mechanics.

How does ConversionEngine handle words with equal frequencies?

When the comparator calculates zero (equal frequencies), it returns NSOrderedSame. This preserves the original relative order of those items as returned by the MARISA trie query, maintaining stability in the sort. Words with identical frequency values appear consecutively in the final candidate list without additional prioritization between them.

Can I customize the frequency sorting behavior?

The current implementation in src/ConversionEngine.mm hardcodes the frequency-based comparator in -sortWordsByFrequency:. To modify sorting behavior—such as ascending order, alphabetical tie-breaking, or weighting by word length—you would need to subclass ConversionEngine or modify the comparator block at lines 89-102. The engine does not expose a delegate protocol or configuration flag for alternative sorting strategies in its current architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}