# How Hallelujah IM Compares Word Frequency Between Multiple Candidate Matches

> Discover how Hallelujah IM compares word frequency between candidate matches. This input method ranks words using a JSON dictionary and custom comparator for precise sorting.

- Repository: [dongyuwei/hallelujahim](https://github.com/dongyuwei/hallelujahim)
- Tags: internals
- Published: 2026-02-28

---

**The Hallelujah input method ranks candidates by looking up each word's frequency in a pre-loaded JSON dictionary and sorting them in descending order using a custom comparator in `sortWordsByFrequency:`.**

The Hallelujah input method (dongyuwei/hallelujahim) implements a deterministic two-stage pipeline to ensure the most commonly used words appear first when multiple candidates match the user's input buffer. The system leverages a static frequency table bundled with the application to perform numeric comparisons between candidate words in real time.

## Two-Stage Pipeline for Frequency-Based Ranking

### Stage 1: Loading the Frequency Dictionary

When the conversion engine initializes, it loads the bundled JSON file [`words_with_frequency_and_translation_and_ipa.json`](https://github.com/dongyuwei/hallelujahim/blob/main/words_with_frequency_and_translation_and_ipa.json) into memory. The engine stores this data in the `wordsWithFrequencyAndTranslation` property, which maps each word to a dictionary containing its `frequency` value and other metadata.

This occurs in `src/ConversionEngine.mm` during engine startup (lines 46-48), ensuring the frequency data is immediately available for all subsequent lookups without file I/O overhead during typing.

### Stage 2: Gathering and Sorting Candidates

For a given input buffer, the engine first queries the MARISA trie using `wordsStartsWith:` (lines 75-86) to retrieve all words matching the current prefix. If the trie returns results, the raw array passes directly to the sorting phase; otherwise, the system falls back to spell-checking mechanisms.

The method `sortWordsByFrequency:` receives this raw candidate array and performs the actual comparison logic implemented in `src/ConversionEngine.mm` (lines 89-101).

## The Frequency Comparison Algorithm

The sorting mechanism uses a custom comparator that subtracts frequency values to determine ordering. For any two candidate words, the engine retrieves their `frequency` fields from the pre-loaded dictionary and computes the difference:

```objc
int64_t n = [dict1[@"frequency"] longLongValue] -
            [dict2[@"frequency"] longLongValue];
if (n > 0)   return NSOrderedAscending;   // word1 higher freq
if (n < 0)   return NSOrderedDescending;  // word2 higher freq

```

**Key logic details:**
- **Higher frequency results in `NSOrderedAscending`** – When `freq1` exceeds `freq2`, the positive difference triggers `NSOrderedAscending`, placing the higher-frequency word earlier in the array.
- **64-bit integer precision** – The code uses `int64_t` and `longLongValue` to handle large frequency counts without overflow.
- **Stable secondary ordering** – If frequencies are equal (`n == 0`), the method returns `NSOrderedSame`, preserving the original trie retrieval order.

## Assembling the Final Candidate List

The `getCandidates:` method (lines 182-197) constructs the complete suggestion list by concatenating three distinct sources in priority order:

1. **User-defined substitutions** – Custom shortcuts take absolute precedence
2. **Frequency-sorted candidates** – The output from `sortWordsByFrequency:` providing the primary ranking
3. **Fallback suggestions** – Spell-check corrections or pinyin matches when primary candidates are exhausted

The UI layer in `src/InputController.mm` consumes this pre-ordered array directly, displaying candidates exactly as ranked by the engine without additional client-side sorting.

## Code Examples

### Getting Ranked Candidates for User Input

```objc
// buffer contains the current typed text
NSArray *candidates = [[ConversionEngine sharedEngine] getCandidates:buffer];
// candidates[0] represents the highest-frequency match

```

### Direct Frequency Sorting

```objc
NSArray *raw = @[@"hello", @"world", @"apple"];
NSArray *sorted = [[ConversionEngine sharedEngine] sortWordsByFrequency:raw];
// sorted array is ordered by descending frequency according to the JSON table

```

### Manual Frequency Lookup

```objc
NSDictionary *freqTable = [[ConversionEngine sharedEngine] wordsWithFrequencyAndTranslation];
int64_t helloFreq = [freqTable[@"hello"][@"frequency"] longLongValue];
int64_t worldFreq = [freqTable[@"world"][@"frequency"] longLongValue];

```

## Summary

- The engine loads frequency data from [`words_with_frequency_and_translation_and_ipa.json`](https://github.com/dongyuwei/hallelujahim/blob/main/words_with_frequency_and_translation_and_ipa.json) at startup into `wordsWithFrequencyAndTranslation`.
- Raw candidates come from the MARISA trie via `wordsStartsWith:`.
- `sortWordsByFrequency:` compares words by subtracting their frequency values, returning `NSOrderedAscending` for higher frequencies.
- Final candidate lists prioritize user substitutions first, then frequency-sorted matches, then fallbacks.
- The UI receives fully ordered arrays from `getCandidates:` without additional processing.

## Frequently Asked Questions

### How does the input method handle words with identical frequencies?

When two candidates share the same frequency value, the comparator returns `NSOrderedSame` (the `n == 0` case), which preserves their original relative order as returned by the MARISA trie. This ensures deterministic, stable sorting even for equally common words.

### Where is the word frequency data stored in the repository?

The frequency table resides in [`dictionary/words_with_frequency_and_translation_and_ipa.json`](https://github.com/dongyuwei/hallelujahim/blob/main/dictionary/words_with_frequency_and_translation_and_ipa.json). The `ConversionEngine` reads this file during initialization and caches it in memory to avoid disk access during active typing sessions.

### What happens if a candidate word is not found in the frequency table?

The `sortWordsByFrequency:` method looks up each word in `wordsWithFrequencyAndTranslation`. If a word lacks a frequency entry, the lookup returns `nil`, and the `longLongValue` conversion yields 0. Such words effectively receive zero frequency and sort to the bottom of the candidate list unless other ordering mechanisms (like user substitutions) override the ranking.

### Does the frequency comparison support real-time learning or user-specific frequency adjustments?

According to the source code in `src/ConversionEngine.mm`, the system uses a static JSON file bundled with the application. The ranking relies on pre-computed global frequencies rather than personal usage statistics, ensuring consistent behavior across sessions and users.