# Algorithmic Challenges for Large Datasets in Java: Solving TopK Unique Integers > Solve TopK unique integers challenges with large Java datasets using a min-heap and HashSet. Achieve O(N log K) time and O(K) space efficiency, processing billions of rows without full memory load. - Repository: [Doocs/advanced-java](https://github.com/doocs/advanced-java) - Tags: tutorial - Published: 2026-02-28 --- **Use a min-heap (`PriorityQueue`) combined with a `HashSet` to extract the top K unique integers from massive datasets in O(N log K) time and O(K) space, allowing you to process billions of rows without loading them entirely into memory.** The `doocs/advanced-java` repository provides battle-tested patterns for handling algorithmic challenges for large datasets in Java. When you need to find the top K unique integers from a stream that exceeds available memory, the min-heap approach documented in [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) offers the most reliable balance of speed and memory efficiency. ## Why Min-Heap is the Standard Solution for TopK Problems When facing algorithmic challenges for large datasets in Java, the **min-heap** (implemented as `java.util.PriorityQueue`) is the canonical choice for streaming TopK problems. The repository emphasizes this approach because it requires only a single pass through the data and bounds memory usage regardless of input size. **Core algorithm:** 1. Initialize a min-heap of size K. 2. For each incoming integer, insert it into the heap. 3. If the heap size exceeds K, remove the smallest element (`poll()`). 4. After processing all N items, the heap contains the K largest values. **Complexity characteristics:** - **Time:** `O(N log K)` — each insertion and removal costs `log K`. - **Space:** `O(K)` — only K elements are stored, making this suitable for terabyte-scale inputs. As noted in [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) (lines 46-48), Java's `PriorityQueue` provides exactly the building block needed for this pattern, contrasting with the C++ implementation shown elsewhere in the file. ## Java Implementation: Extracting Top K Unique Integers To solve the specific variant of finding **unique** integers, you must combine the min-heap with a `HashSet` to filter duplicates. The following implementation mirrors the logic described in the repository while providing production-ready Java code. ```java import java.io.*; import java.nio.file.*; import java.util.*; import java.util.stream.*; public class TopKUniqueIntegers { /** * Returns the K largest **unique** integers from the given stream. * * @param source a {@link IntStream} that may be huge (e.g., read from a file) * @param k how many top values to keep (k > 0) * @return a list containing the top K values in descending order */ public static List topKUnique(IntStream source, int k) { // Use a min‑heap of size K to retain the largest values. PriorityQueue minHeap = new PriorityQueue<>(k); // To guarantee uniqueness we keep a HashSet of the values already seen. Set seen = new HashSet<>(); source.forEach(value -> { if (seen.add(value)) { // first time we see this integer minHeap.offer(value); if (minHeap.size() > k) { int removed = minHeap.poll(); // discard the smallest // also remove it from the seen set – it’s no longer in the top‑K seen.remove(removed); } } }); // Extract heap content into a list sorted descending. List result = new ArrayList<>(minHeap); result.sort(Comparator.reverseOrder()); return result; } // -------------------------------------------------------------------- // Demo driver that reads ints from a text file (one per line) and prints // the top 10 unique numbers. // -------------------------------------------------------------------- public static void main(String[] args) throws IOException { if (args.length != 2) { System.err.println("Usage: java TopKUniqueIntegers "); System.exit(1); } Path path = Path.of(args[0]); int k = Integer.parseInt(args[1]); try (IntStream lines = Files.lines(path) .mapToInt(Integer::parseInt)) { List topK = topKUnique(lines, k); System.out.println("Top " + k + " unique integers:"); topK.forEach(System.out::println); } } } ``` **Key implementation details:** - **`PriorityQueue`** creates the min-heap that automatically orders elements ascending, allowing `O(log K)` insertion and removal. - **`HashSet`** enforces uniqueness; the `seen.add(value)` call returns `true` only for first occurrences, preventing duplicate entries from inflating the heap. - **Streaming support** via `IntStream` allows the method to process files, database cursors, or Kafka streams without materializing the entire dataset in RAM. - **Cleanup on eviction** removes evicted elements from `seen` to prevent memory leaks when the same integer reappears later in the stream. ## Alternative Algorithms for Large Dataset Challenges While the min-heap is the general-purpose winner for algorithmic challenges for large datasets in Java, the `doocs/advanced-java` repository documents several alternatives optimized for specific constraints. ### Quick-Select (Partial Quicksort) **Best for:** When data fits in memory and you need average-case linear time. **Approach:** Use the partition scheme from quicksort to recursively narrow down the K-th largest element. Once found, the elements to its left are your top K. **Trade-offs:** Requires `O(N)` average time but `O(N^2)` worst-case. Needs random access to the full dataset, making it unsuitable for true streaming scenarios. ### Bitmap (Bit Vector) **Best for:** Bounded integer domains (e.g., 0 to 10^9) with no duplicates or when duplicates are handled separately. **Approach:** Allocate a bit array where each bit represents the presence of an integer. Scan the bitmap from the highest index downward to collect K set bits. **Trade-offs:** Consumes memory proportional to the integer range, not the data size. Excellent for dense data but wasteful for sparse 64-bit integers. ### Hash-Bucket Counting **Best for:** Frequency-based TopK (e.g., finding the K most frequent items rather than largest values). **Approach:** Use a HashMap to count occurrences, then apply a min-heap or quick-select on the frequency buckets. **Trade-offs:** Requires storing all unique keys and counts. The `doocs/advanced-java` repository notes this is ideal when the cardinality is manageable but the stream volume is massive. ### Trie (Prefix Tree) **Best for:** String data or numbers with shared prefixes (e.g., IP addresses, URLs). **Approach:** Insert data into a trie and traverse to find the lexicographically or numerically largest K entries. **Trade-offs:** Higher memory overhead per node but supports efficient prefix-based queries and incremental updates. ### Hybrid Distributed Approach **Best for:** Multi-terabyte datasets spread across a cluster. **Approach:** Compute local TopK on each shard using min-heaps, then merge the partial results with a final heap or sort. **Trade-offs:** Adds network latency and coordination complexity but scales horizontally without bound. ## Key Files in the doocs/advanced-java Repository The repository organizes big-data algorithmic solutions under `docs/big-data/`. These files provide the theoretical foundation for the Java implementations discussed above. | File | Purpose | Direct Link | |------|---------|-------------| | [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) | Comprehensive guide to TopK algorithms including heap, quick-select, and bitmap methods. | [topk-problems-and-solutions.md](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) | | [`docs/big-data/find-no-repeat-number.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-no-repeat-number.md) | Techniques for identifying non-repeating numbers, useful for uniqueness validation in TopK. | [find-no-repeat-number.md](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-no-repeat-number.md) | | [`docs/big-data/find-rank-top-500-numbers.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-rank-top-500-numbers.md) | Specific implementation for ranking large sets, demonstrating scaling TopK to larger K values. | [find-rank-top-500-numbers.md](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-rank-top-500-numbers.md) | | [`docs/big-data/find-mid-value-in-500-millions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-mid-value-in-500-millions.md) | Median-finding algorithms for massive data, closely related to selection problems. | [find-mid-value-in-500-millions.md](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-mid-value-in-500-millions.md) | ## Summary - **Min-heap with HashSet** is the most robust solution for algorithmic challenges for large datasets in Java, providing `O(N log K)` time and `O(K)` space complexity. - The `doocs/advanced-java` repository documents this pattern in [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md), emphasizing streaming compatibility. - For bounded integer ranges, consider **bitmap** approaches to reduce memory overhead. - **Quick-select** offers better average-case performance when the entire dataset fits in memory and random access is available. - In distributed environments, compute **local TopK** on each shard using heaps, then merge partial results. ## Frequently Asked Questions ### What is the time complexity of finding TopK unique integers using a min-heap? The algorithm runs in **O(N log K)** time, where N is the total number of elements in the stream. Each insertion into the `PriorityQueue` costs `O(log K)`, and we perform at most N insertions and N removals. This is significantly more efficient than sorting the entire dataset, which would require **O(N log N)** time. ### How do I handle duplicate values when computing TopK in Java? To ensure uniqueness, maintain a **`HashSet`** alongside your min-heap. Before inserting a value into the `PriorityQueue`, check `seen.add(value)`. If it returns `true` (indicating the value is new), proceed with heap insertion. When the heap exceeds size K and you remove the smallest element, also remove that element from the `HashSet` to prevent memory leaks and allow the value to be reprocessed if it appears again later in the stream. ### When should I use a bitmap instead of a heap for large dataset problems? Use a **bitmap** when your integer domain is bounded and dense (e.g., values range from 0 to 10⁹ with few gaps) and you need to check existence or rank rather than maintain a dynamic top-K. Bitmaps consume memory proportional to the range of values (one bit per possible integer), making them unsuitable for sparse 64-bit integers or unbounded streams. For TopK problems with arbitrary integer ranges, the heap approach remains superior. ### Can the TopK algorithm be distributed across multiple machines? Yes. In distributed environments, implement a **hybrid approach**: compute local TopK results on each shard or node using min-heaps (`O(N/K log K)` per shard), then transmit these partial results to a central reducer. The reducer merges the incoming streams by maintaining a final min-heap of size K. This approach scales horizontally because each node processes only its local partition, and the network transfer is limited to K elements per shard rather than the full dataset.