# How to Process Big Data Efficiently Using Java: 6 Battle-Tested Strategies

> Learn how to process big data efficiently using Java with 6 battle-tested strategies covering hash partitioning heap selection and external merge sort. Master large datasets.

- Repository: [Doocs/advanced-java](https://github.com/doocs/advanced-java)
- Tags: best-practices
- Published: 2026-02-28

---

**Processing massive datasets in Java requires hash partitioning to split data into memory-sized chunks, heap-based selection for Top-K queries, and external merge sort for full ordering—all implemented with standard `java.io` and `java.util` classes.**

When JVM heap constraints prevent loading gigabytes of data into memory, you must rely on external algorithms and streaming I/O. The `doocs/advanced-java` repository provides production-ready patterns for processing big data efficiently using Java, covering everything from frequency counting to external sorting without heavy frameworks.

## Hash Partitioning (Divide-and-Conquer)

When data volume far exceeds available heap—for example, processing a 1 GB file with only 1 MB of memory—**hash partitioning** is the foundational technique. As detailed in [`docs/big-data/find-top-100-words.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-top-100-words.md), the strategy involves splitting the input by hashing each record into *N* smaller files so each fits in memory.

Use `java.io.BufferedReader` to stream the input line-by-line and `java.io.FileWriter` to emit records into temporary partition files. For a given word `w`, compute `Math.abs(w.hashCode()) % PARTITIONS` to determine the target file. Once partitioned, each file can be processed independently with in-memory structures like `java.util.HashMap`.

## Heap-Based Top-K Selection

For finding the *k* largest or smallest items without sorting the entire dataset, the **small-top-heap** approach provides **O(n log k)** time and **O(k)** memory. According to [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md), maintain a min-heap (for largest-K) or max-heap (for smallest-K) using `java.util.PriorityQueue`.

When a new element exceeds the heap root, replace and re-heapify. This bounds memory usage to the size of the result set rather than the input size, making it ideal for hotspot detection in query logs or IP frequency analysis.

## External Merge Sort

When you need a fully ordered output—for example, generating a global ranking—and data cannot fit in memory, **external merge sort** executes in **O(n log n)** time with limited heap. The algorithm, referenced across the big-data documentation, uses `java.io.RandomAccessFile` or NIO `FileChannel` for I/O.

First, read chunk-sized blocks (e.g., 100 MB), sort each in-memory using `Collections.sort()`, and write sorted runs to disk. Then merge runs using a priority queue that streams the smallest element from each run, yielding a globally sorted stream without loading everything into RAM.

## Bitmap/BitSet for Integer Deduplication

When elements are integers within a known range and duplicates are irrelevant, `java.util.BitSet` offers **O(range/8)** memory usage. As shown in [`docs/big-data/count-different-phone-numbers.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/count-different-phone-numbers.md), set the bit at index *value* for each element. Scanning the BitSet yields ordered unique values with minimal overhead, perfect for counting distinct phone numbers or IDs.

## Trie Structures for String Aggregation

For frequent prefix queries on massive string sets—such as URLs or query strings—a **Trie (prefix tree)** aggregates counts per node. The repository explains in [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) how to build a custom `TrieNode` class, then walk the tree keeping only the highest-frequency children at each level using a `PriorityQueue` for node-level Top-K extraction.

## End-to-End Pattern: The Top-100 Word Frequency Pipeline

The canonical example from [`docs/big-data/find-top-100-words.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-top-100-words.md) demonstrates a complete five-phase workflow that combines these strategies to process big data efficiently using Java:

1. **Partition Phase** – Read the massive text file line-by-line with `BufferedReader`. For each word, compute `Math.abs(word.hashCode()) % PARTITIONS` and write into the corresponding temporary file.
2. **Local Aggregation Phase** – Load each partition file (now ≤ 1 MB) into a `HashMap<String, Integer>` to count frequencies.
3. **Local Top-K Extraction** – Maintain a per-partition min-heap (`PriorityQueue<Map.Entry<String,Integer>>`) of size 100 to keep the most frequent words for that partition.
4. **Global Merge Phase** – Merge the 100-element heaps from all partitions using another min-heap of size 100 to obtain the final global Top-100.
5. **Result Output** – Write the final list to a result file or stream it to the caller.

This pipeline runs in **O(N log k)** time where *N* is the total number of words, and uses **O(k + P·M)** memory, where *P* is the number of partitions and *M* is the per-partition memory limit.

## Production-Ready Code Examples

### Generic Heap-Based Top-K Implementation

```java
import java.util.*;

public class TopK<T extends Comparable<T>> {
    private final int k;
    private final PriorityQueue<T> minHeap;

    public TopK(int k) {
        this.k = k;
        this.minHeap = new PriorityQueue<>(k);
    }

    public void add(T value) {
        if (minHeap.size() < k) {
            minHeap.offer(value);
        } else if (value.compareTo(minHeap.peek()) > 0) {
            minHeap.poll();
            minHeap.offer(value);
        }
    }

    public List<T> getTopK() {
        List<T> result = new ArrayList<>(minHeap);
        Collections.sort(result, Collections.reverseOrder());
        return result;
    }
}

```

*Usage for integers:*

```java
TopK<Integer> top10 = new TopK<>(10);
int[] data = {4,1,5,8,7,2,3,0,6,9};
for (int v : data) top10.add(v);
System.out.println(top10.getTopK()); // [9,8,7,6,5,4,3,2,1,0]

```

### Partition-Then-Count for Top-100 Words

```java
import java.io.*;
import java.nio.file.*;
import java.util.*;

public class TopWords {
    private static final int PARTITIONS = 5_000;
    private static final Path TMP_DIR = Paths.get("tmp-partitions");

    public static void main(String[] args) throws IOException {
        Path input = Paths.get("big.txt");          // 1 GB file
        partition(input);
        List<Map.Entry<String,Integer>> globalTop = mergePartitions();
        globalTop.forEach(e -> System.out.println(e.getKey() + ": " + e.getValue()));
    }

    private static void partition(Path src) throws IOException {
        Files.createDirectories(TMP_DIR);
        BufferedWriter[] writers = new BufferedWriter[PARTITIONS];
        for (int i = 0; i < PARTITIONS; i++) {
            writers[i] = Files.newBufferedWriter(TMP_DIR.resolve("part-" + i + ".txt"));
        }

        try (BufferedReader br = Files.newBufferedReader(src)) {
            String line;
            while ((line = br.readLine()) != null) {
                int idx = Math.abs(line.hashCode()) % PARTITIONS;
                writers[idx].write(line);
                writers[idx].newLine();
            }
        }

        for (BufferedWriter w : writers) w.close();
    }

    private static List<Map.Entry<String,Integer>> mergePartitions() throws IOException {
        PriorityQueue<Map.Entry<String,Integer>> globalHeap =
            new PriorityQueue<>(100, Comparator.comparingInt(Map.Entry::getValue));

        for (int i = 0; i < PARTITIONS; i++) {
            Path part = TMP_DIR.resolve("part-" + i + ".txt");
            if (!Files.exists(part)) continue;

            Map<String,Integer> counter = new HashMap<>();
            try (BufferedReader br = Files.newBufferedReader(part)) {
                String word;
                while ((word = br.readLine()) != null) {
                    counter.merge(word, 1, Integer::sum);
                }
            }

            for (Map.Entry<String,Integer> e : counter.entrySet()) {
                if (globalHeap.size() < 100) {
                    globalHeap.offer(e);
                } else if (e.getValue() > Objects.requireNonNull(globalHeap.peek()).getValue()) {
                    globalHeap.poll();
                    globalHeap.offer(e);
                }
            }
        }

        List<Map.Entry<String,Integer>> result = new ArrayList<>(globalHeap);
        result.sort((a,b) -> Integer.compare(b.getValue(), a.getValue()));
        return result;
    }
}

```

### External Merge Sort Skeleton

```java
import java.io.*;
import java.nio.file.*;
import java.util.*;

public class ExternalSort {
    private static final long CHUNK_SIZE = 100 * 1024 * 1024; // 100 MB

    public static void sort(Path source, Path target) throws IOException {
        List<Path> runs = createSortedRuns(source);
        mergeRuns(runs, target);
        for (Path p : runs) Files.deleteIfExists(p);
    }

    private static List<Path> createSortedRuns(Path src) throws IOException {
        List<Path> runs = new ArrayList<>();
        try (BufferedReader br = Files.newBufferedReader(src)) {
            List<String> buffer = new ArrayList<>();
            String line;
            long currentSize = 0;
            while ((line = br.readLine()) != null) {
                buffer.add(line);
                currentSize += line.length() + System.lineSeparator().length();
                if (currentSize >= CHUNK_SIZE) {
                    runs.add(writeRun(buffer));
                    buffer.clear();
                    currentSize = 0;
                }
            }
            if (!buffer.isEmpty()) runs.add(writeRun(buffer));
        }
        return runs;
    }

    private static Path writeRun(List<String> data) throws IOException {
        data.sort(Comparator.naturalOrder());
        Path run = Files.createTempFile("run", ".txt");
        try (BufferedWriter bw = Files.newBufferedWriter(run)) {
            for (String s : data) bw.write(s + System.lineSeparator());
        }
        return run;
    }

    private static void mergeRuns(List<Path> runs, Path target) throws IOException {
        PriorityQueue<RunReader> pq = new PriorityQueue<>(Comparator.comparing(RunReader::peek));
        for (Path p : runs) {
            RunReader rr = new RunReader(p);
            if (!rr.isEmpty()) pq.offer(rr);
        }

        try (BufferedWriter bw = Files.newBufferedWriter(target)) {
            while (!pq.isEmpty()) {
                RunReader rr = pq.poll();
                String smallest = rr.pop();
                bw.write(smallest);
                bw.newLine();
                if (!rr.isEmpty()) pq.offer(rr);
            }
        }

        for (RunReader rr : pq) rr.close();
    }

    private static class RunReader implements Closeable {
        private final BufferedReader br;
        private String cache;

        RunReader(Path p) throws IOException {
            this.br = Files.newBufferedReader(p);
            this.cache = br.readLine();
        }

        String peek() { return cache; }

        String pop() throws IOException {
            String cur = cache;
            cache = br.readLine();
            return cur;
        }

        boolean isEmpty() { return cache == null; }

        public void close() throws IOException { br.close(); }
    }
}

```

## Key Implementation Files in doocs/advanced-java

- **[`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md)** – Explains heap-based Top-K, bitmap, trie, and hybrid strategies.
- **[`docs/big-data/find-top-100-words.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-top-100-words.md)** – Detailed partition-hash plus heap solution for the classic interview problem.
- **[`docs/big-data/sort-the-query-strings-by-counts.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/sort-the-query-strings-by-counts.md)** – External sort concepts applied to massive query-string logs.
- **[`docs/big-data/find-top-1-ip.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-top-1-ip.md)** – Minimal heap plus HashMap example for single-value frequency analysis.
- **[`docs/big-data/find-hotest-query-string.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-hotest-query-string.md)** – Streaming detection of query hotspots using priority queues.
- **[`docs/big-data/find-common-urls.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-common-urls.md)** – Hash-partition plus local counting for frequently accessed URLs.
- **[`docs/big-data/count-different-phone-numbers.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/count-different-phone-numbers.md)** – BitSet implementation for large-range integer deduplication.
- **[`Main.java`](https://github.com/doocs/advanced-java/blob/main/Main.java)** – Minimal entry point adaptable to run any of the above snippets.

## Summary

- **Hash partitioning** splits massive datasets into memory-sized chunks using `BufferedReader` and `FileWriter`, enabling parallel local processing.
- **Heap-based selection** with `PriorityQueue` achieves **O(n log k)** time for Top-K problems while bounding memory to **O(k)**.
- **External merge sort** produces fully ordered outputs using multi-way merging with limited heap, avoiding full dataset materialization.
- **BitSet** provides **O(range/8)** memory for integer deduplication when value ranges are bounded.
- **Trie structures** efficiently aggregate string frequencies for prefix-based Top-K queries.
- The five-phase pipeline (partition, aggregate, local Top-K, global merge, output) forms a reusable pattern for production big data processing in Java.

## Frequently Asked Questions

### How does the hash partitioning strategy handle data skew?

If certain hash buckets receive disproportionate traffic, the partition files may still exceed memory limits. As implemented in [`docs/big-data/find-top-100-words.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/find-top-100-words.md), you can detect oversized partitions during the write phase and recursively repartition them using a different hash function or a larger modulus until each file fits within your memory constraints.

### Why use a min-heap instead of sorting the entire dataset for Top-K?

Sorting the entire dataset requires **O(n)** memory and **O(n log n)** time. The min-heap approach in [`docs/big-data/topk-problems-and-solutions.md`](https://github.com/doocs/advanced-java/blob/main/docs/big-data/topk-problems-and-solutions.md) reduces this to **O(k)** memory and **O(n log k)** time by maintaining only the current best candidates. When *k* is small relative to *n* (e.g., Top-100 from billions of records), this difference prevents OutOfMemoryError while significantly improving throughput.

### Can these algorithms leverage Java Streams for parallel processing?

Yes, for CPU-bound phases you can combine `Files.lines(Path)` with `.parallel()` to distribute work across `ForkJoinPool` threads. However, as noted in the repository's concurrency sections, the I/O-bound partitioning phase should remain single-threaded or use asynchronous NIO `FileChannel` operations to avoid thread contention on disk writes. The heap-based merge phases are inherently sequential but execute quickly due to the small *k* size.

### What is the optimal chunk size for external merge sort?

The chunk size in `ExternalSort` should match your available heap minus overhead for the JVM and other structures. The example uses 100 MB chunks, but you should tune this based on `-Xmx` settings and observed GC behavior. Larger chunks reduce the number of runs and merge passes, but too large will trigger GC thrashing or OOM errors during the in-memory sort phase.