# Spark Coalesce vs Repartition: Understanding Data Shuffling Differences

> Understand Spark coalesce vs repartition for data shuffling. Coalesce merges partitions locally avoiding shuffles, while repartition triggers a full shuffle for even distribution. Learn the key differences.

- Repository: [DataTalksClub/data-engineering-zoomcamp](https://github.com/DataTalksClub/data-engineering-zoomcamp)
- Tags: deep-dive
- Published: 2026-05-31

---

**Spark `coalesce()` avoids shuffles when reducing partition counts by merging existing partitions locally, while `repartition()` always triggers a full shuffle to redistribute data evenly across the cluster.**

When optimizing Apache Spark DataFrame operations, choosing between `coalesce()` and `repartition()` determines whether your pipeline triggers expensive shuffle operations. The DataTalksClub data-engineering-zoomcamp repository demonstrates distinct patterns for using `DataFrame.coalesce()` and `DataFrame.repartition()` to optimize everything from file output sizes to join performance.

## Core Architectural Differences

### Shuffle Behavior and Data Movement

The fundamental distinction lies in how each method handles data redistribution across the cluster. **`coalesce(numPartitions)`** avoids shuffling data when the target number is **less than** the current partition count. Spark simply collapses existing partitions on the same executors, minimizing network I/O. Conversely, **`repartition(numPartitions)`** always triggers a shuffle operation because it computes a hash of each row to distribute data randomly across all requested partitions, regardless of whether the target count is higher or lower than the current setting.

According to the DataTalksClub source code, when `numPartitions` exceeds the current count, `coalesce` actually falls back to `repartition` behavior and performs a shuffle.

### Partition Size Distribution

Since `coalesce` only merges existing partitions without redistributing individual rows, it often produces **unevenly sized partitions**. This occurs when original partitions contain different data volumes. In contrast, `repartition` creates **roughly equal-sized partitions** because the hash-based redistribution scatters rows randomly, balancing the workload across executors for downstream operations.

## Performance Characteristics

**Coalesce** executes faster when shrinking partition counts because it eliminates the costly shuffle stage. This makes it ideal for write operations where you want fewer output files without the overhead of a full data redistribution.

**Repartition** incurs higher latency due to the mandatory shuffle, but delivers superior parallelism for compute-intensive operations like aggregations and joins. The DataTalksClub curriculum uses this method specifically to prevent data skew before heavy transformations.

## Implementation Examples from the DataTalksClub Repository

### Write Optimization with Coalesce

In [`06-batch/code/06_spark_sql.py`](https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/06-batch/code/06_spark_sql.py) at line 106, the codebase demonstrates using `coalesce(1)` to write a single CSV file without triggering a shuffle:

```python
df_result = spark.sql("SELECT * FROM taxi_data")
df_result.coalesce(1).write.mode("overwrite").csv("gs://bucket/output")

```

This pattern reduces the default 200 partitions to a single output file while keeping data on existing executors, avoiding network overhead entirely.

### Computation Optimization with Repartition

For operations requiring balanced data distribution, the notebooks consistently use `repartition()` before joins and aggregations. In `06-batch/code/homework.ipynb` at line 113, the code increases parallelism before processing:

```python
df = df.repartition(24)

```

Similarly, `06-batch/code/07_groupby_join.ipynb` uses `repartition(20)` at lines 102 and 156 before group-by operations to ensure uniform partition sizing:

```python
df_repartitioned = df.repartition(20)
df_repartitioned.groupBy("column").agg(sum("value")).show()

```

The curriculum reinforces this pattern across multiple files. `06-batch/code/05_taxi_schema.ipynb` contains `repartition(4)` calls at lines 307, 464, 682, and 836, while `06-batch/code/04_pyspark.ipynb` demonstrates `repartition(24)` at line 268 for maintaining optimal parallelism.

## Summary

- **Use `coalesce()`** when reducing partition counts for file output operations to avoid shuffle overhead and minimize network I/O.
- **Use `repartition()`** when you need evenly distributed data across partitions, particularly before joins, aggregations, or window functions to prevent data skew.
- **`coalesce()`** may produce uneven partition sizes because it only merges adjacent partitions without redistributing individual rows.
- **`repartition()`** always triggers a shuffle but creates roughly equal-sized partitions through hash-based redistribution.
- When `coalesce` receives a `numPartitions` value greater than the current count, it behaves exactly like `repartition`.

## Frequently Asked Questions

### Does coalesce always avoid shuffling in Spark?

No. **`coalesce()`** only avoids shuffling when the target number of partitions is less than the current count. If you attempt to increase partitions using `coalesce()`, Spark automatically falls back to `repartition()` behavior and performs a full shuffle.

### When should I use repartition instead of coalesce?

Use **`repartition()`** when you need to balance data distribution across partitions before compute-intensive operations like joins, groupBy, or window functions. The DataTalksClub curriculum demonstrates this pattern in `07_groupby_join.ipynb`, where `repartition(20)` prepares data for aggregation by ensuring uniform partition sizes.

### Can coalesce increase the number of partitions?

Technically yes, but it triggers a shuffle. When `coalesce(numPartitions)` receives a value greater than the current partition count, it cannot simply split existing partitions locally and must instead perform a full shuffle, functioning identically to `repartition()`.

### Why does repartition create more evenly sized partitions than coalesce?

**`repartition()`** computes a hash of each row's key to randomly redistribute data across all target partitions, statistically balancing the volume of data per partition. **`coalesce()`** merely merges existing adjacent partitions together, preserving any original data skew present in the source partitions.