# Apache Iceberg Table Maintenance and Performance Optimization Techniques: A Practical Guide

> Optimize Apache Iceberg tables with strategic partitioning, file compaction, and query tuning. Minimize I/O and enhance performance with practical techniques from DataExpert-io.

- Repository: [DataExpert.io/data-engineer-handbook](https://github.com/DataExpert-io/data-engineer-handbook)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Apache Iceberg table maintenance and performance optimization techniques center on strategic partitioning, controlled file sizing through compaction, and query execution tuning to minimize I/O and avoid expensive shuffles.**

Keeping Apache Iceberg tables performant requires more than just creating tables with the `USING iceberg` syntax. The DataExpert-io/data-engineer-handbook repository provides hands-on examples demonstrating how to implement a complete maintenance loop that ensures low-latency analytics even as data volumes scale.

## Optimize Data Layout with Partitioning and Bucketing

Efficient data layout is the foundation of Iceberg performance. By organizing data to maximize partition pruning and minimize shuffle operations during joins, you can dramatically reduce query execution time.

### Time-Based Partitioning for Efficient Pruning

Partitioning on high-cardinality time columns allows the query engine to skip irrelevant files entirely. In `intermediate-bootcamp/materials/3-spark-fundamentals/notebooks/event_data_pyspark.ipynb`, the authors demonstrate creating a table partitioned by year to enable early partition pruning:

```python
spark.sql("""
  CREATE TABLE IF NOT EXISTS bootcamp.events (
    event_date DATE,
    event_time TIMESTAMP,
    host STRING,
    event_type STRING
  )
  USING iceberg
  PARTITIONED BY (years(event_date))
""")

```

This structure ensures that queries filtering on `event_date` read only the necessary files, reducing I/O overhead significantly.

### Bucketing and Sorting for Join Performance

For tables requiring frequent joins on specific keys, **bucketing** creates deterministic file placement that eliminates the need for shuffling. The `bucket-joins-in-iceberg.ipynb` notebook demonstrates combining bucketing with sorting to approximate Z-order clustering:

```scala
spark.sql("""
  CREATE TABLE IF NOT EXISTS bootcamp.matches_bucketed (
    match_id STRING,
    is_team_game BOOLEAN,
    playlist_id STRING,
    completion_date TIMESTAMP
  )
  USING iceberg
  PARTITIONED BY (completion_date, bucket(16, match_id))
""")

```

By specifying `bucket(16, match_id)`, rows with the same `match_id` hash to the same file group, enabling fast merge joins without data movement.

## Control File Size and Compaction

Iceberg tables naturally accumulate many small files during streaming or frequent batch writes. Without intervention, this creates metadata overhead and increases remote read requests.

### Target File Size Management

The repository emphasizes compacting files to approximately **128 MiB** for optimal read performance. While small files increase metadata overhead, the `OPTIMIZE` command (referenced in [`intermediate-bootcamp/materials/3-spark-fundamentals/README.md`](https://github.com/DataExpert-io/data-engineer-handbook/blob/main/intermediate-bootcamp/materials/3-spark-fundamentals/README.md)) merges these into larger, more efficient units.

Before compaction, you should explicitly control the write layout using `repartition` and `sortWithinPartitions`. As shown in `event_data_pyspark.ipynb`:

```python
sorted_df = df.repartition(10, col("event_date")) \
               .sortWithinPartitions(col("event_date"), col("host")) \
               .withColumn("event_time", col("event_time").cast("timestamp"))

sorted_df.write.mode("overwrite").saveAsTable("bootcamp.events_sorted")

```

This deterministic layout ensures that subsequent compaction operations produce well-organized files.

### The Maintenance Loop

According to the source code examples, a complete maintenance workflow follows this sequence:

1. **Write** new data using bucketed layouts with explicit sorting
2. **Refresh** table metadata using `REFRESH TABLE <table_name>` to guarantee the query planner sees the latest snapshots
3. **Run `OPTIMIZE`** periodically to compact small files into the target size range
4. **Monitor** file metrics using the `$files` metadata table

You can verify file distribution with this query from `event_data_pyspark.ipynb`:

```sql
SELECT SUM(file_size_in_bytes) AS total_size,
       COUNT(1) AS num_files,
       AVG(file_size_in_bytes) AS avg_file_size
FROM demo.bootcamp.events_sorted.files

```

## Tune Query Execution

Beyond physical layout, Spark-specific configurations prevent performance regressions during query execution.

### Disable Broadcast Joins for Large Tables

Broadcasting large Iceberg tables can cause out-of-memory errors or excessive network traffic. The `bucket-joins-in-iceberg.ipynb` notebook explicitly disables automatic broadcast joins when working with large bucketed tables:

```scala
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

```

Setting this threshold to `-1` forces Spark to use sort-merge joins, which handle large datasets more reliably when combined with properly bucketed tables.

### Leverage Predicate Push-Down

Iceberg maintains detailed column statistics in its metadata layer. By partitioning on query-filtered columns and using Iceberg's native format (`USING iceberg`), you automatically enable predicate push-down. This allows the engine to skip files entirely based on partition values and column statistics without reading the actual data files.

## Implementation Examples from the Data Engineer Handbook

The DataExpert-io/data-engineer-handbook repository provides concrete implementations in `intermediate-bootcamp/materials/3-spark-fundamentals/notebooks/`. The `event_data_pyspark.ipynb` file demonstrates end-to-end patterns including:

- Creating time-partitioned tables with `PARTITIONED BY (years(event_date))`
- Using `repartition()` and `sortWithinPartitions()` before writes
- Querying the `$files` metadata table to monitor file sizes

Meanwhile, `bucket-joins-in-iceberg.ipynb` focuses on:

- Creating bucketed tables with `bucket(16, match_id)`
- Disabling broadcast joins for large table integrity
- Analyzing query plans to verify bucketed join optimization

## Summary

- **Partition strategically** on time-oriented columns to enable partition pruning and reduce scanned data volumes.
- **Implement bucketing** on high-frequency join keys to eliminate shuffle operations and enable efficient merge joins.
- **Control write layout** using `repartition` and `sortWithinPartitions` to create deterministic file structures that compact efficiently.
- **Monitor file metrics** via the `$files` metadata table to identify when compaction is necessary.
- **Disable broadcast joins** (`spark.sql.autoBroadcastJoinThreshold = -1`) when working with large Iceberg tables to prevent memory issues.
- **Run periodic maintenance** including `REFRESH TABLE` and `OPTIMIZE` commands to maintain the target file size of approximately 128 MiB.

## Frequently Asked Questions

### How do I determine the optimal number of buckets for an Iceberg table?

Choose a bucket count that creates files close to your target size (128 MiB) while ensuring enough parallelism for your query engines. As demonstrated in `bucket-joins-in-iceberg.ipynb`, using `bucket(16, match_id)` provides a balance between file count and join performance. For tables with billions of rows, consider 256 or more buckets to keep individual files manageable.

### What is the difference between partitioning and bucketing in Apache Iceberg?

**Partitioning** organizes data into directory-like structures based on column values (such as date ranges), allowing the engine to skip entire partitions during queries. **Bucketing** hashes a column value into a fixed number of buckets, ensuring that rows with the same key always land in the same file group. This enables efficient joins without shuffling data across the network, as shown in the repository's bucketed table examples.

### When should I run the OPTIMIZE command on Iceberg tables?

Run `OPTIMIZE` when the `$files` metadata table shows many small files (significantly under 128 MiB) or when query latency degrades due to excessive file reads. According to the Data Engineer Handbook patterns, this typically follows heavy write operations or streaming ingestion. The command rewrites smaller files into larger, more efficient units while preserving the table's partitioning and bucketing structure.

### How can I verify that my Iceberg table maintenance is improving performance?

Query the `$files` metadata table to compare file counts and sizes before and after optimization, as illustrated in `event_data_pyspark.ipynb`. Additionally, examine Spark query plans to confirm that bucketed joins are executing without exchanges (shuffles). Reduced task counts and lower scan durations in your query history indicate successful optimization.