# How to Handle Large Files with Kanzi: Block Size Configuration and Memory Management

> Optimize large file compression with Kanzi. Learn how block size configuration and memory management impact performance and resource usage in your Kanzi project.

- Repository: [flanglet/kanzi-cpp](https://github.com/flanglet/kanzi-cpp)
- Tags: how-to-guide
- Published: 2026-03-02

---

**Kanzi compresses large inputs by partitioning them into independent blocks that are transformed and entropy-coded in parallel, where the block size—configurable via the `--block=` argument or automatically derived from file size and thread count—directly dictates per-thread memory allocation because each worker maintains dual buffers sized to `max(blockSize × 1.125, 64 KiB)`.**

The flanglet/kanzi-cpp repository implements block-based compression to balance throughput and resident memory. When processing multi-gigabyte files, selecting an appropriate block size prevents excessive RAM consumption while maintaining compression efficiency, and the `BlockCompressor` class exposes both hard limits and dynamic scaling to accommodate hardware constraints.

## Block Size Architecture and Limits

The `BlockCompressor` class defines the boundaries for valid block sizes in [`src/app/BlockCompressor.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/app/BlockCompressor.hpp) and initializes defaults in the corresponding implementation file.

### Default and Boundary Values

According to `src/app/BlockCompressor.cpp#L38`, the library declares a **default block size** of **4 MiB**:

```cpp
const int BlockCompressor::DEFAULT_BLOCK_SIZE = 4 * 1024 * 1024;

```

Hard limits are enforced via constants in `src/app/BlockCompressor.hpp#L41-L42`, restricting user-supplied values to a **minimum of 1 KiB** and a **maximum of 1 GiB**:

```cpp
static const int MIN_BLOCK_SIZE = 1024;
static const int MAX_BLOCK_SIZE = 1024 * 1024 * 1024;

```

Additionally, `CompressedOutputStream` validates that the block size is a **multiple of 16 bytes** in its constructor (`src/io/CompressedOutputStream.cpp#L82-L84`), throwing an `invalid_argument` if the alignment requirement is violated.

## Configuring Block Size via Command Line

The [`Kanzi.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/Kanzi.cpp) entry point parses user preferences and populates a context object that `BlockCompressor` consumes during initialization.

### Manual Specification

The parser recognizes `-b` or `--block=` followed by an integer and optional scale suffix (`K`, `M`, `G`). In `src/app/Kanzi.cpp#L887-L896`, the value is extracted and scaled:

```cpp
if ((ctx == ARG_IDX_BLOCK) || (arg.compare(0, 8, "--block=") == 0)) {
    // ...
    blockSize = int(uint64(blockSize) * scale);   // e.g., 64M becomes 67108864
    ctx = -1;
}

```

If the user omits the flag, the `BlockCompressor` constructor selects a default based on the **compression level** (e.g., level 9 doubles the default to **8 MiB**) via a `switch` statement around `src/app/BlockCompressor.cpp#L20-L36`.

### Automatic Block Calculation

When the `--auto` flag is present, Kanzi derives the block size from the input file dimensions and the number of jobs. Inside `BlockCompressor::compress` (`src/app/BlockCompressor.cpp#L34-L38` for single-file mode and lines 85‑89 for per-task paths), the logic executes:

```cpp
if ((_autoBlockSize == true) && (_jobs > 0)) {
    const int64 bl = files[0]._size / _jobs;
    _blockSize = int(max(min((bl + 63) & ~63,
                     int64(MAX_BLOCK_SIZE)), int64(MIN_BLOCK_SIZE)));
    _ctx.putInt("blockSize", _blockSize);
}

```

The calculation divides the file size by the job count, **aligns the result to 64-byte multiples** using bitwise masking `(bl + 63) & ~63`, and clamps the value within the 1 KiB–1 GiB range.

## Memory Consumption Model

Each compression job allocates **two buffers** to enable overlapping I/O and processing, totaling `2 × bufSize` per thread.

### Buffer Allocation Formula

In `src/io/CompressedOutputStream.cpp#L40-L44`, the buffer size is computed as:

```cpp
const int bufSize = max(_blockSize + (_blockSize >> 3), DEFAULT_BUFFER_SIZE);
_buffers[0] = new SliceArray<kanzi::byte>(new kanzi::byte[bufSize], bufSize, 0);

```

Key components of this formula include:

- **`_blockSize >> 3`** adds a **12.5 % safety margin** (`blockSize / 8`) to accommodate incompressible blocks.
- **`DEFAULT_BUFFER_SIZE`** is defined as **64 KiB** (`65536` bytes) in `src/app/BlockCompressor.hpp#L100`.

Therefore, the **total RAM usage** follows:

```

Total Memory ≈ jobs × 2 × max(blockSize × 1.125, 64 KiB)

```

For example, using the default **4 MiB** block size with **8 jobs** yields approximately **72 MiB** of buffer memory (8 × 2 × 4.5 MiB). Decompression mirrors this allocation strategy in [`src/io/CompressedInputStream.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/io/CompressedInputStream.cpp).

## Python API Control

The Python wrapper in [`src/api/kanzi.py`](https://github.com/flanglet/kanzi-cpp/blob/main/src/api/kanzi.py) exposes the **C API** parameter `blockSize` as the keyword argument `block_size`.

```python
from kanzi import Compressor

# Compress a 20 GiB file using 64 MiB blocks and 4 threads

with Compressor(
        dst_path='bigfile.knz',
        block_size=64 * 1024 * 1024,   # 64 MiB blocks

        jobs=4) as comp:
    # Feed data in chunks (e.g., 8 MiB reads)

    while chunk := input_file.read(8 * 1024 * 1024):
        comp.compress(chunk)

```

While the Python wrapper defaults to **1 MiB**, passing `block_size` overrides this value and forwards it to the native `cData.blockSize` field used by `CompressedOutputStream`.

## Optimization Strategies for Large Files

The interaction between block size, thread count, and available RAM determines throughput and stability:

| Scenario | Recommended Configuration |
|----------|---------------------------|
| **Very large files (≥ 10 GiB)** | Use `--auto` or manually set a large block (e.g., `--block=64M`). Ensure `jobs` matches physical core count. |
| **Memory-constrained hosts (≤ 2 GiB)** | Retain the default **4 MiB** block size and limit jobs using `jobs ≤ RAM / (2 × blockSize)`. |
| **Maximum compression ratio** | Larger blocks provide more context for transforms (BWT, RLT); try `--block=256M` if memory permits. |
| **Low-latency streaming** | Smaller blocks (1–2 MiB) reduce latency; keep `jobs` low to prevent buffer thrashing. |

### CLI Examples

```bash

# Compress 50 GiB with 128 MiB blocks and 8 threads

kanzi -c -i huge.bin -o huge.bin.knz --block=128M --jobs=8

# Let Kanzi auto-scale blocks for 4 cores

kanzi -c -i huge.bin -o huge.bin.knz --auto --jobs=4

```

### Python Examples

```python

# Manual block size for predictable memory

from kanzi import Compressor
with Compressor('huge.knz', block_size=128*1024*1024, jobs=8) as c:
    c.compress(data)

# Auto block size (requires wrapper support for autoBlockSize flag)

# with Compressor('huge.knz', auto=True, jobs=4) as c:

#     c.compress(data)

```

## Summary

- **Block size** in Kanzi defaults to **4 MiB** but is adjustable between **1 KiB** and **1 GiB**, and must be a **multiple of 16 bytes** ([`src/io/CompressedOutputStream.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/io/CompressedOutputStream.cpp)).
- **Memory usage** scales linearly with jobs and block size following the formula `jobs × 2 × max(blockSize × 1.125, 64 KiB)`.
- **Automatic sizing** via `--auto` derives blocks from `fileSize / jobs`, aligns to **64 bytes**, and clamps to limits ([`src/app/BlockCompressor.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/app/BlockCompressor.cpp)).
- **Configuration** is available both via CLI (`--block=`, `--auto`) and the Python API (`block_size` parameter).

## Frequently Asked Questions

### What is the default block size in Kanzi?

The `BlockCompressor` class defines a default of **4 MiB** (`4 * 1024 * 1024` bytes) in `src/app/BlockCompressor.cpp#L38`. If you invoke the CLI without specifying `--block`, this value is used unless the compression level triggers an override (e.g., level 9 uses **8 MiB**).

### How does the `--auto` flag calculate block size?

When `--auto` is supplied, Kanzi divides the input file size by the number of jobs (`files[0]._size / _jobs`) inside `BlockCompressor::compress`, rounds the result up to the **nearest 64-byte boundary** using the expression `(bl + 63) & ~63`, and clamps the result between `MIN_BLOCK_SIZE` (1 KiB) and `MAX_BLOCK_SIZE` (1 GiB).

### Why does Kanzi require more RAM than the block size I specify?

Each job allocates **two buffers** (for overlapping I/O and processing), and each buffer includes a **12.5 % overhead** (`_blockSize >> 3`) to handle incompressible data, computed in `src/io/CompressedOutputStream.cpp#L42`. Therefore, actual allocation per job is `2 × max(blockSize × 1.125, 64 KiB)`, not merely `2 × blockSize`.

### What is the minimum valid block size?

The library enforces a hard minimum of **1024 bytes** (1 KiB) defined in `src/app/BlockCompressor.hpp#L41`. Additionally, the `CompressedOutputStream` constructor validates that the provided size is a multiple of **16 bytes**; otherwise, it throws an `invalid_argument` exception.