How to Handle Large Files with Kanzi: Block Size Configuration and Memory Management

Kanzi compresses large inputs by partitioning them into independent blocks that are transformed and entropy-coded in parallel, where the block size—configurable via the --block= argument or automatically derived from file size and thread count—directly dictates per-thread memory allocation because each worker maintains dual buffers sized to max(blockSize × 1.125, 64 KiB).

The flanglet/kanzi-cpp repository implements block-based compression to balance throughput and resident memory. When processing multi-gigabyte files, selecting an appropriate block size prevents excessive RAM consumption while maintaining compression efficiency, and the BlockCompressor class exposes both hard limits and dynamic scaling to accommodate hardware constraints.

Block Size Architecture and Limits

The BlockCompressor class defines the boundaries for valid block sizes in src/app/BlockCompressor.hpp and initializes defaults in the corresponding implementation file.

Default and Boundary Values

According to src/app/BlockCompressor.cpp#L38, the library declares a default block size of 4 MiB:

const int BlockCompressor::DEFAULT_BLOCK_SIZE = 4 * 1024 * 1024;

Hard limits are enforced via constants in src/app/BlockCompressor.hpp#L41-L42, restricting user-supplied values to a minimum of 1 KiB and a maximum of 1 GiB:

static const int MIN_BLOCK_SIZE = 1024;
static const int MAX_BLOCK_SIZE = 1024 * 1024 * 1024;

Additionally, CompressedOutputStream validates that the block size is a multiple of 16 bytes in its constructor (src/io/CompressedOutputStream.cpp#L82-L84), throwing an invalid_argument if the alignment requirement is violated.

Configuring Block Size via Command Line

The Kanzi.cpp entry point parses user preferences and populates a context object that BlockCompressor consumes during initialization.

Manual Specification

The parser recognizes -b or --block= followed by an integer and optional scale suffix (K, M, G). In src/app/Kanzi.cpp#L887-L896, the value is extracted and scaled:

if ((ctx == ARG_IDX_BLOCK) || (arg.compare(0, 8, "--block=") == 0)) {
    // ...
    blockSize = int(uint64(blockSize) * scale);   // e.g., 64M becomes 67108864
    ctx = -1;
}

If the user omits the flag, the BlockCompressor constructor selects a default based on the compression level (e.g., level 9 doubles the default to 8 MiB) via a switch statement around src/app/BlockCompressor.cpp#L20-L36.

Automatic Block Calculation

When the --auto flag is present, Kanzi derives the block size from the input file dimensions and the number of jobs. Inside BlockCompressor::compress (src/app/BlockCompressor.cpp#L34-L38 for single-file mode and lines 85‑89 for per-task paths), the logic executes:

if ((_autoBlockSize == true) && (_jobs > 0)) {
    const int64 bl = files[0]._size / _jobs;
    _blockSize = int(max(min((bl + 63) & ~63,
                     int64(MAX_BLOCK_SIZE)), int64(MIN_BLOCK_SIZE)));
    _ctx.putInt("blockSize", _blockSize);
}

The calculation divides the file size by the job count, aligns the result to 64-byte multiples using bitwise masking (bl + 63) & ~63, and clamps the value within the 1 KiB–1 GiB range.

Memory Consumption Model

Each compression job allocates two buffers to enable overlapping I/O and processing, totaling 2 × bufSize per thread.

Buffer Allocation Formula

In src/io/CompressedOutputStream.cpp#L40-L44, the buffer size is computed as:

const int bufSize = max(_blockSize + (_blockSize >> 3), DEFAULT_BUFFER_SIZE);
_buffers[0] = new SliceArray<kanzi::byte>(new kanzi::byte[bufSize], bufSize, 0);

Key components of this formula include:

• _blockSize >> 3 adds a 12.5 % safety margin (blockSize / 8) to accommodate incompressible blocks.
• DEFAULT_BUFFER_SIZE is defined as 64 KiB (65536 bytes) in src/app/BlockCompressor.hpp#L100.

Therefore, the total RAM usage follows:

Total Memory ≈ jobs × 2 × max(blockSize × 1.125, 64 KiB)

For example, using the default 4 MiB block size with 8 jobs yields approximately 72 MiB of buffer memory (8 × 2 × 4.5 MiB). Decompression mirrors this allocation strategy in src/io/CompressedInputStream.cpp.

Python API Control

The Python wrapper in src/api/kanzi.py exposes the C API parameter blockSize as the keyword argument block_size.

from kanzi import Compressor

# Compress a 20 GiB file using 64 MiB blocks and 4 threads

with Compressor(
        dst_path='bigfile.knz',
        block_size=64 * 1024 * 1024,   # 64 MiB blocks

        jobs=4) as comp:
    # Feed data in chunks (e.g., 8 MiB reads)

    while chunk := input_file.read(8 * 1024 * 1024):
        comp.compress(chunk)

While the Python wrapper defaults to 1 MiB, passing block_size overrides this value and forwards it to the native cData.blockSize field used by CompressedOutputStream.

Optimization Strategies for Large Files

The interaction between block size, thread count, and available RAM determines throughput and stability:

Scenario	Recommended Configuration
Very large files (≥ 10 GiB)	Use --auto or manually set a large block (e.g., --block=64M). Ensure jobs matches physical core count.
Memory-constrained hosts (≤ 2 GiB)	Retain the default 4 MiB block size and limit jobs using jobs ≤ RAM / (2 × blockSize).
Maximum compression ratio	Larger blocks provide more context for transforms (BWT, RLT); try --block=256M if memory permits.
Low-latency streaming	Smaller blocks (1–2 MiB) reduce latency; keep jobs low to prevent buffer thrashing.

CLI Examples

# Compress 50 GiB with 128 MiB blocks and 8 threads

kanzi -c -i huge.bin -o huge.bin.knz --block=128M --jobs=8

# Let Kanzi auto-scale blocks for 4 cores

kanzi -c -i huge.bin -o huge.bin.knz --auto --jobs=4

Python Examples

# Manual block size for predictable memory

from kanzi import Compressor
with Compressor('huge.knz', block_size=128*1024*1024, jobs=8) as c:
    c.compress(data)

# Auto block size (requires wrapper support for autoBlockSize flag)

# with Compressor('huge.knz', auto=True, jobs=4) as c:

#     c.compress(data)

Summary

• Block size in Kanzi defaults to 4 MiB but is adjustable between 1 KiB and 1 GiB, and must be a multiple of 16 bytes (src/io/CompressedOutputStream.cpp).
• Memory usage scales linearly with jobs and block size following the formula jobs × 2 × max(blockSize × 1.125, 64 KiB).
• Automatic sizing via --auto derives blocks from fileSize / jobs, aligns to 64 bytes, and clamps to limits (src/app/BlockCompressor.cpp).
• Configuration is available both via CLI (--block=, --auto) and the Python API (block_size parameter).

Frequently Asked Questions

What is the default block size in Kanzi?

The BlockCompressor class defines a default of 4 MiB (4 * 1024 * 1024 bytes) in src/app/BlockCompressor.cpp#L38. If you invoke the CLI without specifying --block, this value is used unless the compression level triggers an override (e.g., level 9 uses 8 MiB).

How does the --auto flag calculate block size?

When --auto is supplied, Kanzi divides the input file size by the number of jobs (files[0]._size / _jobs) inside BlockCompressor::compress, rounds the result up to the nearest 64-byte boundary using the expression (bl + 63) & ~63, and clamps the result between MIN_BLOCK_SIZE (1 KiB) and MAX_BLOCK_SIZE (1 GiB).

Why does Kanzi require more RAM than the block size I specify?

Each job allocates two buffers (for overlapping I/O and processing), and each buffer includes a 12.5 % overhead (_blockSize >> 3) to handle incompressible data, computed in src/io/CompressedOutputStream.cpp#L42. Therefore, actual allocation per job is 2 × max(blockSize × 1.125, 64 KiB), not merely 2 × blockSize.

What is the minimum valid block size?

The library enforces a hard minimum of 1024 bytes (1 KiB) defined in src/app/BlockCompressor.hpp#L41. Additionally, the CompressedOutputStream constructor validates that the provided size is a multiple of 16 bytes; otherwise, it throws an invalid_argument exception.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}