Optimize Kanzi for Compression Ratio vs. Decompression Speed: A Complete Guide

To optimize Kanzi for compression ratio versus decompression speed, adjust the compression level (-l), block size (-b), and thread count (-j), where higher levels (5‑9) prioritize ratio using complex transforms like BWT and TPAQX, while lower levels (1‑4) and custom pipelines favor fast, single‑pass decoding.

The flanglet/kanzi-cpp repository implements a modular C++ compression engine that lets you explicitly trade archival density for decoding velocity. By understanding how the BlockCompressor class assembles transform and entropy‑coding pipelines, you can tune the library for anything from high‑throughput log processing to maximum‑ratio cold storage.

Understanding Kanzi's Two‑Stage Pipeline

Kanzi processes data through a Transform stage followed by an Entropy stage, both selected by the compression level or manual overrides. The command‑line driver in src/app/Kanzi.cpp builds a Context object that stores your settings, then instantiates BlockCompressor (or BlockDecompressor) to execute the pipeline per block.

The mapping from user‑friendly level numbers to concrete algorithms lives in BlockCompressor::getTransformAndCodec inside src/app/BlockCompressor.cpp. This function populates a two‑element string array with the transform chain and entropy codec:

// src/app/BlockCompressor.cpp – lines 21‑73
void BlockCompressor::getTransformAndCodec(int level, string tranformAndCodec[2])
{
    switch (level) {
    case 0:  tranformAndCodec[0] = "NONE";                tranformAndCodec[1] = "NONE";      break;
    case 1:  tranformAndCodec[0] = "LZX";                 tranformAndCodec[1] = "NONE";      break;
    case 2:  tranformAndCodec[0] = "DNA+LZ";              tranformAndCodec[1] = "HUFFMAN";   break;
    case 3:  tranformAndCodec[0] = "TEXT+UTF+PACK+MM+LZX";tranformAndCodec[1] = "HUFFMAN";   break;
    case 4:  tranformAndCodec[0] = "TEXT+UTF+EXE+PACK+MM+ROLZ"; tranformAndCodec[1] = "NONE";      break;
    case 5:  tranformAndCodec[0] = "TEXT+UTF+BWT+RANK+ZRLT"; tranformAndCodec[1] = "ANS0";    break;
    case 6:  tranformAndCodec[0] = "TEXT+UTF+BWT+SRT+ZRLT";  tranformAndCodec[1] = "FPAQ";    break;
    case 7:  tranformAndCodec[0] = "LZP+TEXT+UTF+BWT+LZP";  tranformAndCodec[1] = "CM";      break;
    case 8:  tranformAndCodec[0] = "EXE+RLT+TEXT+UTF+DNA"; tranformAndCodec[1] = "TPAQ";    break;
    case 9:  tranformAndCodec[0] = "EXE+RLT+TEXT+UTF+DNA"; tranformAndCodec[1] = "TPAQX";   break;
    default: tranformAndCodec[0] = "Unknown";             tranformAndCodec[1] = "Unknown";
    }
}

Higher levels inject Burrows‑Wheeler Transform (BWT), Zero‑Run Length Transform (ZRLT), and context‑mixing entropy coders (TPAQX) that improve statistical compression but require heavier CPU work to invert during decompression.

How Compression Level Impacts the Ratio‑Speed Trade‑Off

The compression level (-l 0 through -l 9) is the primary dial for optimizing Kanzi. Each step up the ladder adds more sophisticated (and computationally expensive) transforms:

• Levels 0‑3 use lightweight transforms (LZX, PACK, MM) and simple entropy coders (HUFFMAN or none). Decompression is essentially single‑pass and memory‑light.
• Levels 5‑6 introduce BWT and ANS0/FPAQ. The inverse BWT requires sorting or matrix operations that add latency to every block decoded.
• Levels 8‑9 activate TPAQ and TPAQX, which employ advanced context modelling and arithmetic coding. These yield the best ratios but can be 2‑3× slower to decode than Level 3.

When you optimize Kanzi for compression ratio versus decompression speed, consider that transform reversal (e.g., inverse BWT, inverse RLT) and entropy decoder complexity directly extend decompression wall‑clock time. Level 9 may achieve superior density, but it forces the BlockDecompressor to run expensive inverse transforms on every block.

Tuning Block Size and Parallelism

Beyond the algorithmic selection, two hardware‑aware options dominate decompression throughput:

Block Size (-b or --block)
Larger blocks (e.g., -b 64m) give transforms like BWT more context, improving ratio, but they increase memory pressure and can degrade cache locality during decode. The default -b 4m (4 MiB) fits comfortably in L3 cache for most modern CPUs. Very large blocks (>256 MiB) may slow decompression due to RAM bandwidth limits.

Parallelism (-j or --jobs)
The jobs parameter controls how many blocks are processed concurrently. Because each block decompresses independently, setting -j to your physical core count (or slightly less) maximizes throughput without impacting compression ratio. This is the most effective way to recover speed when you must use a high compression level.

Auto‑Block (--autoBlock)
When enabled, BlockCompressor::compress automatically selects a block size based on input size and thread count to balance workload distribution. This helps maintain speed across heterogeneous file sizes but sacrifices reproducibility for manual tuning.

Custom Pipelines: Bypassing the Level Mapping

Advanced users can override the level‑to‑pipeline mapping entirely using --transform and --entropy. This lets you drop heavy transforms while keeping useful ones, effectively sliding anywhere on the ratio‑speed spectrum.

For example, removing BWT but keeping text‑specific transforms yields a fast‑decode, moderate‑ratio configuration:

kanzi -c -i dataset.dat -o dataset.knz \
      --transform=TEXT+UTF+PACK+MM+LZX \
      --entropy=HUFFMAN \
      -j 16 -b 4m

According to the source in src/transform/TransformFactory.hpp and src/entropy/EntropyEncoderFactory.hpp, any combination listed in the level mapping is valid here, plus additional experimental chains.

Practical Configuration Examples

Fast Decompression with Acceptable Ratio (Level 4)

For archives that are written once and read frequently, Level 4 avoids BWT and heavy entropy coding:

kanzi -c -i data.txt -o data.knz -l 4 -j 8 -b 4m

This invokes TEXT+UTF+EXE+PACK+MM+ROLZ with no entropy codec. ROLZ (Reduced Offset Lempel‑Ziv) is lightweight to invert, and 4 MiB blocks stay L3‑resident, keeping decompression latency low.

Maximum Compression Ratio (Level 9)

For cold storage where decode speed is irrelevant, use the strongest pipeline:

kanzi -c -i bigfile.iso -o bigfile.knz -l 9 -j 4 -b 64m

Level 9 selects EXE+RLT+TEXT+UTF+DNA plus TPAQX. The 64 MiB block maximizes transform context, and 4 threads prevent memory exhaustion on large files while still parallelizing the heavy arithmetic coding work.

Custom Lightweight Pipeline for Log Files

When compressing repetitive logs that require instant decoding, bypass the level system for pure LZ:

kanzi -c -i logfile.log -o logfile.knz \
      --transform=LZX --entropy=HUFFMAN -j 16 -b 2m

This configuration uses only LZX (Lempel‑Ziv‑X) and Huffman coding—both single‑pass algorithms that decompress at memory‑copy speeds.

Programmatic Configuration via C++ API

Embed the same optimizations in your application by populating a Context object before constructing BlockCompressor:

#include "kanzi.hpp"

kanzi::Context ctx;
ctx.putInt("level", 5);          // BWT+RANK+ZRLT + ANS0
ctx.putInt("jobs", 8);           // Parallelism
ctx.putInt("blockSize", 8<<20);  // 8 MiB blocks
ctx.putInt("checksum", 32);      // 32‑bit integrity check

kanzi::BlockCompressor bc(ctx);
uint64 outSize = 0;
int rc = bc.compress(outSize);

The API respects the identical mapping logic found in BlockCompressor::getTransformAndCodec, ensuring CLI and programmatic behavior match.

Key Source Files for Reference

File	Purpose
src/app/Kanzi.cpp	Parses CLI arguments, instantiates Context, and launches compress/decompress workers.
src/app/BlockCompressor.cpp	Contains getTransformAndCodec() and the threading logic for compression.
src/app/BlockDecompressor.cpp	Mirrors the compressor for decoding; handles block size and parallel extraction.
src/transform/TransformFactory.hpp	Factory that instantiates transform objects (BWT, RLT, LZX, etc.) from string tokens.
src/entropy/EntropyEncoderFactory.hpp	Factory for entropy coders (HUFFMAN, ANS0, FPAQ, TPAQX).
src/entropy/EntropyEncoder.hpp	Base interface for all entropy encoders; implementations reside in src/entropy/.
src/util/Clock.hpp	Timing utilities used to report throughput in InfoPrinter.

Summary

• Compression level (-l) is the dominant factor: Levels 1‑4 favor speed; Levels 5‑9 add BWT and complex entropy coding for better ratio at the cost of decode time.
• Block size (-b) trades memory pressure for transform efficiency; 4 MiB is the sweet spot for most CPUs, while 64 MiB+ helps ratio on large files.
• Parallelism (-j) recovers wall‑clock speed without affecting ratio; set to physical core count for both compress and decompress.
• Custom pipelines (--transform, --entropy) let you design a bespoke balance, such as dropping BWT to halve decompression latency.
• The BlockCompressor and BlockDecompressor classes in src/app/ enforce these behaviors consistently across the CLI and C++ API.

Frequently Asked Questions

What is the best compression level for fast decompression?

Level 4 or 5 provides the best balance. Level 4 uses ROLZ without BWT, offering single‑pass decoding, while Level 5 adds BWT but keeps the faster ANS0 entropy coder. Both avoid the expensive TPAQX context modelling found in Levels 8‑9.

Does block size affect compression ratio?

Yes. Larger blocks (e.g., -b 64m) improve ratio—especially for BWT‑based levels—by giving the transform more data context. However, blocks larger than 256 MiB can hurt decompression speed due to cache misses and increased memory traffic.

Can I change the transform chain without recompiling Kanzi?

Absolutely. Use the --transform=<chain> and --entropy=<codec> command‑line options to override the level mapping at runtime. The TransformFactory and EntropyEncoderFactory classes parse these strings dynamically, as seen in src/transform/TransformFactory.hpp.

Why is Level 9 decompression slower than Level 3?

Level 9 employs TPAQX, a context‑mixing entropy codec with arithmetic coding, and multiple invertible transforms (RLT, BWT). These require significant CPU work to reverse compared to Level 3’s simple LZX + HUFFMAN pipeline, which is essentially a single memory‑streaming pass.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}