Best Transforms for Text Compression in Kanzi: TEXT + UTF + PACK Pipeline Explained

The optimal transform pipeline for compressing UTF-8 text in Kanzi is TEXT+UTF+PACK, which chains dictionary-based word replacement, UTF-8 symbol aliasing, and residual byte packing to minimize alphabet size before entropy coding.

The Kanzi compression library (flanglet/kanzi-cpp) achieves high compression ratios by chaining transform modules prior to entropy coding. For natural language text, selecting the right sequence of transforms determines whether you achieve maximum compression or optimal encoding speed. This guide explains the definitive TEXT+UTF+PACK pipeline based on the actual source implementation in the C++ repository.

How the TEXT + UTF + PACK Pipeline Works

The three-transform combo represents the sweet spot for UTF-8 text compression, progressively reducing data redundancy through distinct mechanisms.

TEXT (Dictionary Codec)

The TEXT transform implements a dynamic dictionary codec that scans input blocks to identify the most frequent words. In src/transform/TextCodec.hpp, the encoder builds a dictionary on-the-fly and replaces each word instance with a compact integer index. This eliminates repetition of common terms while the dictionary itself is stored compactly for the decoder.

UTF (UTF-8 Alias Codec)

The UTF transform, implemented in src/transform/UTFCodec.hpp, encodes each UTF-8 code point as a small integer based on symbol frequency. It transforms multi-byte UTF-8 sequences (1-4 bytes) into single-byte aliases, dramatically reducing the alphabet size for natural-language text before entropy coding begins.

PACK (Alias Codec)

The PACK transform serves as the final alphabet optimizer. Located in src/transform/AliasCodec.hpp, it replaces any remaining unused byte values in the stream with even smaller aliases. This low-overhead step ensures the entropy coder processes the smallest possible symbol set with minimal CPU cost.

Implementation in the Kanzi Source Code

The transform pipeline is constructed through the factory pattern in src/transform/TransformFactory.hpp. The getTypeToken method (lines 86-92) maps textual transform names to internal enum values:

// TransformFactory.hpp
if (name == "TEXT")   return DICT_TYPE;   // TextCodec
if (name == "UTF")    return UTF_TYPE;    // UTFCodec  
if (name == "PACK")   return PACK_TYPE;   // AliasCodec

To instantiate the pipeline programmatically, call TransformFactory<byte>::getType() with a plus-separated string, which returns a 64-bit token encoding the transform sequence. The newTransform() method then constructs the concrete TransformSequence containing the actual codec objects.

Compression Levels and Default Pipelines

Kanzi automatically configures transform pipelines based on compression level selections. According to the help text in src/app/Kanzi.cpp (around line 136), Level 3 (the first level utilizing dictionary coding) defaults to:

TEXT+UTF+PACK+MM+LZX

For pure text compression, drop the trailing MM (multimedia) and LZX (fast LZ) transforms. The first three components handle natural language efficiently, while the additional transforms target mixed binary content or longer repetitive sequences.

When to Modify the Pipeline

Adjust the standard TEXT+UTF+PACK pipeline based on your content characteristics:

• Embedded binary content: Append MM+LZX when compressing HTML containing images or documents with mixed formats. MM decorrelates channel-wise data, while LZX captures longer repetitions missed by dictionary coding.
• Maximum speed: Use only UTF+PACK or standalone PACK to skip dictionary overhead, sacrificing a few percentage points of compression for faster encoding.
• Small blocks (< 4 KB): Omit TEXT entirely, as dictionary construction overhead dominates for tiny inputs. UTF+PACK provides sufficient alphabet reduction for small text fragments.

Code Examples

Building the Pipeline with the C++ API

#include "kanzi/Context.hpp"
#include "kanzi/TransformFactory.hpp"

int main()
{
    kanzi::Context ctx;
    ctx.putString("entropy", "HUFFMAN");

    // Parse transform string into 64-bit type token
    uint64 type = kanzi::TransformFactory<byte>::getType("TEXT+UTF+PACK");
    kanzi::TransformSequence<byte>* tr = 
        kanzi::TransformFactory<byte>::newTransform(ctx, type);

    // Use with Compressor/Decompressor
    // kanzi::Compressor<byte> cmp(ctx, tr);
}

The getType() method handles the plus-separated syntax (lines 98-136 in TransformFactory.hpp), while newTransform() instantiates the concrete TextCodec, UTFCodec, and AliasCodec objects.

Command-Line Usage

kanzi -c -i foo.txt -o foo.knz -t TEXT+UTF+PACK -e HUFFMAN -l 3

The -t flag explicitly sets the transform pipeline, overriding the default Level 3 configuration to exclude MM and LZX for pure text optimization.

Python API Implementation

import kanzi

ctx = kanzi.Context()
ctx.set('entropy', 'HUFFMAN')
tr = kanzi.TransformFactory.get_transform('TEXT+UTF+PACK', ctx)

cmp = kanzi.Compressor(ctx, tr)
with open('foo.txt', 'rb') as fin, open('foo.knz', 'wb') as fout:
    cmp.compress(fin, fout)

The Python bindings in src/api/kanzi.py expose the same factory logic, parsing transform strings through the underlying C++ implementation.

Summary

• TEXT (TextCodec.hpp) compresses repeated words into dictionary indices, eliminating lexical redundancy.
• UTF (UTFCodec.hpp) shrinks UTF-8 code points to single-byte aliases based on frequency analysis.
• PACK (AliasCodec.hpp) removes unused byte values from the final alphabet with minimal CPU overhead.
• The TransformFactory API assembles these components via getType() and newTransform() methods.
• Compression Level 3 defaults include these three transforms plus MM and LZX for handling mixed content.
• Omit dictionary coding for blocks under 4 KB, and add multimedia transforms when binary data is present.

Frequently Asked Questions

What is the difference between TEXT and UTF transforms in Kanzi?

TEXT is a dictionary codec that replaces whole words with integer indices, targeting repetitive lexical patterns. UTF is an alias codec that maps individual UTF-8 code points to smaller integer values, reducing character encoding overhead. Together they minimize both pattern repetition and alphabet size.

When should I use the PACK transform?

Always use PACK as the final step in text pipelines to alias unused byte values after dictionary and UTF-8 processing. It executes with minimal CPU overhead while ensuring the entropy coder receives the smallest possible symbol set, improving compression efficiency.

How do I handle text containing embedded binary data?

Append MM (multimedia transform) and LZX (fast LZ codec) to the pipeline when compressing HTML with embedded images or mixed-format documents. These capture channel-wise correlations and longer repetitions that pure dictionary coding cannot identify.

Does the TEXT transform work efficiently on small files?

For blocks smaller than 4 KB, omit the TEXT transform because dictionary construction overhead exceeds compression gains. Use the UTF+PACK combination for small text fragments to maintain speed while still achieving alphabet reduction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}