# Best Transforms for Text Compression in Kanzi: TEXT + UTF + PACK Pipeline Explained

> Discover the best transforms for text compression in Kanzi with the TEXT+UTF+PACK pipeline. Optimize UTF-8 text compression by chaining dictionary replacement, symbol aliasing, and byte packing.

- Repository: [flanglet/kanzi-cpp](https://github.com/flanglet/kanzi-cpp)
- Tags: deep-dive
- Published: 2026-03-02

---

**The optimal transform pipeline for compressing UTF-8 text in Kanzi is `TEXT+UTF+PACK`, which chains dictionary-based word replacement, UTF-8 symbol aliasing, and residual byte packing to minimize alphabet size before entropy coding.**

The Kanzi compression library (`flanglet/kanzi-cpp`) achieves high compression ratios by chaining transform modules prior to entropy coding. For natural language text, selecting the right sequence of transforms determines whether you achieve maximum compression or optimal encoding speed. This guide explains the definitive `TEXT+UTF+PACK` pipeline based on the actual source implementation in the C++ repository.

## How the TEXT + UTF + PACK Pipeline Works

The three-transform combo represents the sweet spot for UTF-8 text compression, progressively reducing data redundancy through distinct mechanisms.

### TEXT (Dictionary Codec)

The **TEXT** transform implements a dynamic dictionary codec that scans input blocks to identify the most frequent words. In [`src/transform/TextCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/TextCodec.hpp), the encoder builds a dictionary on-the-fly and replaces each word instance with a compact integer index. This eliminates repetition of common terms while the dictionary itself is stored compactly for the decoder.

### UTF (UTF-8 Alias Codec)

The **UTF** transform, implemented in [`src/transform/UTFCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.hpp), encodes each UTF-8 code point as a small integer based on symbol frequency. It transforms multi-byte UTF-8 sequences (1-4 bytes) into single-byte aliases, dramatically reducing the alphabet size for natural-language text before entropy coding begins.

### PACK (Alias Codec)

The **PACK** transform serves as the final alphabet optimizer. Located in [`src/transform/AliasCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/AliasCodec.hpp), it replaces any remaining unused byte values in the stream with even smaller aliases. This low-overhead step ensures the entropy coder processes the smallest possible symbol set with minimal CPU cost.

## Implementation in the Kanzi Source Code

The transform pipeline is constructed through the factory pattern in [`src/transform/TransformFactory.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/TransformFactory.hpp). The `getTypeToken` method (lines 86-92) maps textual transform names to internal enum values:

```cpp
// TransformFactory.hpp
if (name == "TEXT")   return DICT_TYPE;   // TextCodec
if (name == "UTF")    return UTF_TYPE;    // UTFCodec  
if (name == "PACK")   return PACK_TYPE;   // AliasCodec

```

To instantiate the pipeline programmatically, call `TransformFactory<byte>::getType()` with a plus-separated string, which returns a 64-bit token encoding the transform sequence. The `newTransform()` method then constructs the concrete `TransformSequence` containing the actual codec objects.

## Compression Levels and Default Pipelines

Kanzi automatically configures transform pipelines based on compression level selections. According to the help text in [`src/app/Kanzi.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/app/Kanzi.cpp) (around line 136), **Level 3** (the first level utilizing dictionary coding) defaults to:

```

TEXT+UTF+PACK+MM+LZX

```

For pure text compression, drop the trailing `MM` (multimedia) and `LZX` (fast LZ) transforms. The first three components handle natural language efficiently, while the additional transforms target mixed binary content or longer repetitive sequences.

## When to Modify the Pipeline

Adjust the standard `TEXT+UTF+PACK` pipeline based on your content characteristics:

- **Embedded binary content**: Append `MM+LZX` when compressing HTML containing images or documents with mixed formats. `MM` decorrelates channel-wise data, while `LZX` captures longer repetitions missed by dictionary coding.
- **Maximum speed**: Use only `UTF+PACK` or standalone `PACK` to skip dictionary overhead, sacrificing a few percentage points of compression for faster encoding.
- **Small blocks (< 4 KB)**: Omit `TEXT` entirely, as dictionary construction overhead dominates for tiny inputs. `UTF+PACK` provides sufficient alphabet reduction for small text fragments.

## Code Examples

### Building the Pipeline with the C++ API

```cpp
#include "kanzi/Context.hpp"
#include "kanzi/TransformFactory.hpp"

int main()
{
    kanzi::Context ctx;
    ctx.putString("entropy", "HUFFMAN");

    // Parse transform string into 64-bit type token
    uint64 type = kanzi::TransformFactory<byte>::getType("TEXT+UTF+PACK");
    kanzi::TransformSequence<byte>* tr = 
        kanzi::TransformFactory<byte>::newTransform(ctx, type);

    // Use with Compressor/Decompressor
    // kanzi::Compressor<byte> cmp(ctx, tr);
}

```

The `getType()` method handles the plus-separated syntax (lines 98-136 in [`TransformFactory.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/TransformFactory.hpp)), while `newTransform()` instantiates the concrete `TextCodec`, `UTFCodec`, and `AliasCodec` objects.

### Command-Line Usage

```bash
kanzi -c -i foo.txt -o foo.knz -t TEXT+UTF+PACK -e HUFFMAN -l 3

```

The `-t` flag explicitly sets the transform pipeline, overriding the default Level 3 configuration to exclude `MM` and `LZX` for pure text optimization.

### Python API Implementation

```python
import kanzi

ctx = kanzi.Context()
ctx.set('entropy', 'HUFFMAN')
tr = kanzi.TransformFactory.get_transform('TEXT+UTF+PACK', ctx)

cmp = kanzi.Compressor(ctx, tr)
with open('foo.txt', 'rb') as fin, open('foo.knz', 'wb') as fout:
    cmp.compress(fin, fout)

```

The Python bindings in [`src/api/kanzi.py`](https://github.com/flanglet/kanzi-cpp/blob/main/src/api/kanzi.py) expose the same factory logic, parsing transform strings through the underlying C++ implementation.

## Summary

- **TEXT** ([`TextCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/TextCodec.hpp)) compresses repeated words into dictionary indices, eliminating lexical redundancy.
- **UTF** ([`UTFCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/UTFCodec.hpp)) shrinks UTF-8 code points to single-byte aliases based on frequency analysis.
- **PACK** ([`AliasCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/AliasCodec.hpp)) removes unused byte values from the final alphabet with minimal CPU overhead.
- The **TransformFactory** API assembles these components via `getType()` and `newTransform()` methods.
- Compression **Level 3** defaults include these three transforms plus `MM` and `LZX` for handling mixed content.
- Omit dictionary coding for blocks under 4 KB, and add multimedia transforms when binary data is present.

## Frequently Asked Questions

### What is the difference between TEXT and UTF transforms in Kanzi?

**TEXT** is a dictionary codec that replaces whole words with integer indices, targeting repetitive lexical patterns. **UTF** is an alias codec that maps individual UTF-8 code points to smaller integer values, reducing character encoding overhead. Together they minimize both pattern repetition and alphabet size.

### When should I use the PACK transform?

Always use **PACK** as the final step in text pipelines to alias unused byte values after dictionary and UTF-8 processing. It executes with minimal CPU overhead while ensuring the entropy coder receives the smallest possible symbol set, improving compression efficiency.

### How do I handle text containing embedded binary data?

Append `MM` (multimedia transform) and `LZX` (fast LZ codec) to the pipeline when compressing HTML with embedded images or mixed-format documents. These capture channel-wise correlations and longer repetitions that pure dictionary coding cannot identify.

### Does the TEXT transform work efficiently on small files?

For blocks smaller than 4 KB, **omit the TEXT transform** because dictionary construction overhead exceeds compression gains. Use the `UTF+PACK` combination for small text fragments to maintain speed while still achieving alphabet reduction.