# How Kanzi’s UTF-8 Transform Handles International Character Data

> Discover how Kanzi's UTF-8 transform efficiently handles international character data by validating Unicode text, packing code points, and using short aliases for lossless multilingual data reconstruction.

- Repository: [flanglet/kanzi-cpp](https://github.com/flanglet/kanzi-cpp)
- Tags: deep-dive
- Published: 2026-03-02

---

**The Kanzi UTF-8 transform validates Unicode text against statistical patterns, packs code points into 22-bit integers, and replaces frequent symbols with short 1- or 2-byte aliases to minimize entropy while guaranteeing lossless reconstruction of multilingual data.**

Kanzi is a high-performance compression library that treats UTF-8 encoding as a reversible preprocessing step. The `UTFCodec` class, found in [`src/transform/UTFCodec.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.cpp), implements this transform by validating international character sequences and applying frequency-based symbol aliasing before entropy coding begins.

## Stage 1: Input Validation and BOM Handling

The transform begins in `UTFCodec::forward()` (lines 48‑90 in [`src/transform/UTFCodec.cpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.cpp)) by enforcing strict entry criteria that protect downstream compression stages from corrupt data.

- **Minimum block size**: Blocks smaller than 1024 bytes are rejected immediately to ensure statistical significance.
- **BOM stripping**: If a UTF-8 Byte-Order-Mark (`0xEF 0xBB 0xBF`) is detected, the transform removes it and sets the start offset to 3 bytes.
- **Context-aware filtering**: When a `Context` object is provided, the codec checks the stored `dataType`. If the type is explicitly set and not `UTF8`, the block is rejected without processing.
- **Statistical validation**: For unknown data types, `UTFCodec::validate()` (lines 305‑424) performs a fast byte-pair frequency analysis to confirm the block follows legal UTF-8 patterns defined by the Unicode 16 standard, rejecting overlong encodings or invalid start bytes like `0xC0`/`0xC1`.

## Stage 2: Symbol Packing and Alias Map Construction

Once validated, the transform constructs a frequency-based alias table to shorten common Unicode values. This occurs in `UTFCodec::forward()` (lines 110‑176).

**Symbol packing**: Each UTF-8 code point is packed into a 22-bit integer by `UTFCodec::pack()` (lines 70‑102 in [`src/transform/UTFCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.hpp)). The high-order bits encode the byte-length (1, 2, 3, or 4), while the low-order bits store the Unicode scalar value.

**Frequency mapping**: While scanning the block, the transform counts every packed value in an `aliasMap`. New code points trigger the creation of `sdUTF` entries (value + frequency) in a temporary vector `v`.

**Alias generation**: After the scan, the transform retains only symbols that appear frequently enough to keep the map size below approximately 10% of the block size. These symbols are sorted by decreasing frequency to create a rank-ordered alias table. The first 128 symbols receive 1-byte aliases; all others receive 2-byte aliases.

## Stage 3: Block Encoding and Decoding

The encoding phase writes a compact binary representation that preserves all international character data exactly.

**Header structure**: The output begins with two header bytes indicating the start offset and the count of invalid/truncated symbols at the block boundaries.

**Alias emission**: Pre-BOM bytes and truncated start symbols are copied verbatim. Every validated Unicode symbol is then replaced by its alias, emitted as a little-endian integer (low byte first, high byte only if alias ≥ 0x100). Truncated symbols at the end are copied verbatim.

**Decoding**: The inverse operation in `UTFCodec::inverse()` (lines 208‑297) reads the header, rebuilds the identical alias table from the map data, and expands each alias back to the original packed value using `UTFCodec::unpack()` (lines 113‑151 in [`src/transform/UTFCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.hpp)).

## Design Advantages for International Text

The `LEN_SEQ` classification table (line 29 in [`src/transform/UTFCodec.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/UTFCodec.hpp)) enables full UTF-8 support by categorizing bytes into 1-, 2-, 3-, or 4-byte sequences. This allows the `pack()` routine to recognize any legal Unicode scalar from U+0000 to U+10FFFF, including language-specific characters like é, 汉, or अ.

By mapping frequently-used code points to short aliases, the transform dramatically reduces entropy for multilingual texts. The statistical validation step prevents malicious or corrupt data from entering the compression pipeline, while the embedded alias map ensures a lossless round-trip for all characters, including those outside the Basic Multilingual Plane.

## Practical Implementation Examples

### Using the UTF-8 Transform in the Compression Pipeline

```cpp
#include "kanzi/api/Compressor.hpp"
#include "kanzi/api/Decompressor.hpp"
#include "kanzi/Context.hpp"
#include <vector>

int main()
{
    // Sample multilingual text (Latin, Cyrillic, CJK, Arabic)
    const char* txt = u8"Hello, мир, 你好, مرحبا!";

    // Prepare input / output buffers
    kanzi::SliceArray<kanzi::byte> in((kanzi::byte*)txt,
                                      (int)strlen(txt), 0);
    std::vector<kanzi::byte> outBuf(2 * in._length);
    kanzi::SliceArray<kanzi::byte> out(outBuf.data(),
                                       (int)outBuf.size(), 0);

    // Create a context that forces the UTF-8 transform
    kanzi::Context ctx;
    ctx.putInt("dataType", kanzi::Global::UTF8);   // optional hint

    // Build a compressor with the UTF-8 transform in the pipeline
    kanzi::Compressor* comp = kanzi::Compressor::newInstance(
                               ctx, "utf", nullptr);
    comp->compress(in, out);          // forward() of UTFCodec is invoked

    // --- now decompress -------------------------------------------------
    kanzi::Decompressor* decomp = kanzi::Decompressor::newInstance(
                                 ctx, "utf", nullptr);
    kanzi::SliceArray<kanzi::byte> decoded(outBuf.data(),
                                           out._index, 0);
    std::vector<kanzi::byte> plain(in._length);
    kanzi::SliceArray<kanzi::byte> plainArr(plain.data(),
                                            (int)plain.size(), 0);

    decomp->decompress(decoded, plainArr);   // inverse() of UTFCodec

    // plainArr now holds the original UTF-8 bytes
}

```

The `newInstance("utf",…)` call selects the `UTFCodec` transform via the factory entry in [`src/transform/TransformFactory.hpp`](https://github.com/flanglet/kanzi-cpp/blob/main/src/transform/TransformFactory.hpp) (line 287).

### Manual Transform Invocation

```cpp
#include "kanzi/transform/UTFCodec.hpp"
#include "kanzi/Context.hpp"

int main()
{
    const char* data = u8"Привет мир! 🌍";   // contains Cyrillic + emoji
    int len = (int)strlen(data);

    kanzi::SliceArray<kanzi::byte> src((kanzi::byte*)data, len, 0);
    std::vector<kanzi::byte> encBuf(len + 8192);
    kanzi::SliceArray<kanzi::byte> dst(encBuf.data(),
                                       (int)encBuf.size(), 0);

    // No explicit context – the codec will auto-detect UTF-8
    kanzi::UTFCodec codec;
    bool ok = codec.forward(src, dst, len);   // true if block passes validation

    // Decode back
    kanzi::SliceArray<kanzi::byte> decSrc(encBuf.data(),
                                          dst._index, 0);
    std::vector<kanzi::byte> outBuf(len);
    kanzi::SliceArray<kanzi::byte> out(outBuf.data(),
                                       (int)outBuf.size(), 0);

    codec.inverse(decSrc, out, dst._index);
    // `out` now contains the original UTF-8 bytes.
}

```

## Summary

- **Validation first**: The transform rejects small blocks and validates UTF-8 legality using statistical byte-pair analysis in `UTFCodec::validate()`.
- **22-bit packing**: Code points are packed into 22-bit integers via `UTFCodec::pack()`, preserving full Unicode range support including 4-byte sequences.
- **Frequency aliasing**: Common symbols are mapped to 1-byte aliases and rare symbols to 2-byte aliases, reducing entropy for multilingual corpora.
- **Lossless reconstruction**: The `UTFCodec::inverse()` method rebuilds the exact alias table from stream headers and restores original bytes via `UTFCodec::unpack()`.

## Frequently Asked Questions

### Does the Kanzi UTF-8 transform support emoji and rare Unicode characters?

Yes. The `LEN_SEQ` table and `UTFCodec::pack()` handle all legal UTF-8 sequences including 4-byte encodings for emoji and characters above U+FFFF. The 22-bit packing scheme accommodates the full Unicode range up to U+10FFFF.

### How does the transform detect invalid UTF-8 sequences?

The `UTFCodec::validate()` method runs a fast statistical test on byte-pair frequencies to identify illegal patterns such as overlong encodings, forbidden start bytes (`0xC0`/`0xC1`), or code points exceeding U+10FFFF. Invalid blocks are rejected before processing.

### What is the minimum block size for UTF-8 processing in Kanzi?

The transform requires blocks of at least 1024 bytes. Smaller blocks are rejected immediately to ensure the frequency-based alias map remains statistically meaningful and does not exceed 10% of the block size.

### Is the UTF-8 transform reversible without data loss?

Yes. The transform is fully bijective. The alias map is embedded in the compressed stream header, allowing `UTFCodec::inverse()` to reconstruct the exact original byte sequence, including BOM markers and truncated boundary symbols.