How Kanzi’s UTF-8 Transform Handles International Character Data

The Kanzi UTF-8 transform validates Unicode text against statistical patterns, packs code points into 22-bit integers, and replaces frequent symbols with short 1- or 2-byte aliases to minimize entropy while guaranteeing lossless reconstruction of multilingual data.

Kanzi is a high-performance compression library that treats UTF-8 encoding as a reversible preprocessing step. The UTFCodec class, found in src/transform/UTFCodec.cpp, implements this transform by validating international character sequences and applying frequency-based symbol aliasing before entropy coding begins.

Stage 1: Input Validation and BOM Handling

The transform begins in UTFCodec::forward() (lines 48‑90 in src/transform/UTFCodec.cpp) by enforcing strict entry criteria that protect downstream compression stages from corrupt data.

• Minimum block size: Blocks smaller than 1024 bytes are rejected immediately to ensure statistical significance.
• BOM stripping: If a UTF-8 Byte-Order-Mark (0xEF 0xBB 0xBF) is detected, the transform removes it and sets the start offset to 3 bytes.
• Context-aware filtering: When a Context object is provided, the codec checks the stored dataType. If the type is explicitly set and not UTF8, the block is rejected without processing.
• Statistical validation: For unknown data types, UTFCodec::validate() (lines 305‑424) performs a fast byte-pair frequency analysis to confirm the block follows legal UTF-8 patterns defined by the Unicode 16 standard, rejecting overlong encodings or invalid start bytes like 0xC0/0xC1.

Stage 2: Symbol Packing and Alias Map Construction

Once validated, the transform constructs a frequency-based alias table to shorten common Unicode values. This occurs in UTFCodec::forward() (lines 110‑176).

Symbol packing: Each UTF-8 code point is packed into a 22-bit integer by UTFCodec::pack() (lines 70‑102 in src/transform/UTFCodec.hpp). The high-order bits encode the byte-length (1, 2, 3, or 4), while the low-order bits store the Unicode scalar value.

Frequency mapping: While scanning the block, the transform counts every packed value in an aliasMap. New code points trigger the creation of sdUTF entries (value + frequency) in a temporary vector v.

Alias generation: After the scan, the transform retains only symbols that appear frequently enough to keep the map size below approximately 10% of the block size. These symbols are sorted by decreasing frequency to create a rank-ordered alias table. The first 128 symbols receive 1-byte aliases; all others receive 2-byte aliases.

Stage 3: Block Encoding and Decoding

The encoding phase writes a compact binary representation that preserves all international character data exactly.

Header structure: The output begins with two header bytes indicating the start offset and the count of invalid/truncated symbols at the block boundaries.

Alias emission: Pre-BOM bytes and truncated start symbols are copied verbatim. Every validated Unicode symbol is then replaced by its alias, emitted as a little-endian integer (low byte first, high byte only if alias ≥ 0x100). Truncated symbols at the end are copied verbatim.

Decoding: The inverse operation in UTFCodec::inverse() (lines 208‑297) reads the header, rebuilds the identical alias table from the map data, and expands each alias back to the original packed value using UTFCodec::unpack() (lines 113‑151 in src/transform/UTFCodec.hpp).

Design Advantages for International Text

The LEN_SEQ classification table (line 29 in src/transform/UTFCodec.hpp) enables full UTF-8 support by categorizing bytes into 1-, 2-, 3-, or 4-byte sequences. This allows the pack() routine to recognize any legal Unicode scalar from U+0000 to U+10FFFF, including language-specific characters like é, 汉, or अ.

By mapping frequently-used code points to short aliases, the transform dramatically reduces entropy for multilingual texts. The statistical validation step prevents malicious or corrupt data from entering the compression pipeline, while the embedded alias map ensures a lossless round-trip for all characters, including those outside the Basic Multilingual Plane.

Practical Implementation Examples

Using the UTF-8 Transform in the Compression Pipeline

#include "kanzi/api/Compressor.hpp"
#include "kanzi/api/Decompressor.hpp"
#include "kanzi/Context.hpp"
#include <vector>

int main()
{
    // Sample multilingual text (Latin, Cyrillic, CJK, Arabic)
    const char* txt = u8"Hello, мир, 你好, مرحبا!";

    // Prepare input / output buffers
    kanzi::SliceArray<kanzi::byte> in((kanzi::byte*)txt,
                                      (int)strlen(txt), 0);
    std::vector<kanzi::byte> outBuf(2 * in._length);
    kanzi::SliceArray<kanzi::byte> out(outBuf.data(),
                                       (int)outBuf.size(), 0);

    // Create a context that forces the UTF-8 transform
    kanzi::Context ctx;
    ctx.putInt("dataType", kanzi::Global::UTF8);   // optional hint

    // Build a compressor with the UTF-8 transform in the pipeline
    kanzi::Compressor* comp = kanzi::Compressor::newInstance(
                               ctx, "utf", nullptr);
    comp->compress(in, out);          // forward() of UTFCodec is invoked

    // --- now decompress -------------------------------------------------
    kanzi::Decompressor* decomp = kanzi::Decompressor::newInstance(
                                 ctx, "utf", nullptr);
    kanzi::SliceArray<kanzi::byte> decoded(outBuf.data(),
                                           out._index, 0);
    std::vector<kanzi::byte> plain(in._length);
    kanzi::SliceArray<kanzi::byte> plainArr(plain.data(),
                                            (int)plain.size(), 0);

    decomp->decompress(decoded, plainArr);   // inverse() of UTFCodec

    // plainArr now holds the original UTF-8 bytes
}

The newInstance("utf",…) call selects the UTFCodec transform via the factory entry in src/transform/TransformFactory.hpp (line 287).

Manual Transform Invocation

#include "kanzi/transform/UTFCodec.hpp"
#include "kanzi/Context.hpp"

int main()
{
    const char* data = u8"Привет мир! 🌍";   // contains Cyrillic + emoji
    int len = (int)strlen(data);

    kanzi::SliceArray<kanzi::byte> src((kanzi::byte*)data, len, 0);
    std::vector<kanzi::byte> encBuf(len + 8192);
    kanzi::SliceArray<kanzi::byte> dst(encBuf.data(),
                                       (int)encBuf.size(), 0);

    // No explicit context – the codec will auto-detect UTF-8
    kanzi::UTFCodec codec;
    bool ok = codec.forward(src, dst, len);   // true if block passes validation

    // Decode back
    kanzi::SliceArray<kanzi::byte> decSrc(encBuf.data(),
                                          dst._index, 0);
    std::vector<kanzi::byte> outBuf(len);
    kanzi::SliceArray<kanzi::byte> out(outBuf.data(),
                                       (int)outBuf.size(), 0);

    codec.inverse(decSrc, out, dst._index);
    // `out` now contains the original UTF-8 bytes.
}

Summary

• Validation first: The transform rejects small blocks and validates UTF-8 legality using statistical byte-pair analysis in UTFCodec::validate().
• 22-bit packing: Code points are packed into 22-bit integers via UTFCodec::pack(), preserving full Unicode range support including 4-byte sequences.
• Frequency aliasing: Common symbols are mapped to 1-byte aliases and rare symbols to 2-byte aliases, reducing entropy for multilingual corpora.
• Lossless reconstruction: The UTFCodec::inverse() method rebuilds the exact alias table from stream headers and restores original bytes via UTFCodec::unpack().

Frequently Asked Questions

Does the Kanzi UTF-8 transform support emoji and rare Unicode characters?

Yes. The LEN_SEQ table and UTFCodec::pack() handle all legal UTF-8 sequences including 4-byte encodings for emoji and characters above U+FFFF. The 22-bit packing scheme accommodates the full Unicode range up to U+10FFFF.

How does the transform detect invalid UTF-8 sequences?

The UTFCodec::validate() method runs a fast statistical test on byte-pair frequencies to identify illegal patterns such as overlong encodings, forbidden start bytes (0xC0/0xC1), or code points exceeding U+10FFFF. Invalid blocks are rejected before processing.

What is the minimum block size for UTF-8 processing in Kanzi?

The transform requires blocks of at least 1024 bytes. Smaller blocks are rejected immediately to ensure the frequency-based alias map remains statistically meaningful and does not exceed 10% of the block size.

Is the UTF-8 transform reversible without data loss?

Yes. The transform is fully bijective. The alias map is embedded in the compressed stream header, allowing UTFCodec::inverse() to reconstruct the exact original byte sequence, including BOM markers and truncated boundary symbols.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}