TPAQ vs TPAQX: Choosing Neural Network-Based Compression in Kanzi

Use TPAQ for balanced speed and compression with moderate memory usage, and reserve TPAQX for maximum compression ratio when you can allocate twice the RAM and accept slower encoding speeds.

The Kanzi C++ compression library provides two neural network-based entropy codecs that deliver state-of-the-art compression through context mixing. Both variants are implemented in the TPAQPredictor template class in flanglet/kanzi-cpp, differing primarily in context depth, memory allocation strategy, and secondary estimation layers. Understanding the architectural distinction between TPAQ and TPAQX ensures you select the optimal trade-off between compression ratio, speed, and memory consumption for your specific workload.

Core Architectural Differences

Prediction Model and Context Inputs

The fundamental distinction lies in the number of contexts fed into the neural network mixer. In src/entropy/TPAQPredictor.hpp, the template class TPAQPredictor<bool T> selects the operational mode at compile time.

TPAQ (T = false) feeds seven inputs into the mixer: six neural network inputs derived from contexts 0 through 5, plus a match length predictor.

TPAQX (T = true) feeds eight inputs by adding a seventh context _ctx6 that combines two extra hash-derived signatures from recent bytes (h1, h2). This additional context is computed only when the template parameter is true【TPAQPredictor.hpp line 40‑46】, providing the mixer with more temporal information about the input stream.

Memory Footprint and Scaling

Memory allocation doubles between the two variants through the extraMem flag set in the constructor:

extraMem = (T == true) ? 1 : 0;

【TPAQPredictor.hpp line 44‑45】

This flag expands the size of state tables (_statesMask, _mixersMask, _hashMask, _bufferMask) by a factor of two.

• TPAQ: Approximately 0.5 GiB for a 4 MiB block (extraMem = 0)
• TPAQX: Approximately 1 GiB for the same block size (extraMem = 1)

Secondary Symbol Estimation

TPAQX activates an additional probability refinement stage. While both codecs use a primary secondary symbol estimator (SSE), TPAQX instantiates a second estimator _sse1 to further polish the mixed probability before entropy coding【TPAQPredictor.hpp line 111‑114】. This extra computation contributes to both the improved compression ratio and the reduced throughput of TPAQX.

Performance Characteristics and Selection Guide

Compression Ratio vs. Speed

TPAQ delivers very good compression, particularly on text-heavy data, while maintaining higher throughput and lower cache pressure. TPAQX typically achieves 1%–2% better compression on heterogeneous or highly compressible data such as source code repositories and binary distributions, but at the cost of significant speed reduction due to extra context lookups and larger table traversals.

When to Use TPAQ

Select TPAQ in the following scenarios:

• Memory constraints: Systems with ≤ 1 GiB available RAM for compression tasks
• Speed-critical applications: Real-time streaming, on-the-fly compression, or network protocols
• General workloads: Default choice for mixed data types where compression speed matters as much as ratio

When to Use TPAQX

Reserve TPAQX for these situations:

• Maximum archival compression: Static datasets, software distribution packages, or long-term storage where every byte counts
• Heterogeneous data: Files containing mixed text and binary patterns that benefit from the extra hash context
• Sufficient resources: When the system can accommodate the doubled memory footprint without paging

Implementation Details

The predictor is defined as a template where the boolean T toggles "X" mode【TPAQPredictor.hpp line 59‑61】. During the bit-update step, the mixing stage invokes _mixer->get with seven arguments for TPAQ and eight for TPAQX (including the extra match-length predictor p7 which is always present)【TPAQPredictor.hpp line 101‑108】.

Kanzi exposes these codecs through compression levels in the command-line interface:

8 = EXE+RLT+TEXT+UTF+DNA&TPAQ
9 = EXE+RLT+TEXT+UTF+DNA&TPAQX

【Kanzi.cpp line 141‑146】

Level 8 maps to TPAQ, while level 9 automatically selects TPAQX with its maximum compression configuration.

Code Examples

Command-Line Selection

Select the entropy codec explicitly or via compression level:

# Fast compression with moderate memory usage (TPAQ)

kanzi -c -i input.bin -e TPAQ -b 8m -l 8

# Maximum compression with higher memory usage (TPAQX)

kanzi -c -i input.bin -e TPAQX -b 8m -l 9

The -e flag sets the entropy codec directly, while -l 8 and -l 9 map to TPAQ and TPAQX respectively according to the help output in src/app/Kanzi.cpp.

Programmatic Instantiation

Instantiate the predictor templates directly in C++:

#include "TPAQPredictor.hpp"
#include "BinaryEntropyEncoder.hpp"

using namespace kanzi;

// Classic TPAQ (moderate memory, faster)
Predictor* predTPAQ = new TPAQPredictor<false>(nullptr);

// Extended TPAQX (double memory, maximum compression)
Predictor* predTPAQX = new TPAQPredictor<true>(nullptr);

// Use with a binary entropy encoder
BinaryEntropyEncoder enc(outputStream, predTPAQ, true);
// or for maximum compression:
BinaryEntropyEncoder encX(outputStream, predTPAQX, true);

The template argument false or true directly selects the mode at compile time【TPAQPredictor.hpp line 59‑61】.

Benchmarking Both Variants

The test harness in src/test/TestEntropyCodec.cpp allows direct comparison:

if (type.compare("TPAQ") == 0)
    return new TPAQPredictor<false>();
if (type.compare("TPAQX") == 0)
    return new TPAQPredictor<true>();

Running ./TestEntropyCodec reports compression ratio and throughput for both variants, enabling data-driven selection for your specific corpus.

Summary

• TPAQ uses 7 mixer inputs and ~0.5 GiB per 4 MiB block, offering the best speed-to-compression ratio for general use.
• TPAQX adds an 8th context input (_ctx6), doubles memory to ~1 GiB, and activates a second SSE (_sse1) for 1%–2% better compression on complex data.
• The TPAQPredictor<bool T> template selects the variant at instantiation; T = false selects TPAQ, T = true selects TPAQX.
• Command-line levels 8 (TPAQ) and 9 (TPAQX) in src/app/Kanzi.cpp provide convenient presets.
• Choose TPAQ for limited RAM or speed-critical paths; choose TPAQX for archival storage with abundant memory.

Frequently Asked Questions

What is the exact memory difference between TPAQ and TPAQX?

TPAQX consumes twice the RAM of TPAQ for equivalent block sizes. According to the constructor logic in src/entropy/TPAQPredictor.hpp, TPAQX sets extraMem = 1 while TPAQ uses extraMem = 0, doubling the allocation for state tables, mixers, and hash buffers【TPAQPredictor.hpp line 44‑45】. For a typical 4 MiB block, expect approximately 0.5 GiB for TPAQ and 1 GiB for TPAQX.

Can I use TPAQX on systems with limited RAM?

You should avoid TPAQX on memory-constrained systems. The doubled footprint (up to 1 GiB for 4 MiB blocks) increases cache pressure and can lead to swapping, negating the compression benefits. TPAQ provides nearly equivalent compression with half the memory requirements, making it the safer choice for embedded or shared environments.

How much compression improvement does TPAQX actually provide?

TPAQX typically achieves 1%–2% better compression than TPAQ on heterogeneous or highly compressible data such as source code repositories, mixed binary/text files, and structured data formats. On purely text-heavy or random data, the improvement diminishes, making the speed and memory penalties less worthwhile.

Which compression levels in Kanzi map to TPAQ and TPAQX?

Compression level 8 maps to TPAQ, and level 9 maps to TPAQX. This mapping appears in the help output generated by src/app/Kanzi.cpp, which lists 8 = EXE+RLT+TEXT+UTF+DNA&TPAQ and 9 = EXE+RLT+TEXT+UTF+DNA&TPAQX【Kanzi.cpp line 141‑146】. You can also invoke either codec explicitly using the -e TPAQ or -e TPAQX entropy flag regardless of compression level.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}