How to Ensure Data Integrity with Kanzi's 32-bit vs 64-bit Checksums

Kanzi-CPP detects data corruption at the block level by computing XXHash32 or XXHash64 checksums on transformed data during compression and validating them during decompression, aborting with ERR_CRC_CHECK if any mismatch occurs.

Kanzi-CPP (flanglet/kanzi-cpp) implements optional block-level checksums to guarantee data integrity across compression and decompression cycles. When you ensure data integrity with Kanzi's 32-bit vs 64-bit checksums, you select between fast 32-bit validation or stronger 64-bit collision resistance, each computed on post-transform block data and embedded directly into the bitstream.

Checksum Options: 32-bit vs 64-bit

Kanzi-CPP provides three integrity modes controlled by the checksum context parameter:

• XXHash32 (32-bit): Adds 4 bytes per block. Provides 32-bit collision resistance suitable for most general-purpose workloads and error detection.
• XXHash64 (64-bit): Adds 8 bytes per block. Offers significantly stronger collision resistance (2⁶⁴ space) with negligible performance overhead compared to the 32-bit variant.
• No checksum (0): Adds zero overhead. Disables integrity checking entirely.

Both hash implementations are non-cryptographic and optimized for speed, processing data at approximately 10 GB/s on modern CPUs.

Checksum Generation During Compression

The compression pipeline computes checksums after the forward transform stage but before entropy coding. This ensures the hash covers the actual transformed block content.

Hasher Instantiation

In src/io/CompressedOutputStream.cpp, the constructor and submitBlock() method instantiate the appropriate hasher based on the checksum context value:

// src/io/CompressedOutputStream.cpp
if (checksum == 0) {
   _hasher32 = nullptr;
   _hasher64 = nullptr;
}
else if (checksum == 32) {
   _hasher32 = new XXHash32(BITSTREAM_TYPE);
   _hasher64 = nullptr;
}
else if (checksum == 64) {
   _hasher32 = nullptr;
   _hasher64 = new XXHash64(BITSTREAM_TYPE);
}

Block-Level Hashing

During block processing, EncodingTask::run() computes the hash on the transformed block data:

// src/io/CompressedOutputStream.cpp – inside EncodingTask::run()
if (_hasher32 != nullptr) {
    checksum = _hasher32->hash(&_data->_array[_data->_index], blockLength);
    hashType = Event::SIZE_32;
}
else if (_hasher64 != nullptr) {
    checksum = _hasher64->hash(&_data->_array[_data->_index], blockLength);
    hashType = Event::SIZE_64;
}

Bitstream Integration

The computed checksum value is written immediately after the block header:

// src/io/CompressedOutputStream.cpp
if (_hasher32 != nullptr)
    obs.writeBits(checksum, 32);
else if (_hasher64 != nullptr)
    obs.writeBits(checksum, 64);

Checksum Verification During Decompression

During decompression, CompressedInputStream::readHeader() reads the checksum size from the stream header and reconstructs the corresponding hasher. The decoder validates integrity by recomputing the hash on the post-entropy-decode data and comparing it against the stored value.

Runtime Validation

Inside DecodingTask::run() in src/io/CompressedInputStream.cpp, the decoder verifies each block:

// src/io/CompressedInputStream.cpp – inside DecodingTask::run()
if (_hasher32 != nullptr) {
    const uint32 checksum2 = _hasher32->hash(&_data->_array[savedIdx], decoded);
    if (checksum2 != uint32(checksum1))
        return T(*_data, blockId, decoded, checksum1, Error::ERR_CRC_CHECK,
                 "Corrupted bitstream …");
}
else if (_hasher64 != nullptr) {
    const uint64 checksum2 = _hasher64->hash(&_data->_array[savedIdx], decoded);
    if (checksum2 != checksum1)
        return T(*_data, blockId, decoded, checksum1, Error::ERR_CRC_CHECK,
                 "Corrupted bitstream …");
}

If the recomputed hash differs from the stored value, the decoder immediately returns Error::ERR_CRC_CHECK, aborting further processing and preventing corrupted data from reaching the output.

Configuring Checksum Size

You specify the checksum size through the Context object or command-line arguments. Valid values are 0, 32, or 64; any other value causes the constructors to throw invalid_argument before I/O begins.

Command-Line Interface

From the terminal, use the -x32 or -x64 flags parsed in src/app/Kanzi.cpp:

./kanzi -c input.txt output.knz -x64

Programmatic Configuration

For library integration, set the checksum parameter in the context:

#include "kanzi/Context.hpp"
#include "kanzi/BlockCompressor.hpp"

int main() {
    kanzi::Context ctx;
    ctx.putString("entropy", "HUFFMAN");
    ctx.putString("transform", "LZX");
    ctx.putInt("checksum", 32);               // Enable 32-bit checksum
    ctx.putInt("blockSize", 4 * 1024 * 1024); // 4 MiB blocks

    kanzi::BlockCompressor compressor(ctx);
    uint64 outSize = 0;
    int rc = compressor.compress(outSize);
    return rc;
}

The BlockCompressor constructor forwards this value to CompressedOutputStream, which instantiates the appropriate XXHash implementation.

Decompression Configuration

To verify checksums during decompression, ensure the context matches the stream's header:

#include "kanzi/Context.hpp"
#include "kanzi/Decompressor.hpp"

int main() {
    kanzi::Context ctx;
    ctx.putInt("checksum", 64);   // Expect 64-bit checksums
    kanzi::Decompressor d(ctx);
    d.decompress("output.knz", "recovered.txt");
}

Performance Characteristics

Because Kanzi computes the hash once per block, the performance impact scales with the number of blocks rather than total file size. Both XXHash32 and XXHash64 achieve throughput exceeding 10 GB/s on modern hardware, making them suitable for high-performance compression pipelines.

The 32-bit variant consumes half the storage overhead (4 bytes vs. 8 bytes) and executes marginally faster per block, while the 64-bit variant provides exponentially larger collision resistance at a negligible computational cost. The hashing occurs on transformed data, ensuring integrity covers the actual compressed payload.

Summary

• Kanzi-CPP offers three integrity modes: 32-bit (XXHash32), 64-bit (XXHash64), or disabled.
• Checksums are computed on post-transform data during compression and written to the bitstream after the block header.
• The decoder validates hashes during decompression in DecodingTask::run(), returning ERR_CRC_CHECK on any mismatch.
• Configure via Context (checksum = 0|32|64) or CLI flags (-x32, -x64).
• Implementation files: src/io/CompressedOutputStream.cpp, src/io/CompressedInputStream.cpp, and src/util/XXHash.hpp.

Frequently Asked Questions

What happens if Kanzi detects a checksum mismatch during decompression?

The decoder aborts immediately. In src/io/CompressedInputStream.cpp, DecodingTask::run() returns Error::ERR_CRC_CHECK, which propagates as an IOException to the caller. This prevents corrupted data from being written to the output file or buffer.

How do I choose between 32-bit and 64-bit checksums in Kanzi?

Select 32-bit (-x32) for general-purpose use where 4 bytes of overhead per block is acceptable and 32-bit collision resistance is sufficient. Select 64-bit (-x64) when you require stronger integrity guarantees or work with extremely large datasets where collision probability must be minimized, accepting 8 bytes of overhead per block.

Are Kanzi's checksums cryptographically secure?

No. Kanzi uses XXHash32 and XXHash64, which are non-cryptographic hash functions designed for speed and error detection, not security. They detect accidental corruption (bit flips, transmission errors) but should not be used to verify authenticity or protect against malicious tampering.

Can I disable checksums entirely in Kanzi-CPP?

Yes. Set the checksum context parameter to 0 or omit the -x32/-x64 flags. This eliminates the per-block overhead and computational cost of hashing, though it removes all integrity checking from the compressed stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}