Implementing Custom Transforms or Entropy Codecs in Kanzi: A Complete Guide

To implement custom transforms or entropy codecs in Kanzi, inherit from the Transform<byte> or EntropyEncoder/EntropyDecoder interfaces, implement the required pure virtual methods, and register your component in the respective factory class by adding a numeric type identifier and a creation case to the switch statements in TransformFactory or EntropyEncoderFactory/EntropyDecoderFactory.

Kanzi is a modular C++ compression library from the flanglet/kanzi-cpp repository that treats transforms (pre-processing stages) and entropy codecs (final bit-level coders) as runtime-discoverable plug-ins. Both component types are instantiated through factory classes that map short textual tokens to concrete implementations, allowing you to inject custom algorithms into the compression pipeline without modifying the core API.

Architecture of Kanzi's Plug-In System

The Transform Interface

All transforms must implement the Transform<T> template defined in src/Transform.hpp. The interface requires three pure virtual methods:

template <class T>
class Transform {
public:
    virtual bool forward (SliceArray<T>& src, SliceArray<T>& dst, int length) = 0;
    virtual bool inverse (SliceArray<T>& src, SliceArray<T>& dst, int length) = 0;
    virtual int  getMaxEncodedLength(int srcLen) const = 0;
    virtual ~Transform(){}
};

Critical constraint: Transforms must be stateless. As noted in the source comments, the forward and inverse methods must produce identical results regardless of how many parallel jobs invoke them, because Kanzi may process blocks concurrently.

Transform Factory Mechanics

The TransformFactory class in src/transform/TransformFactory.hpp manages transform discovery and chaining. It exposes three key static methods:

• getType(const char* tName) – parses a +-separated list of up to eight transform names and packs each 6-bit token into a uint64
• getName(uint64 functionType) – unpacks the token back to a display string
• newTransform(Context& ctx, uint64 functionType) – builds a TransformSequence by delegating to newToken

The private newToken method contains a switch-case that maps each TransformType enum value to its concrete class constructor (e.g., BWT_TYPE → new BWTBlockCodec). To add your own transform, you must append a case to this switch.

Entropy Codec Interfaces

Entropy components split into encoder and decoder interfaces defined in src/EntropyEncoder.hpp and src/EntropyDecoder.hpp:

class EntropyEncoder {
public:
    virtual int encode(const byte block[], uint blkptr, uint len) = 0;
    virtual OutputBitStream& getBitStream() const = 0;
    virtual void dispose() = 0;
    virtual ~EntropyEncoder(){}
};

The decoder interface mirrors this with decode(const byte block[], uint blkptr, uint len). Both require access to the underlying bit stream and a dispose() method for resource cleanup.

Entropy Factory Registration

Registration occurs in src/entropy/EntropyEncoderFactory.hpp and its decoder counterpart. Each factory provides:

• newEncoder(OutputBitStream& obs, Context& ctx, short entropyType) – switches on the numeric type ID
• getName(short entropyType) – returns the textual token (e.g., "HUFFMAN")
• getType(const char* name) – reverse lookup from string to numeric ID

When a Compressor or Decompressor is instantiated (see src/api/Compressor.cpp and src/api/Decompressor.cpp), these factories resolve the requested names into concrete object instances.

Step-by-Step: Adding a Custom Transform

1. Implement the Transform Class

Create a new header file (e.g., MyTransform.hpp) that inherits from kanzi::Transform<byte>. You must override forward, inverse, and getMaxEncodedLength. An optional constructor accepting Context& allows runtime configuration.

// MyTransform.hpp
#pragma once
#include "../Transform.hpp"

namespace kanzi {

class MyTransform FINAL : public Transform<byte> {
public:
    MyTransform() {}
    MyTransform(Context&) {}
    ~MyTransform() {}

    bool forward (SliceArray<byte>& src, SliceArray<byte>& dst, int length) override {
        for (int i = 0; i < length; ++i)
            dst._array[dst._index + i] = ~src._array[src._index + i];
        src._index += length;
        dst._index += length;
        return true;
    }

    bool inverse (SliceArray<byte>& src, SliceArray<byte>& dst, int length) override {
        return forward(src, dst, length);  // Bitwise NOT is self-inverse
    }

    int getMaxEncodedLength(int srcLen) const override { return srcLen; }
};

}

2. Register with TransformFactory

Edit src/transform/TransformFactory.hpp to wire your transform into the runtime:

1. Add an enum value near line 48 (choose an unused value between 0 and 63):

   MYTRANSFORM_TYPE = 23,

2. Map the name to ID in getTypeToken (around line 90):

   if (name == "MYTRANSFORM")
       return MYTRANSFORM_TYPE;

3. Instantiate the class in newToken (around line 124):

   case MYTRANSFORM_TYPE:
       return new MyTransform(ctx);

4. Provide the display name in getNameToken (around line 332):

   case MYTRANSFORM_TYPE:
       return "MYTRANSFORM";

3. Activate the Transform

Pass the token name via a Context object when creating the compressor:

kanzi::Context ctx;
ctx.putString("transform", "MYTRANSFORM");  // Or "BWT+MYTRANSFORM" for chaining
kanzi::Compressor cmp(&outputStream, &ctx);

Step-by-Step: Adding a Custom Entropy Codec

1. Implement Encoder and Decoder

Create paired classes that handle the bit-level I/O. The encoder writes to an OutputBitStream, while the decoder reads from an InputBitStream.

// MyEntropyEncoder.hpp
#pragma once
#include "../EntropyEncoder.hpp"

namespace kanzi {

class MyEntropyEncoder FINAL : public EntropyEncoder {
public:
    explicit MyEntropyEncoder(OutputBitStream& obs) : _obs(obs) {}
    ~MyEntropyEncoder() {}

    int encode(const byte block[], uint blkptr, uint len) override {
        for (uint i = 0; i < len; ++i)
            _obs.writeByte(block[blkptr + i]);
        return len;
    }

    OutputBitStream& getBitStream() const override { return _obs; }
    void dispose() override {}
private:
    OutputBitStream& _obs;
};

}

// MyEntropyDecoder.hpp
#pragma once
#include "../EntropyDecoder.hpp"

namespace kanzi {

class MyEntropyDecoder FINAL : public EntropyDecoder {
public:
    explicit MyEntropyDecoder(InputBitStream& ibs) : _ibs(ibs) {}
    ~MyEntropyDecoder() {}

    int decode(byte block[], uint blkptr, uint len) override {
        for (uint i = 0; i < len; ++i)
            block[blkptr + i] = _ibs.readByte();
        return len;
    }

    InputBitStream& getBitStream() const override { return _ibs; }
    void dispose() override {}
private:
    InputBitStream& _ibs;
};

}

2. Register with Entropy Factories

Update both src/entropy/EntropyEncoderFactory.hpp and src/entropy/EntropyDecoderFactory.hpp:

1. Define a type constant (around line 60, use a value < 64):

   static const short MYENT_TYPE = 16;

2. Add name mappings in getName and getType:

   // In getName:
   case MYENT_TYPE: return "MYENTROPY";
   
   // In getType:
   if (name == "MYENTROPY") return MYENT_TYPE;

3. Add creation cases in newEncoder and newDecoder:

   // EntropyEncoderFactory
   case MYENT_TYPE: return new MyEntropyEncoder(obs);
   
   // EntropyDecoderFactory
   case MYENT_TYPE: return new MyEntropyDecoder(ibs);

3. Configure the Pipeline

Select your codec via the context string "entropy":

kanzi::Context ctx;
ctx.putString("entropy", "MYENTROPY");
kanzi::Compressor cmp(&outputStream, &ctx);

Complete Minimal Example

The following program demonstrates integrating both a custom transform and a custom entropy codec into a single compression job:

#include "api/kanzi.h"
#include "MyTransform.hpp"
#include "MyEntropyEncoder.hpp"
#include "MyEntropyDecoder.hpp"

int main() {
    // Assume outStream and inStream are properly initialized 
    // kanzi::FileOutputStream / FileInputStream instances
    
    kanzi::Context ctx;
    ctx.putString("transform", "MYTRANSFORM");
    ctx.putString("entropy", "MYENTROPY");

    // Compression
    kanzi::Compressor compressor(&outStream, &ctx);
    compressor.compress(inputBuffer, inputSize);

    // Decompression
    kanzi::Decompressor decompressor(&inStream, &ctx);
    decompressor.decompress(outputBuffer, outputSize);
    
    return 0;
}

When executed, this invokes the custom bit-invert transform followed by the raw-byte entropy pass, verifying that your plug-ins are first-class pipeline citizens.

Key Source Files Reference

Purpose	File Path
Generic transform interface	src/Transform.hpp
Transform factory and token parsing	src/transform/TransformFactory.hpp
Example built-in transform (Null)	src/transform/NullTransform.hpp
Entropy encoder interface	src/EntropyEncoder.hpp
Entropy decoder interface	src/EntropyDecoder.hpp
Encoder factory and registration	src/entropy/EntropyEncoderFactory.hpp
Decoder factory and registration	src/entropy/EntropyDecoderFactory.hpp
Example built-in codec (Huffman)	src/entropy/HuffmanEncoder.hpp
Public API entry points	src/api/Compressor.cpp, src/api/Decompressor.cpp

Summary

• Implement the interface: Inherit from kanzi::Transform<byte> for pre-processing stages or EntropyEncoder/EntropyDecoder for bit-level coders, overriding all pure virtual methods.
• Maintain statelessness: Ensure forward/inverse methods produce deterministic results independent of parallel job count.
• Register the token: Add a unique numeric ID (0–63) to the factory enums, map the textual name in getType/getName, and instantiate your class in the newToken or newEncoder/newDecoder switch cases.
• Activate at runtime: Pass the token string (e.g., "MYTRANSFORM" or "MYENTROPY") through a kanzi::Context object when constructing the compressor or decompressor.

Frequently Asked Questions

Do custom transforms need to be thread-safe?

Yes. According to the source code comments in src/Transform.hpp, transforms must be stateless and thread-safe. Kanzi may invoke forward or inverse from multiple threads concurrently during block-based parallel compression, so your implementation must not rely on mutable instance state between calls.

What is the maximum number of transforms allowed in a sequence?

Kanzi supports up to eight transforms chained in a single TransformSequence. The TransformFactory::getType method packs each transform token into 6 bits of a uint64, limiting the sequence length to eight components. Attempting to register more than eight in a single "NAME+NAME+..." string will result in parsing errors.

How do I choose a type identifier for my codec?

Select an unused integer between 0 and 63 for transforms (6-bit space) and between 0 and 32767 for entropy codecs (short integer). Check the existing TransformType enum in TransformFactory.hpp or the HUFFMAN_TYPE, ANS_TYPE, etc., constants in the entropy factories to avoid collisions. The value is arbitrary as long as it is unique within the respective factory.

Can I use the Context object to pass configuration parameters?

Yes. Both TransformFactory::newToken and the entropy factory methods accept a Context& parameter, which is a key-value store. Your transform or codec constructor can accept Context& to read settings (e.g., ctx.getInt("myParam")). This allows runtime configuration without recompiling the library.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}