How to Set Advanced Compression Parameters in Zstandard (zstd)

Zstandard provides an experimental advanced compression API through ZSTD_CCtx_setParameter() that enables fine-grained control over memory budgets, search heuristics, and multi-threading when compiled with ZSTD_STATIC_LINKING_ONLY.

Zstandard (zstd) offers significantly more flexibility than simple integer compression levels. The facebook/zstd repository exposes an advanced interface that allows developers to configure internal compression parameters in zstd, optimizing the trade-off between speed, memory usage, and compression ratio. This API requires explicit opt-in via compile-time macros and provides sticky parameter contexts that persist across multiple compression operations.

Enabling the Experimental API

Advanced parameters are guarded by the ZSTD_STATIC_LINKING_ONLY macro to prevent accidental use with dynamically-linked libraries. According to the source code in lib/zstd.h, you must define this macro before including the header to access the extended parameter enums and functions.

#define ZSTD_STATIC_LINKING_ONLY
#include <zstd.h>

This unlocks the ZSTD_cParameter enum and the ZSTD_CCtx_setParameter() function, both of which are marked as experimental APIs in the library.

Creating a Compression Context

Advanced parameters are stored within a compression context object. Use ZSTD_createCCtx() for one-shot operations or ZSTD_createCStream() for streaming scenarios. Both functions return a ZSTD_CCtx* pointer that maintains your configured settings across multiple frames.

ZSTD_CCtx *cctx = ZSTD_createCCtx();
if (!cctx) { /* handle allocation failure */ }

The context stores parameter values internally, applying them automatically when compression begins. This design separates configuration from execution, allowing you to set parameters once and compress multiple inputs with consistent settings.

Configuring Advanced Compression Parameters

Call ZSTD_CCtx_setParameter() with a context pointer, a ZSTD_cParameter enum value, and an integer argument. The full list of available parameters is defined in lib/zstd.h around line 3550.

Window Size and Memory Budget

ZSTD_c_windowLog controls the maximum back-reference distance (window size), directly affecting memory consumption during decompression. Valid values range from 10 to 30, representing 1 KiB to 1 GiB respectively.

ZSTD_CCtx_setParameter(cctx, ZSTD_c_windowLog, 24);  /* 16 MiB window */

Hash Table and Search Configuration

The match-finder behavior is controlled through several interrelated parameters:

• ZSTD_c_hashLog – Sets the size of the hash table used for dictionary lookups. Larger values improve compression ratio at the cost of memory.
• ZSTD_c_chainLog – Configures the size of the chain table for the "double hash" strategy.
• ZSTD_c_searchLog – Determines the number of search attempts the algorithm will make when looking for matches.
• ZSTD_c_minMatch – Sets the minimum match length in bytes.

ZSTD_CCtx_setParameter(cctx, ZSTD_c_hashLog, 20);      /* 1 MiB hash table */
ZSTD_CCtx_setParameter(cctx, ZSTD_c_searchLog, 6);     /* 64 search attempts */
ZSTD_CCtx_setParameter(cctx, ZSTD_c_minMatch, 4);      /* 4-byte minimum match */

Compression Strategy

ZSTD_c_strategy selects the core algorithm from the ZSTD_strategy enum defined in lib/compress/zstd_opt.h. Options range from ZSTD_fast for speed-focused compression to ZSTD_btultra2 for maximum ratio.

ZSTD_CCtx_setParameter(cctx, ZSTD_c_strategy, ZSTD_btultra);

Available strategies include ZSTD_fast, ZSTD_dfast, ZSTD_greedy, ZSTD_lazy, ZSTD_lazy2, ZSTD_btlazy2, ZSTD_btopt, ZSTD_btultra, and ZSTD_btultra2.

Parallel Compression

ZSTD_c_nbWorkers enables multi-threaded compression by specifying the number of worker threads. This parameter is implemented in lib/compress/zstdmt_compress.h and requires the library to be compiled with multithreading support.

ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4);  /* Use 4 threads */

Frame Integrity

ZSTD_c_checksumFlag enables a 32-bit XXH64-based checksum stored in the frame header, allowing decompression functions to verify data integrity.

ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 1);

Declaring Source Size

If the input size is known before compression begins, declare it using ZSTD_CCtx_setPledgedSrcSize(). This allows the encoder to optimize internal buffer allocations and write the correct uncompressed size into the frame header, which aids decompressors in pre-allocating output buffers.

size_t const srcSize = 1024 * 1024;  /* 1 MiB */
ZSTD_CCtx_setPledgedSrcSize(cctx, srcSize);

Setting the pledged source size is particularly beneficial for streaming compression, as it enables more efficient block splitting and memory management within the internal structures defined in lib/compress/zstd_compress_internal.h.

Executing Compression

Parameters remain sticky until explicitly modified or reset. You can perform compression using either one-shot or streaming APIs.

One-Shot Compression

Use ZSTD_compress2() to compress an entire buffer at once, utilizing all previously set parameters:

#include <stdlib.h>

int compress_buffer(ZSTD_CCtx *cctx, 
                    void *dst, size_t dstCapacity,
                    const void *src, size_t srcSize)
{
    /* Parameters already set in cctx */
    size_t const cSize = ZSTD_compress2(cctx, dst, dstCapacity, src, srcSize);
    
    if (ZSTD_isError(cSize)) {
        fprintf(stderr, "Compression error: %s\n", 
                ZSTD_getErrorName(cSize));
        return 1;
    }
    return 0;
}

Streaming Compression

For large or continuous data, use the streaming API. The context type is interchangeable (ZSTD_CStream is an alias for ZSTD_CCtx), and parameters persist across ZSTD_compressStream2() calls:

void stream_compress(ZSTD_CCtx *cctx, FILE *in, FILE *out)
{
    size_t const inBufSize = ZSTD_CStreamInSize();
    size_t const outBufSize = ZSTD_CStreamOutSize();
    
    void *const src = malloc(inBufSize);
    void *const dst = malloc(outBufSize);
    
    ZSTD_inBuffer input = { src, 0, 0 };
    ZSTD_outBuffer output = { dst, outBufSize, 0 };
    
    while (!feof(in)) {
        input.size = fread(src, 1, inBufSize, in);
        input.pos = 0;
        
        while (input.pos < input.size) {
            size_t const ret = ZSTD_compressStream2(
                cctx, &output, &input, ZSTD_e_continue);
            
            if (ZSTD_isError(ret)) { /* handle error */ }
            
            fwrite(dst, 1, output.pos, out);
            output.pos = 0;
        }
    }
    
    /* Flush remaining data and end frame */
    size_t remaining;
    do {
        remaining = ZSTD_compressStream2(
            cctx, &output, &input, ZSTD_e_end);
        fwrite(dst, 1, output.pos, out);
        output.pos = 0;
    } while (remaining > 0);
    
    free(src);
    free(dst);
}

Resetting Context State

When reusing a context for different compression jobs, clear previous settings using ZSTD_CCtx_reset() with a ZSTD_ResetDirective:

• ZSTD_reset_session_only – Clears internal compression state but preserves parameters.
• ZSTD_reset_parameters – Clears all set parameters but keeps the session state.
• ZSTD_reset_session_and_parameters – Returns the context to a pristine state.

ZSTD_CCtx_reset(cctx, ZSTD_reset_session_and_parameters);

This mechanism is frequently utilized in the library's test suite; see tests/zstreamtest.c for practical examples of context reuse with parameter resets.

Summary

• Enable experimental features by defining ZSTD_STATIC_LINKING_ONLY before including lib/zstd.h to access advanced compression parameters in zstd.
• Allocate a context using ZSTD_createCCtx() to store sticky parameters that persist across compression operations.
• Configure parameters such as ZSTD_c_windowLog, ZSTD_c_strategy, and ZSTD_c_nbWorkers using ZSTD_CCtx_setParameter().
• Declare source size with ZSTD_CCtx_setPledgedSrcSize() when possible to optimize memory usage and frame headers.
• Compress data using ZSTD_compress2() for one-shot operations or ZSTD_compressStream2() for streaming pipelines.
• Reset contexts via ZSTD_CCtx_reset() to clear state or parameters between unrelated compression jobs.

Frequently Asked Questions

What is the difference between ZSTD_compress() and ZSTD_compress2()?

ZSTD_compress() is the simple API that takes a compression level integer and internally creates a temporary context. ZSTD_compress2() requires a pre-configured ZSTD_CCtx and uses whatever advanced parameters you have set via ZSTD_CCtx_setParameter(), giving you full control over the compression behavior.

Do I need to link against special libraries to use multi-threading?

Multi-threading support via ZSTD_c_nbWorkers requires that the zstd library was compiled with threading enabled (typically using pthreads on POSIX systems). The implementation resides in lib/compress/zstdmt_compress.h. If the library was built without multithreading support, setting this parameter will return an error.

Can I change parameters between frames in a streaming session?

Yes. You can modify parameters between calls to ZSTD_compressStream2() as long as you are at a frame boundary (after ZSTD_e_end). Alternatively, call ZSTD_CCtx_reset(cctx, ZSTD_reset_parameters) to clear existing settings and push new ones before starting the next frame.

Why are advanced parameters marked as experimental?

Advanced parameters are marked experimental in lib/zstd.h to allow the facebook/zstd maintainers to modify enum values or behaviors in future releases without breaking the stable ABI. By requiring ZSTD_STATIC_LINKING_ONLY, the library ensures that applications using these features are statically linked and can be recompiled against newer versions rather than relying on dynamic linking compatibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}