How to Enable Parallel Compression in Zstandard (zstd) with ZSTD_c_nbWorkers

Set the ZSTD_c_nbWorkers parameter on a ZSTD_CCtx compression context to a value ≥ 1 using ZSTD_CCtx_setParameter(), and the library will spawn a thread pool to compress input chunks concurrently.

Zstandard (zstd) is an open-source compression library developed by Meta that supports multi-threaded compression through the ZSTD_c_nbWorkers parameter. When enabled, the library divides input data into blocks and processes them across multiple CPU cores, significantly improving throughput on large datasets. This article explains how to configure parallel compression using the C API and command-line tools based on the facebook/zstd source code.

How Parallel Compression Works in zstd

Parallel compression in zstd operates by splitting the input stream into independent blocks that worker threads compress simultaneously. The main thread then aggregates these compressed blocks into the final output frame while preserving the standard zstd format.

The Compression Context and Parameter System

All runtime configuration for zstd compression resides in the ZSTD_CCtx structure defined in lib/zstd.h (line 469). The ZSTD_c_nbWorkers parameter is processed in lib/compress/zstd_compress.c within the parameter handling logic (case ZSTD_c_nbWorkers:), which validates the value and forwards it to the multi-threaded engine. According to the source at line 2920 of lib/zstd.h, parallel compression requires the "regular" compression mode and will fail if combined with an external sequence producer.

Thread Pool and Block Processing

When ZSTD_c_nbWorkers is set to ≥ 1, the library initializes a thread pool in lib/compress/zstdmt_compress.c (line 947) via ZSTD_CCtxParams_setParameter. By default, the input is partitioned into 128 KB blocks, with each worker thread running the single-threaded compressor on its assigned block. The main thread collects results in order and concatenates them into the final frame without altering the compressed output format.

Setting ZSTD_c_nbWorkers in C

You can enable workers through the simple context API or the advanced parameters interface. Both methods require linking against a zstd library compiled with multi-threading support.

Basic Context Configuration

The most straightforward approach uses ZSTD_CCtx_setParameter() immediately after creating the context:

#include <zstd.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

int main(void) {
    const char *src = "…large input data…";
    size_t srcSize = strlen(src);

    /* Create a compression context */
    ZSTD_CCtx *cctx = ZSTD_createCCtx();
    if (!cctx) return 1;

    /* Enable parallel compression with 4 worker threads */
    ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4);

    /* Allocate destination buffer */
    size_t dstCap = ZSTD_compressBound(srcSize);
    void *dst = malloc(dstCap);
    if (!dst) {
        ZSTD_freeCCtx(cctx);
        return 1;
    }

    /* Perform compression at level 3 */
    size_t compSize = ZSTD_compressCCtx(cctx, dst, dstCap,
                                        src, srcSize, 3);
    if (ZSTD_isError(compSize)) {
        fprintf(stderr, "Compression error: %s\n",
                ZSTD_getErrorName(compSize));
        free(dst);
        ZSTD_freeCCtx(cctx);
        return 1;
    }

    printf("Compressed %zu → %zu bytes using 4 workers\n",
           srcSize, compSize);

    ZSTD_freeCCtx(cctx);
    free(dst);
    return 0;
}

Key implementation detail: The parameter is validated in lib/compress/zstd_compress.c before the thread pool is allocated.

Using ZSTD_CCtxParams

For reusable configurations, set the parameter on a ZSTD_CCtx_params object as demonstrated in tests/zstreamtest.c (line 1507):

ZSTD_CCtx *cctx = ZSTD_createCCtx();
ZSTD_CCtx_params *params = ZSTD_createCCtxParams();

/* Configure compression level and worker count */
ZSTD_CCtxParams_setParameter(params, ZSTD_c_level, 5);
ZSTD_CCtxParams_setParameter(params, ZSTD_c_nbWorkers, 8);

/* Apply parameters to the context */
ZSTD_CCtx_setCParams(cctx, params);

/* Compression proceeds with 8 threads... */

ZSTD_freeCCtxParams(params);
ZSTD_freeCCtx(cctx);

Command-Line Usage with the -T Flag

The zstd CLI exposes parallel compression through the -T (or --threads) option, which maps directly to ZSTD_c_nbWorkers. The implementation in programs/fileio.c (line 1459) forwards this value to the compression context:

# Compress using 6 worker threads

zstd -T6 input.txt -o output.zst

# Use all available cores (0 means auto-detect)

zstd -T0 large-file.tar -o archive.tar.zst

Limitations and Considerations

Compilation Requirements

Parallel compression requires building zstd with multi-threading enabled. Calling ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, N) returns an error code if the library was compiled without thread support. Verify support by checking the return value or reviewing the build configuration.

Input Size and Overhead

Worker threads impose initialization overhead. For very small inputs (typically less than a few megabytes), the cost of thread creation may exceed the benefit of parallel processing. Keep ZSTD_c_nbWorkers set to 0 for small payloads to ensure single-threaded efficiency.

Incompatible Modes

As noted in the zstd header file (line 2920), ZSTD_c_nbWorkers is incompatible with external sequence producers. Attempting to use both features simultaneously will cause compression to fail with an error status.

Summary

• ZSTD_c_nbWorkers controls the number of threads used for compression.
• Set the parameter via ZSTD_CCtx_setParameter() or ZSTD_CCtxParams_setParameter() defined in lib/zstd.h.
• The implementation resides in lib/compress/zstdmt_compress.c, processing 128 KB blocks per worker.
• CLI users enable this with the -T flag handled in programs/fileio.c.
• Requires multi-threaded build; incompatible with external sequence producers.

Frequently Asked Questions

What is ZSTD_c_nbWorkers?

ZSTD_c_nbWorkers is a compression context parameter in the facebook/zstd library that specifies how many threads to use for parallel compression. When set to a value greater than zero, the library creates a pool of worker threads that simultaneously compress different blocks of the input data.

Does parallel compression change the output format?

No. The output format remains identical to single-threaded compression. The main thread reassembles compressed blocks in the correct order, producing a standard zstd frame that any compliant decompressor can read.

Why does ZSTD_CCtx_setParameter return an error?

The function returns an error if you attempt to set ZSTD_c_nbWorkers to a non-zero value when zstd was compiled without multi-threading support, or if you try to enable workers while using an external sequence producer (which is unsupported as of the current implementation).

How many workers should I use?

For optimal performance, set the number of workers equal to the number of physical CPU cores available on your machine. The CLI -T0 flag auto-detects this. Using more workers than cores typically yields diminishing returns due to context switching overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}