How to Load a Dictionary into Zstandard (zstd) Compression and Decompression Contexts

Zstandard provides three primary methods to load a dictionary: by-copy (ZSTD_CCtx_loadDictionary), by-reference (ZSTD_CCtx_loadDictionary_byReference), or by creating reusable pre-processed objects (ZSTD_CDict/ZSTD_DDict) that attach to contexts.

Loading a dictionary into zstd compression and decompression contexts enables significantly improved compression ratios on small or repetitive data by leveraging pre-trained statistical models. In the facebook/zstd repository, the public API exposes multiple loading strategies that trade memory ownership, performance, and thread-safety differently depending on your application's constraints.

Three Methods for Dictionary Loading

The zstd library implements distinct loading semantics to accommodate different memory management strategies and performance requirements.

Load-by-Copy

Use ZSTD_CCtx_loadDictionary() or ZSTD_DCtx_loadDictionary() when you need a private copy of the dictionary or plan to free the original buffer immediately after loading. According to the source in lib/compress/zstd_compress.c, this method allocates internal storage and copies the dictionary content into the context.

Load-by-Reference

Use ZSTD_CCtx_loadDictionary_byReference() or ZSTD_DCtx_loadDictionary_byReference() to avoid memory duplication. This stores only a pointer to your buffer, requiring you to maintain the dictionary's validity for the entire lifetime of the context.

Reusable Dictionary Objects

For maximum performance when compressing or decompressing with the same dictionary repeatedly, create ZSTD_CDict or ZSTD_DDict objects via ZSTD_createCDict() and ZSTD_createDDict(). These pre-process the dictionary once (building hash tables and entropy tables) and can be referenced by multiple contexts using ZSTD_CCtx_refCDict() or ZSTD_DCtx_refDDict().

Loading a Dictionary into a Compression Context

The implementation of ZSTD_CCtx_loadDictionary() resides in lib/compress/zstd_compress.c (lines 1300-1344), where it handles buffer validation and delegates to the advanced loading routine.

#include <zstd.h>
#include <stdio.h>
#include <stdlib.h>

ZSTD_CCtx* cctx = ZSTD_createCCtx();
if (!cctx) abort();

/* dictBuffer contains raw dictionary bytes, dictSize is its length */
size_t err = ZSTD_CCtx_loadDictionary(cctx, dictBuffer, dictSize);
if (ZSTD_isError(err)) {
    fprintf(stderr, "load dict error: %s\n", ZSTD_getErrorName(err));
    exit(1);
}

/* Compress with the loaded dictionary */
size_t cSize = ZSTD_compress_usingDict(cctx,
                                       dst, dstCap,
                                       src, srcSize,
                                       dictBuffer, dictSize);

To avoid the memory copy when your dictionary resides in static or read-only memory, use the by-reference variant:

size_t err = ZSTD_CCtx_loadDictionary_byReference(cctx, dictBuffer, dictSize);

Loading a Dictionary into a Decompression Context

The decompression counterpart is implemented in lib/decompress/zstd_decompress.c (lines 1697-1725). The API mirrors the compression side, ensuring symmetrical dictionary handling.

ZSTD_DCtx* dctx = ZSTD_createDCtx();
if (!dctx) abort();

size_t err = ZSTD_DCtx_loadDictionary(dctx, dictBuffer, dictSize);
if (ZSTD_isError(err)) {
    fprintf(stderr, "load dict error: %s\n", ZSTD_getErrorName(err));
    exit(1);
}

/* Decompress using the loaded dictionary */
size_t dSize = ZSTD_decompressDCtx(dctx,
                                   dst, dstCap,
                                   src, srcSize);

The by-reference alternative ZSTD_DCtx_loadDictionary_byReference() follows identical semantics to the compression variant.

Using Pre-Built Dictionary Objects for Performance

When the same dictionary will be used across multiple operations, pre-process it once to eliminate redundant initialization overhead. The lib/zdict.h header declares the dictionary creation APIs.

/* Create reusable dictionary object once */
ZSTD_CDict* cdict = ZSTD_createCDict(dictBuffer, dictSize, ZSTD_CLEVEL_DEFAULT);

/* Attach to any compression context */
ZSTD_CCtx* cctx = ZSTD_createCCtx();
ZSTD_CCtx_refCDict(cctx, cdict);  /* Sticky reference persists across compressions */

/* Compress using the referenced dictionary */
size_t cSize = ZSTD_compress_usingCDict(cctx,
                                        dst, dstCap,
                                        src, srcSize,
                                        cdict);

For decompression, the pattern is identical using ZSTD_createDDict() and ZSTD_DCtx_refDDict().

Dictionary Buffer Lifetime and Ownership Rules

Understanding memory ownership prevents use-after-free errors and memory leaks.

• ZSTD_CCtx_loadDictionary() / ZSTD_DCtx_loadDictionary(): The library creates an internal copy. You may free the original dictBuffer immediately after the call succeeds.
• ZSTD_CCtx_loadDictionary_byReference() / ZSTD_DCtx_loadDictionary_byReference(): You must preserve the original buffer unchanged until the context is reset or freed.
• ZSTD_createCDict() / ZSTD_createDDict(): These objects own their internal buffers. After creation, the original dictionary buffer can be freed, but the CDict/DDict objects themselves must be explicitly freed using ZSTD_freeCDict() and ZSTD_freeDDict().
• Context cleanup: ZSTD_freeCCtx() and ZSTD_freeDCtx() automatically release any internally copied dictionaries, but do not free referenced CDict/DDict objects.

Complete Example from the Repository

The facebook/zstd repository includes working implementations in examples/dictionary_compression.c and examples/dictionary_decompression.c. The typical workflow follows this pattern:

/* Compression workflow */
ZSTD_CDict* cdict = ZSTD_createCDict(dictBuffer, dictSize, ZSTD_CLEVEL_DEFAULT);

ZSTD_CCtx* cctx = ZSTD_createCCtx();
size_t cSize = ZSTD_compress_usingCDict(cctx,
                                        dst, dstCap,
                                        src, srcSize,
                                        cdict);
ZSTD_freeCCtx(cctx);
/* cdict can be reused for subsequent compressions */

/* Decompression workflow */
ZSTD_DDict* ddict = ZSTD_createDDict(dictBuffer, dictSize);
ZSTD_DCtx* dctx = ZSTD_createDCtx();
size_t dSize = ZSTD_decompress_usingDDict(dctx,
                                          dst, dstCap,
                                          src, srcSize,
                                          ddict);
ZSTD_freeDCtx(dctx);
ZSTD_freeDDict(ddict);

Summary

• Three loading strategies exist: by-copy for buffer independence, by-reference for zero-copy efficiency, and reusable objects for repeated operations.
• Copy methods (ZSTD_CCtx_loadDictionary) allow immediate freeing of source buffers but incur allocation overhead.
• Reference methods (ZSTD_CCtx_loadDictionary_byReference) require you to maintain the original buffer for the context's lifetime.
• CDict/DDict objects (ZSTD_createCDict) provide optimal performance for repeated dictionary usage by pre-processing tables once.
• Source implementations are located in lib/compress/zstd_compress.c and lib/decompress/zstd_decompress.c, with public APIs declared in lib/zstd.h and lib/zdict.h.

Frequently Asked Questions

What is the difference between ZSTD_CCtx_loadDictionary and ZSTD_CCtx_loadDictionary_byReference?

ZSTD_CCtx_loadDictionary copies the dictionary content into the context's internal memory, allowing you to free the original buffer immediately. ZSTD_CCtx_loadDictionary_byReference stores only a pointer to your buffer, which eliminates the copy but requires you to keep the buffer valid and unmodified until the context is destroyed or reset.

When should I use CDict/DDict objects instead of loading directly into the context?

Use ZSTD_CDict and ZSTD_DDict objects when you plan to reuse the same dictionary across multiple compression or decompression operations. These objects pre-process the dictionary once (building internal acceleration structures), whereas direct loading processes the dictionary on every attachment. This provides measurable performance improvements in batch processing scenarios.

Can I safely share a dictionary across multiple threads?

ZSTD_CDict and ZSTD_DDict objects are read-only after creation and can be safely shared across multiple threads simultaneously. However, ZSTD_CCtx and ZSTD_DCtx contexts are not thread-safe; each thread must maintain its own context while referencing shared dictionary objects.

How do I verify that a dictionary loaded successfully?

All dictionary loading functions return a size_t error code. Use ZSTD_isError() to check the return value, and ZSTD_getErrorName() to retrieve a human-readable description if an error occurred. A successful load returns zero or a non-error size value depending on the specific API variant.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}