How to Build a Zstd Dictionary from a Buffer: In-Memory Training Guide

You can build a Zstandard dictionary from an in-memory buffer by concatenating training samples into a single contiguous block and invoking ZDICT_trainFromBuffer(), which selects optimal parameters using the fastCover algorithm and writes the compressed dictionary structure directly to your output buffer.

Zstandard (zstd) dictionary training allows you to create custom compression dictionaries from sample data without writing intermediate files. According to the facebook/zstd source code, the dictionary builder API in lib/zdict.h provides a complete solution for training dictionaries entirely in memory using raw byte buffers and size arrays.

Core Dictionary Training API

The primary entry point for in-memory dictionary construction is ZDICT_trainFromBuffer(), declared at [line 90 of lib/zdict.h](https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L90) and implemented at [line 1110 of lib/dictBuilder/zdict.c](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.c#L1110).

This function wraps the internal optimization logic, defaulting to the fastCover algorithm with parameters d=8 and steps=4. Internally, it forwards to ZDICT_optimizeTrainFromBuffer_fastCover() to perform the actual dictionary selection and entropy table generation.

The function signature expects:

• void* dictBuffer: Output buffer for the generated dictionary
• size_t dictCapacity: Maximum allowed size for the dictionary (must be at least ZDICT_DICTSIZE_MIN, which is 256 bytes)
• const void* samplesBuffer: Concatenated training samples
• const size_t* samplesSizes: Array holding the length of each individual sample
• unsigned nbSamples: Total number of samples in the array

Preparing Training Data in Memory

Before calling the training API, you must organize your samples into a specific memory layout. The library expects a flat concatenated buffer rather than an array of pointers.

First, arrange all sample data sequentially in a single allocation (samplesBuffer). Second, create a parallel array (samplesSizes) that records the length of each sample in bytes. The sum of all values in samplesSizes must equal the total size of samplesBuffer.

This design eliminates the need for a "noisy guard band" in the fastCover path (the default), though the legacy training algorithm required separator bytes between samples.

Training a Dictionary from a Buffer

The following example demonstrates the complete workflow: concatenating string samples, allocating the dictionary buffer, and training with error checking.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zstd.h>
#include <zdict.h>

int main() {
    /* Training samples representing typical data patterns */
    const char *samples[] = { 
        "json:{\"name\":\"alice\"}", 
        "json:{\"name\":\"bob\"}" 
    };
    const unsigned nbSamples = 2;
    size_t samplesSizes[2];
    size_t totalSize = 0;
    
    /* Calculate individual sizes and total buffer requirement */
    for (int i = 0; i < nbSamples; i++) {
        samplesSizes[i] = strlen(samples[i]);
        totalSize += samplesSizes[i];
    }
    
    /* Allocate and fill contiguous samples buffer */
    void *samplesBuffer = malloc(totalSize);
    size_t pos = 0;
    for (int i = 0; i < nbSamples; i++) {
        memcpy((char*)samplesBuffer + pos, samples[i], samplesSizes[i]);
        pos += samplesSizes[i];
    }
    
    /* Allocate dictionary buffer (110 KB is the zstd default) */
    size_t dictCap = 110 * 1024;
    void *dictBuffer = malloc(dictCap);
    
    /* Train the dictionary */
    size_t dictSize = ZDICT_trainFromBuffer(dictBuffer, dictCap,
                                            samplesBuffer, samplesSizes, 
                                            nbSamples);
    
    if (ZDICT_isError(dictSize)) {
        fprintf(stderr, "Training failed: %s\n", 
                ZDICT_getErrorName(dictSize));
        return 1;
    }
    
    printf("Dictionary generated: %zu bytes\n", dictSize);
    /* dictBuffer now contains a valid zstd dictionary */
    
    free(samplesBuffer);
    free(dictBuffer);
    return 0;
}

Key implementation details:

• The returned dictSize may be smaller than dictCapacity; use the returned value to determine the actual dictionary footprint
• Errors are reported as specialized size_t values; always validate with ZDICT_isError() before using the result
• The generated buffer contains the dictionary header, entropy tables, and selected content bytes

Finalizing Raw Content into a Dictionary

If you already possess raw content bytes (a pre-selected blob of representative data) rather than training samples, convert it into a full zstd dictionary using ZDICT_finalizeDictionary(). This function, declared at [line 62 of lib/zdict.h](https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L62) and implemented at [line 662 of lib/dictBuilder/zdict.c](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.c#L662), adds the required header and entropy tables by analyzing a small set of representative samples.

/* Existing raw content you want to encode as a dictionary */
void *rawContent = /* ... */;
size_t rawSize = /* ... */;

/* Representative samples for entropy calculation */
void *samplesBuffer = /* ... */;
size_t samplesSizes[] = { /* ... */ };
unsigned nbSamples = /* ... */;

/* Configure parameters, especially the target compression level */
ZDICT_params_t params = {0};
params.compressionLevel = 3;
params.notificationLevel = 0;  /* Suppress diagnostic output */

void *dictBuffer = malloc(dictCap);
size_t finalSize = ZDICT_finalizeDictionary(dictBuffer, dictCap,
                                            rawContent, rawSize,
                                            samplesBuffer, samplesSizes,
                                            nbSamples, &params);

if (ZDICT_isError(finalSize)) {
    fprintf(stderr, "Finalize failed: %s\n", 
            ZDICT_getErrorName(finalSize));
    exit(1);
}

Using the Generated Dictionary

Once trained, load the dictionary for compression using ZSTD_createCDict() or ZSTD_decompress_createDDict(). The facebook/zstd repository provides complete examples in [examples/dictionary_compression.c](https://github.com/facebook/zstd/blob/dev/examples/dictionary_compression.c), which demonstrates creating a ZSTD_CDict and compressing data with ZSTD_compress_usingCDict().

Summary

• ZDICT_trainFromBuffer() in lib/dictBuilder/zdict.c is the high-level API for building dictionaries from concatenated memory buffers
• Training requires a samples buffer (concatenated data) and a sizes array (sample boundaries), not a pointer array
• The default fastCover algorithm optimizes dictionary selection without requiring separator bytes between samples
• Minimum capacity is 256 bytes (ZDICT_DICTSIZE_MIN), though 110 KB is recommended for general use
• Use ZDICT_finalizeDictionary() to convert raw content into a complete dictionary with proper headers
• Always validate return values with ZDICT_isError() and diagnose failures with ZDICT_getErrorName()

Frequently Asked Questions

What is the minimum size for a dictionary buffer?

The absolute minimum capacity is 256 bytes, defined as ZDICT_DICTSIZE_MIN in lib/zdict.h. However, the default training target is 110 KB, and dictionaries smaller than a few kilobytes rarely provide meaningful compression improvements for typical data sets.

How do I handle errors from ZDICT_trainFromBuffer?

The function returns a size_t where values equal to or greater than ZDICT_DICTSIZE_MIN indicate success (the actual dictionary size). Smaller values indicate errors. Use ZDICT_isError(returnValue) to check for failure, and ZDICT_getErrorName(returnValue) to retrieve a human-readable error string describing issues such as insufficient samples or inadequate dictionary capacity.

Can I train a dictionary without concatenating samples into one buffer?

No. The current API in lib/zdict.h requires a single contiguous buffer for all samples. You must concatenate your data manually and provide the samplesSizes array to denote boundaries. There is no alternative interface that accepts an array of separate pointers in the public API.

What is the difference between ZDICT_trainFromBuffer and ZDICT_finalizeDictionary?

ZDICT_trainFromBuffer() performs full dictionary training: it analyzes your samples, selects the best content segments, and builds entropy tables. ZDICT_finalizeDictionary() assumes you have already selected the raw content bytes and only need to attach the zstd header and entropy tables using representative samples for statistical analysis. Use the former when building a dictionary from scratch; use the latter when you have pre-computed content to wrap with zstd metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}