# How to Build a Zstd Dictionary from a Buffer: In-Memory Training Guide

> Learn to build a Zstd dictionary from an in-memory buffer using fastCover algorithm and ZDICT_trainFromBuffer() for optimal compression. Get your compressed dictionary directly.

- Repository: [Meta/zstd](https://github.com/facebook/zstd)
- Tags: how-to-guide
- Published: 2026-09-09

---

**You can build a Zstandard dictionary from an in-memory buffer by concatenating training samples into a single contiguous block and invoking `ZDICT_trainFromBuffer()`, which selects optimal parameters using the fastCover algorithm and writes the compressed dictionary structure directly to your output buffer.**

Zstandard (zstd) dictionary training allows you to create custom compression dictionaries from sample data without writing intermediate files. According to the facebook/zstd source code, the dictionary builder API in [`lib/zdict.h`](https://github.com/facebook/zstd/blob/main/lib/zdict.h) provides a complete solution for training dictionaries entirely in memory using raw byte buffers and size arrays.

## Core Dictionary Training API

The primary entry point for in-memory dictionary construction is **`ZDICT_trainFromBuffer()`**, declared at [line 90 of [`lib/zdict.h`](https://github.com/facebook/zstd/blob/main/lib/zdict.h)](https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L90) and implemented at [line 1110 of [`lib/dictBuilder/zdict.c`](https://github.com/facebook/zstd/blob/main/lib/dictBuilder/zdict.c)](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.c#L1110).

This function wraps the internal optimization logic, defaulting to the **fastCover algorithm** with parameters `d=8` and `steps=4`. Internally, it forwards to `ZDICT_optimizeTrainFromBuffer_fastCover()` to perform the actual dictionary selection and entropy table generation.

The function signature expects:
- `void* dictBuffer`: Output buffer for the generated dictionary
- `size_t dictCapacity`: Maximum allowed size for the dictionary (must be at least `ZDICT_DICTSIZE_MIN`, which is 256 bytes)
- `const void* samplesBuffer`: Concatenated training samples
- `const size_t* samplesSizes`: Array holding the length of each individual sample
- `unsigned nbSamples`: Total number of samples in the array

## Preparing Training Data in Memory

Before calling the training API, you must organize your samples into a specific memory layout. The library expects a **flat concatenated buffer** rather than an array of pointers.

First, arrange all sample data sequentially in a single allocation (`samplesBuffer`). Second, create a parallel array (`samplesSizes`) that records the length of each sample in bytes. The sum of all values in `samplesSizes` must equal the total size of `samplesBuffer`.

This design eliminates the need for a "noisy guard band" in the fastCover path (the default), though the legacy training algorithm required separator bytes between samples.

## Training a Dictionary from a Buffer

The following example demonstrates the complete workflow: concatenating string samples, allocating the dictionary buffer, and training with error checking.

```c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zstd.h>
#include <zdict.h>

int main() {
    /* Training samples representing typical data patterns */
    const char *samples[] = { 
        "json:{\"name\":\"alice\"}", 
        "json:{\"name\":\"bob\"}" 
    };
    const unsigned nbSamples = 2;
    size_t samplesSizes[2];
    size_t totalSize = 0;
    
    /* Calculate individual sizes and total buffer requirement */
    for (int i = 0; i < nbSamples; i++) {
        samplesSizes[i] = strlen(samples[i]);
        totalSize += samplesSizes[i];
    }
    
    /* Allocate and fill contiguous samples buffer */
    void *samplesBuffer = malloc(totalSize);
    size_t pos = 0;
    for (int i = 0; i < nbSamples; i++) {
        memcpy((char*)samplesBuffer + pos, samples[i], samplesSizes[i]);
        pos += samplesSizes[i];
    }
    
    /* Allocate dictionary buffer (110 KB is the zstd default) */
    size_t dictCap = 110 * 1024;
    void *dictBuffer = malloc(dictCap);
    
    /* Train the dictionary */
    size_t dictSize = ZDICT_trainFromBuffer(dictBuffer, dictCap,
                                            samplesBuffer, samplesSizes, 
                                            nbSamples);
    
    if (ZDICT_isError(dictSize)) {
        fprintf(stderr, "Training failed: %s\n", 
                ZDICT_getErrorName(dictSize));
        return 1;
    }
    
    printf("Dictionary generated: %zu bytes\n", dictSize);
    /* dictBuffer now contains a valid zstd dictionary */
    
    free(samplesBuffer);
    free(dictBuffer);
    return 0;
}

```

**Key implementation details:**
- The returned `dictSize` may be smaller than `dictCapacity`; use the returned value to determine the actual dictionary footprint
- Errors are reported as specialized `size_t` values; always validate with `ZDICT_isError()` before using the result
- The generated buffer contains the dictionary header, entropy tables, and selected content bytes

## Finalizing Raw Content into a Dictionary

If you already possess **raw content bytes** (a pre-selected blob of representative data) rather than training samples, convert it into a full zstd dictionary using **`ZDICT_finalizeDictionary()`**. This function, declared at [line 62 of [`lib/zdict.h`](https://github.com/facebook/zstd/blob/main/lib/zdict.h)](https://github.com/facebook/zstd/blob/dev/lib/zdict.h#L62) and implemented at [line 662 of [`lib/dictBuilder/zdict.c`](https://github.com/facebook/zstd/blob/main/lib/dictBuilder/zdict.c)](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.c#L662), adds the required header and entropy tables by analyzing a small set of representative samples.

```c
/* Existing raw content you want to encode as a dictionary */
void *rawContent = /* ... */;
size_t rawSize = /* ... */;

/* Representative samples for entropy calculation */
void *samplesBuffer = /* ... */;
size_t samplesSizes[] = { /* ... */ };
unsigned nbSamples = /* ... */;

/* Configure parameters, especially the target compression level */
ZDICT_params_t params = {0};
params.compressionLevel = 3;
params.notificationLevel = 0;  /* Suppress diagnostic output */

void *dictBuffer = malloc(dictCap);
size_t finalSize = ZDICT_finalizeDictionary(dictBuffer, dictCap,
                                            rawContent, rawSize,
                                            samplesBuffer, samplesSizes,
                                            nbSamples, &params);

if (ZDICT_isError(finalSize)) {
    fprintf(stderr, "Finalize failed: %s\n", 
            ZDICT_getErrorName(finalSize));
    exit(1);
}

```

## Using the Generated Dictionary

Once trained, load the dictionary for compression using `ZSTD_createCDict()` or `ZSTD_decompress_createDDict()`. The facebook/zstd repository provides complete examples in [[`examples/dictionary_compression.c`](https://github.com/facebook/zstd/blob/main/examples/dictionary_compression.c)](https://github.com/facebook/zstd/blob/dev/examples/dictionary_compression.c), which demonstrates creating a `ZSTD_CDict` and compressing data with `ZSTD_compress_usingCDict()`.

## Summary

- **`ZDICT_trainFromBuffer()`** in [`lib/dictBuilder/zdict.c`](https://github.com/facebook/zstd/blob/main/lib/dictBuilder/zdict.c) is the high-level API for building dictionaries from concatenated memory buffers
- Training requires a **samples buffer** (concatenated data) and a **sizes array** (sample boundaries), not a pointer array
- The default **fastCover algorithm** optimizes dictionary selection without requiring separator bytes between samples
- **Minimum capacity** is 256 bytes (`ZDICT_DICTSIZE_MIN`), though 110 KB is recommended for general use
- Use **`ZDICT_finalizeDictionary()`** to convert raw content into a complete dictionary with proper headers
- Always validate return values with **`ZDICT_isError()`** and diagnose failures with **`ZDICT_getErrorName()`**

## Frequently Asked Questions

### What is the minimum size for a dictionary buffer?

The absolute minimum capacity is **256 bytes**, defined as `ZDICT_DICTSIZE_MIN` in [`lib/zdict.h`](https://github.com/facebook/zstd/blob/main/lib/zdict.h). However, the default training target is 110 KB, and dictionaries smaller than a few kilobytes rarely provide meaningful compression improvements for typical data sets.

### How do I handle errors from ZDICT_trainFromBuffer?

The function returns a `size_t` where values equal to or greater than `ZDICT_DICTSIZE_MIN` indicate success (the actual dictionary size). Smaller values indicate errors. Use **`ZDICT_isError(returnValue)`** to check for failure, and **`ZDICT_getErrorName(returnValue)`** to retrieve a human-readable error string describing issues such as insufficient samples or inadequate dictionary capacity.

### Can I train a dictionary without concatenating samples into one buffer?

No. The current API in [`lib/zdict.h`](https://github.com/facebook/zstd/blob/main/lib/zdict.h) requires a **single contiguous buffer** for all samples. You must concatenate your data manually and provide the `samplesSizes` array to denote boundaries. There is no alternative interface that accepts an array of separate pointers in the public API.

### What is the difference between ZDICT_trainFromBuffer and ZDICT_finalizeDictionary?

**`ZDICT_trainFromBuffer()`** performs full dictionary training: it analyzes your samples, selects the best content segments, and builds entropy tables. **`ZDICT_finalizeDictionary()`** assumes you have already selected the raw content bytes and only need to attach the zstd header and entropy tables using representative samples for statistical analysis. Use the former when building a dictionary from scratch; use the latter when you have pre-computed content to wrap with zstd metadata.