How to Compress Unbounded Data with the Zstd Streaming API

Zstandard (zstd) provides a stateful streaming compression interface that processes data incrementally using ZSTD_CStream contexts, ZSTD_compressStream2(), and buffer management functions to handle inputs of any size without requiring the total length upfront.

The facebook/zstd library offers a low-level streaming API designed specifically for unbounded data compression scenarios such as log processing, network streams, and large file archives. Unlike the simple one-shot compression functions, the streaming API maintains internal state across multiple calls, allowing you to feed data chunks as they arrive from disk or network sockets. This approach eliminates memory constraints and enables compression of terabyte-scale inputs on modest hardware.

Architecture of the Zstd Streaming Interface

The streaming API centers on the ZSTD_CStream type, which is functionally an alias for ZSTD_CCtx according to the source in lib/zstd.h. This context object encapsulates all compression parameters, history buffers, and state machines needed for incremental processing. Because the API is stateful, once you initialize a context with a compression level or dictionary, those settings persist until explicitly modified or reset.

Key structures include:

• ZSTD_inBuffer – tracks the input pointer, total size, and consumed bytes (pos)
• ZSTD_outBuffer – manages the destination buffer and written bytes
• ZSTD_EndDirective – controls stream flushing behavior (ZSTD_e_continue vs ZSTD_e_end)

Step-by-Step Implementation Guide

Creating and Initializing the Compression Context

Begin by allocating a stream context using ZSTD_createCStream(), declared at line 779 of lib/zstd.h. This returns a pointer that you must later release with ZSTD_freeCStream() (line 780).

Before processing any data, initialize the context with your desired compression level using ZSTD_initCStream() (line 862). This resets the internal state and prepares the encoder.

ZSTD_CStream *cstream = ZSTD_createCStream();
if (!cstream) { /* handle error */ }

int level = 3;  /* compression level 1-22 */
size_t const initResult = ZSTD_initCStream(cstream, level);
if (ZSTD_isError(initResult)) {
    fprintf(stderr, "Init error: %s\n", ZSTD_getErrorName(initResult));
}

Optimizing Buffer Sizes

While the streaming API accepts any buffer size, using the library's recommended sizes minimizes internal memory shuffling. Query these values with ZSTD_CStreamInSize() (line 842) and ZSTD_CStreamOutSize() (line 843), both defined in lib/zstd.h. These typically return approximately 128 KB each.

size_t const inSize  = ZSTD_CStreamInSize();   /* optimal input chunk size */
size_t const outSize = ZSTD_CStreamOutSize();  /* optimal output buffer size */

void *inBuff  = malloc(inSize);
void *outBuff = malloc(outSize);

Processing Data Incrementally

Data flows through the compressor via the ZSTD_compressStream2() function (lines 823-826). Use the ZSTD_e_continue directive to indicate that more input is coming. This function consumes data from the ZSTD_inBuffer and writes compressed output to the ZSTD_outBuffer, returning the minimum number of bytes still waiting to be flushed.

ZSTD_inBuffer input = { inBuff, bytes_read, 0 };
ZSTD_outBuffer output = { outBuff, outSize, 0 };

/* Compress while data remains */
while (input.pos < input.size) {
    size_t const ret = ZSTD_compressStream2(
        cstream, &output, &input, ZSTD_e_continue);
    
    if (ZSTD_isError(ret)) { /* handle error */ }
    
    /* Flush output buffer if full */
    if (output.pos == output.size) {
        fwrite(outBuff, 1, output.pos, fout);
        output.pos = 0;
    }
}

When the return value is non-zero, internal buffers contain data that must be cleared by providing more output space. Continue calling the function until input.pos == input.size, indicating all input has been consumed.

Finalizing the Compressed Frame

After processing the last input chunk, you must close the zstd frame to write the epilogue and checksum. Invoke ZSTD_compressStream2() with the ZSTD_e_end directive (lines 832-836), repeating the call until it returns 0, which signals the frame is completely flushed.

size_t remaining;
do {
    remaining = ZSTD_compressStream2(
        cstream, &output, &input, ZSTD_e_end);
    
    if (ZSTD_isError(remaining)) { /* handle error */ }
    
    fwrite(outBuff, 1, output.pos, fout);
    output.pos = 0;
} while (remaining != 0);

Complete Working Example

The following C program demonstrates compressing an unbounded file stream using the exact API patterns found in tests/zstreamtest.c and zlibWrapper/zstd_zlibwrapper.c.

/* Compile with: gcc -o stream_compress stream_compress.c -lzstd */

#include <stdio.h>
#include <stdlib.h>
#include "zstd.h"

int main(void) {
    /* 1. Create and initialize streaming context */
    ZSTD_CStream *cstream = ZSTD_createCStream();
    if (!cstream) { perror("ZSTD_createCStream"); return 1; }
    
    if (ZSTD_isError(ZSTD_initCStream(cstream, 3))) {
        fprintf(stderr, "Init failed\n");
        return 1;
    }
    
    /* 2. Allocate recommended buffer sizes */
    size_t const inSize = ZSTD_CStreamInSize();
    size_t const outSize = ZSTD_CStreamOutSize();
    void *const inBuff = malloc(inSize);
    void *const outBuff = malloc(outSize);
    
    FILE *fin = fopen("large_input.bin", "rb");
    FILE *fout = fopen("output.zst", "wb");
    
    ZSTD_inBuffer input = { inBuff, 0, 0 };
    ZSTD_outBuffer output = { outBuff, outSize, 0 };
    
    /* 3. Main compression loop */
    while (1) {
        /* Refill input buffer when empty */
        if (input.pos == input.size) {
            input.size = fread(inBuff, 1, inSize, fin);
            input.pos = 0;
            if (input.size == 0) break;  /* EOF */
        }
        
        size_t const ret = ZSTD_compressStream2(
            cstream, &output, &input, ZSTD_e_continue);
        
        if (ZSTD_isError(ret)) {
            fprintf(stderr, "Compress error: %s\n", ZSTD_getErrorName(ret));
            return 1;
        }
        
        /* Flush output when full */
        if (output.pos == output.size) {
            fwrite(outBuff, 1, output.pos, fout);
            output.pos = 0;
        }
    }
    
    /* 4. Finalize frame */
    while (1) {
        size_t const ret = ZSTD_compressStream2(
            cstream, &output, &input, ZSTD_e_end);
        
        if (ZSTD_isError(ret)) {
            fprintf(stderr, "End error: %s\n", ZSTD_getErrorName(ret));
            return 1;
        }
        
        fwrite(outBuff, 1, output.pos, fout);
        output.pos = 0;
        if (ret == 0) break;  /* Frame complete */
    }
    
    /* Cleanup */
    fclose(fin);
    fclose(fout);
    free(inBuff);
    free(outBuff);
    ZSTD_freeCStream(cstream);
    
    return 0;
}

Context Reuse and Reset Strategies

A ZSTD_CStream context can be reused for multiple independent compression jobs without reallocation, improving performance in server applications. To reset the context while preserving compression parameters (level, dictionary, etc.), call ZSTD_CCtx_reset() with ZSTD_reset_session_only. To reset both the session and all parameters to defaults, use ZSTD_reset_session_and_parameters.

/* Reset for new file, keep level=3 setting */
ZSTD_CCtx_reset(cstream, ZSTD_reset_session_only);
ZSTD_initCStream(cstream, 3);  /* Re-init with same or different level */

Summary

• Create a streaming context with ZSTD_createCStream() and initialize it using ZSTD_initCStream() before compression begins.
• Optimize performance by allocating buffers sized according to ZSTD_CStreamInSize() and ZSTD_CStreamOutSize().
• Stream data incrementally using ZSTD_compressStream2() with ZSTD_e_continue, checking return values to determine when to flush internal buffers.
• Finalize by calling ZSTD_compressStream2() with ZSTD_e_end until it returns 0, ensuring the zstd frame epilogue is written.
• Reuse contexts across jobs using reset functions to minimize allocation overhead in high-throughput applications.

Frequently Asked Questions

What is the difference between ZSTD_CStream and ZSTD_CCtx?

ZSTD_CStream is a typedef alias for ZSTD_CCtx defined in lib/zstd.h for backward compatibility. Both refer to the same structure handling compression state. The streaming functions operate on this context object to maintain history and parameters across incremental calls.

How do I handle buffer overflow during streaming compression?

When ZSTD_compressStream2() fills the ZSTD_outBuffer (indicated by output.pos == output.size), write the buffer contents to your destination (file, socket, or network), reset output.pos to 0, and call the function again with the same input parameters. The function preserves unconsumed input in the ZSTD_inBuffer structure.

Can I reuse a ZSTD_CStream for multiple files?

Yes. After completing a frame with ZSTD_e_end, reset the context using ZSTD_CCtx_reset(cstream, ZSTD_reset_session_only) and reinitialize with ZSTD_initCStream(). This avoids the overhead of ZSTD_createCStream() and ZSTD_freeCStream() in batch processing workflows.

What compression level should I use for streaming?

The default level 3 offers an optimal balance of speed and compression ratio for most streaming applications. Levels 1-22 are supported, with higher levels providing better compression at the cost of memory usage and CPU time. According to the source code in lib/zstd.h, you specify this in ZSTD_initCStream(cstream, level) during initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}