# How the Zstandard Streaming Compression API Works: A Complete Technical Guide

> Explore the zstd streaming compression API. Learn how ZSTD_CCtx, ZSTD_inBuffer, ZSTD_outBuffer, and ZSTD_compressStream2 efficiently handle large data inputs without full memory loading.

- Repository: [Meta/zstd](https://github.com/facebook/zstd)
- Tags: deep-dive
- Published: 2026-09-09

---

**The zstd streaming compression API processes data incrementally using a compression context (`ZSTD_CCtx`), input/output buffers (`ZSTD_inBuffer`/`ZSTD_outBuffer`), and the `ZSTD_compressStream2()` function to handle arbitrarily large inputs without loading them entirely into memory.**

This guide examines the **streaming compression API** implemented in the `facebook/zstd` repository. The API enables constant-memory compression of multi-gigabyte files by maintaining encoder state in a persistent context and processing data through sequential buffer operations.

## Core Architecture and Components

### The Compression Context (`ZSTD_CCtx`)

All streaming operations revolve around the **compression context**, declared in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) as `ZSTD_CStream` (an alias for `ZSTD_CCtx`). This opaque structure encapsulates the entire encoder state, including sliding window buffers, match finders, and compression parameters.

```c
ZSTD_CStream* cstream = ZSTD_createCStream();  // Line 779, lib/zstd.h

```

Internally, `ZSTD_createCStream()` allocates and initializes a `ZSTD_CCtx`. Modern code typically uses `ZSTD_createCCtx()` directly, as the two are interchangeable in recent versions. The context must be freed using `ZSTD_freeCStream()` or `ZSTD_freeCCtx()` to release internal buffers and heap allocations.

### Input and Output Buffer Structures

The API uses two simple structs to track buffer progress:

- **`ZSTD_inBuffer`**: Describes the input memory region (`src`, `size`) and current read position (`pos`)
- **`ZSTD_outBuffer`**: Describes the output destination (`dst`, `size`) and write position (`pos`)

These structures appear in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) around line 740. The `pos` fields act as cursors that the encoder updates incrementally, allowing the caller to resume processing without managing pointer offsets manually.

### Recommended Buffer Sizes

Optimal buffer sizes vary by compression level and available memory. The API provides helper functions to retrieve recommended defaults:

```c
size_t inSize  = ZSTD_CStreamInSize();   /* Typical: 128 KB */
size_t outSize = ZSTD_CStreamOutSize();  /* Typical: 128 KB */

```

These functions return conservative values that minimize internal copying while ensuring efficient block formation, defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) near line 292.

## The Streaming Compression Workflow

### Context Creation and Configuration

Before compression begins, create a context and optionally configure parameters using `ZSTD_CCtx_setParameter()` (declared in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) approximately line 998):

```c
ZSTD_CCtx* cctx = ZSTD_createCCtx();
ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 3);
ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 1);

```

Common parameters include `ZSTD_c_compressionLevel` (1-22), `ZSTD_c_checksumFlag` for data integrity verification, and `ZSTD_c_nbWorkers` to enable parallel compression.

### The Main Compression Loop

The core streaming function is `ZSTD_compressStream2()`, declared at line 823 in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h). This single function handles both intermediate data feeding and frame finalization through the `ZSTD_EndDirective` enum:

```c
typedef enum {
    ZSTD_e_continue=0,   /* Collect more data, don't close frame */
    ZSTD_e_end=1,        /* Flush all buffers and close frame */
    [..]
} ZSTD_EndDirective;

```

A typical compression cycle follows this pattern:

1. Fill an input buffer from the source stream
2. Initialize `ZSTD_inBuffer` and `ZSTD_outBuffer` with current positions
3. Call `ZSTD_compressStream2()` with `mode` set to `ZSTD_e_continue` (or `ZSTD_e_end` for the final chunk)
4. Write produced output (up to `output.pos`) to the destination
5. Repeat until the input buffer's `pos` equals its `size`

```c
ZSTD_inBuffer  input  = { inBuf, bytesRead, 0 };
ZSTD_outBuffer output = { outBuf, outSize, 0 };

size_t remaining = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_continue);

/* Write compressed bytes */
fwrite(outBuf, 1, output.pos, fout);

```

The function returns the number of bytes still pending inside internal buffers. When compressing the final chunk with `ZSTD_e_end`, you must continue calling the function (potentially with empty input) until it returns **0**, indicating the frame is fully emitted and all buffers are flushed.

### Frame Boundaries and Flushing

Unlike simple block-based compressors, zstd produces **frames** containing headers, compressed blocks, and checksums. The `ZSTD_EndDirective` controls boundary behavior:

- **`ZSTD_e_continue`**: Accumulates data internally, potentially delaying output to maximize compression ratio
- **`ZSTD_e_end`**: Forces emission of all buffered data, writes frame trailer, and resets internal dictionaries for potential frame concatenation

When `ZSTD_e_end` is specified, the return value indicates remaining internal bytes. Loop until zero:

```c
int finished = 0;
while (!finished) {
    ZSTD_outBuffer out = { outBuf, outSize, 0 };
    size_t remaining = ZSTD_compressStream2(cctx, &out, &input, ZSTD_e_end);
    fwrite(outBuf, 1, out.pos, fout);
    finished = (remaining == 0);
}

```

## Multithreaded Streaming Compression

When `ZSTD_c_nbWorkers` is set greater than zero, the encoder distributes block processing across internal worker threads. The API contract remains identical—`ZSTD_compressStream2()` accepts the same arguments and returns the same error codes. The only observable difference is that calls may return before all internal processing completes, requiring additional iterations to exhaust pending output buffers.

```c
ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, 4);  /* Enable 4 threads */

```

The streaming state machine handles synchronization internally, so caller code requires no thread management logic.

## Error Handling and Context Reset

All API functions return `size_t`, where values are either byte counts or special error codes. Always validate returns using `ZSTD_isError()` and retrieve descriptions via `ZSTD_getErrorName()`:

```c
size_t const ret = ZSTD_compressStream2(cctx, &out, &in, ZSTD_e_continue);
if (ZSTD_isError(ret)) {
    fprintf(stderr, "Compression error: %s\n", ZSTD_getErrorName(ret));
    /* Context must be reset before reuse */
    ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only);
}

```

After any error, the context enters an indeterminate state and must be reset using `ZSTD_CCtx_reset()` before subsequent operations.

## Complete Implementation Example

The following implementation, derived from [`examples/streaming_compression.c`](https://github.com/facebook/zstd/blob/main/examples/streaming_compression.c) in the repository, demonstrates the full workflow including parameter configuration and multithreading support:

```c
#include <stdio.h>
#include <stdlib.h>
#include <zstd.h>

static void compressFile(const char* src, const char* dst,
                         int level, int nbThreads)
{
    FILE* fin  = fopen(src, "rb");
    FILE* fout = fopen(dst, "wb");

    size_t inSize  = ZSTD_CStreamInSize();
    size_t outSize = ZSTD_CStreamOutSize();
    void*  inBuf   = malloc(inSize);
    void*  outBuf  = malloc(outSize);

    ZSTD_CCtx* cctx = ZSTD_createCCtx();
    
    /* Configure parameters */
    ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, level);
    ZSTD_CCtx_setParameter(cctx, ZSTD_c_checksumFlag, 1);
    if (nbThreads > 1)
        ZSTD_CCtx_setParameter(cctx, ZSTD_c_nbWorkers, nbThreads);

    /* Main compression loop */
    for (;;) {
        size_t const read = fread(inBuf, 1, inSize, fin);
        int const lastChunk = (read < inSize);
        ZSTD_EndDirective const mode = lastChunk ? ZSTD_e_end : ZSTD_e_continue;
        
        ZSTD_inBuffer input = { inBuf, read, 0 };
        int finished = 0;
        
        while (!finished) {
            ZSTD_outBuffer output = { outBuf, outSize, 0 };
            size_t const remaining = ZSTD_compressStream2(cctx, &output, &input, mode);
            
            if (ZSTD_isError(remaining)) {
                fprintf(stderr, "Error: %s\n", ZSTD_getErrorName(remaining));
                exit(1);
            }
            
            fwrite(outBuf, 1, output.pos, fout);
            finished = lastChunk ? (remaining == 0) : (input.pos == input.size);
        }
        if (lastChunk) break;
    }

    ZSTD_freeCCtx(cctx);
    fclose(fout);
    fclose(fin);
    free(inBuf);
    free(outBuf);
}

```

## Summary

- **Streaming compression** in zstd utilizes a persistent **compression context** (`ZSTD_CCtx`) to maintain state across incremental writes, defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h).
- The **`ZSTD_compressStream2()`** function (line 823) drives the main loop, accepting `ZSTD_inBuffer`/`ZSTD_outBuffer` structures and an **end directive** (`ZSTD_e_continue` or `ZSTD_e_end`) to control frame boundaries.
- **Memory efficiency** comes from processing data in chunks sized by `ZSTD_CStreamInSize()` and `ZSTD_CStreamOutSize()`, avoiding whole-file loading.
- **Parallel compression** requires only setting `ZSTD_c_nbWorkers` via `ZSTD_CCtx_setParameter()`; the streaming API contract remains unchanged.
- **Error handling** relies on `ZSTD_isError()` checks, with mandatory `ZSTD_CCtx_reset()` calls after failures before context reuse.

## Frequently Asked Questions

### What is the difference between `ZSTD_CStream` and `ZSTD_CCtx`?

`ZSTD_CStream` is a historical typedef for `ZSTD_CCtx` maintained for backward compatibility, as seen in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) line 776. Modern code should use `ZSTD_CCtx*` and the associated `ZSTD_createCCtx()`/`ZSTD_freeCCtx()` functions, though all `ZSTD_CStream*` functions remain functional aliases.

### How do I know when a compressed frame is completely written?

When you pass `ZSTD_e_end` as the mode to `ZSTD_compressStream2()`, the function returns the number of bytes remaining in internal buffers. Continue calling the function (providing output buffers) until it returns exactly **0**, which signifies that the frame header, all compressed blocks, and the checksum have been fully emitted.

### Can I reuse a compression context for multiple files?

Yes. After completing a frame with `ZSTD_e_end` (which resets the stream state), or explicitly calling `ZSTD_CCtx_reset(cctx, ZSTD_reset_session_only)`, you can begin compressing a new input stream using the same context. This preserves allocated memory buffers and dictionary states, improving performance for batch operations.

### What happens if my output buffer is too small?

`ZSTD_compressStream2()` fills the provided `ZSTD_outBuffer` up to its `size` limit, updates `pos` to indicate bytes written, and returns a hint indicating whether more output is waiting. If the buffer is too small to flush all pending data, simply provide a new or cleared buffer and call the function again with the same `mode` (typically `ZSTD_e_continue`). The encoder retains internal state, so no data is lost.