# How to Compress Unbounded Data with the Zstd Streaming API

> Compress unbounded data efficiently with the Zstandard streaming API. Learn how to use ZSTD_CStream and ZSTD_compressStream2 for incremental processing of any data size without upfront length.

- Repository: [Meta/zstd](https://github.com/facebook/zstd)
- Tags: how-to-guide
- Published: 2026-09-09

---

**Zstandard (zstd) provides a stateful streaming compression interface that processes data incrementally using `ZSTD_CStream` contexts, `ZSTD_compressStream2()`, and buffer management functions to handle inputs of any size without requiring the total length upfront.**

The facebook/zstd library offers a low-level streaming API designed specifically for **unbounded data compression** scenarios such as log processing, network streams, and large file archives. Unlike the simple one-shot compression functions, the streaming API maintains internal state across multiple calls, allowing you to feed data chunks as they arrive from disk or network sockets. This approach eliminates memory constraints and enables compression of terabyte-scale inputs on modest hardware.

## Architecture of the Zstd Streaming Interface

The streaming API centers on the `ZSTD_CStream` type, which is functionally an alias for `ZSTD_CCtx` according to the source in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h). This context object encapsulates all compression parameters, history buffers, and state machines needed for incremental processing. Because the API is **stateful**, once you initialize a context with a compression level or dictionary, those settings persist until explicitly modified or reset.

Key structures include:
- `ZSTD_inBuffer` – tracks the input pointer, total size, and consumed bytes (`pos`)
- `ZSTD_outBuffer` – manages the destination buffer and written bytes
- `ZSTD_EndDirective` – controls stream flushing behavior (`ZSTD_e_continue` vs `ZSTD_e_end`)

## Step-by-Step Implementation Guide

### Creating and Initializing the Compression Context

Begin by allocating a stream context using `ZSTD_createCStream()`, declared at line 779 of [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h). This returns a pointer that you must later release with `ZSTD_freeCStream()` (line 780).

Before processing any data, initialize the context with your desired compression level using `ZSTD_initCStream()` (line 862). This resets the internal state and prepares the encoder.

```c
ZSTD_CStream *cstream = ZSTD_createCStream();
if (!cstream) { /* handle error */ }

int level = 3;  /* compression level 1-22 */
size_t const initResult = ZSTD_initCStream(cstream, level);
if (ZSTD_isError(initResult)) {
    fprintf(stderr, "Init error: %s\n", ZSTD_getErrorName(initResult));
}

```

### Optimizing Buffer Sizes

While the streaming API accepts any buffer size, using the library's recommended sizes minimizes internal memory shuffling. Query these values with `ZSTD_CStreamInSize()` (line 842) and `ZSTD_CStreamOutSize()` (line 843), both defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h). These typically return approximately 128 KB each.

```c
size_t const inSize  = ZSTD_CStreamInSize();   /* optimal input chunk size */
size_t const outSize = ZSTD_CStreamOutSize();  /* optimal output buffer size */

void *inBuff  = malloc(inSize);
void *outBuff = malloc(outSize);

```

### Processing Data Incrementally

Data flows through the compressor via the `ZSTD_compressStream2()` function (lines 823-826). Use the `ZSTD_e_continue` directive to indicate that more input is coming. This function consumes data from the `ZSTD_inBuffer` and writes compressed output to the `ZSTD_outBuffer`, returning the minimum number of bytes still waiting to be flushed.

```c
ZSTD_inBuffer input = { inBuff, bytes_read, 0 };
ZSTD_outBuffer output = { outBuff, outSize, 0 };

/* Compress while data remains */
while (input.pos < input.size) {
    size_t const ret = ZSTD_compressStream2(
        cstream, &output, &input, ZSTD_e_continue);
    
    if (ZSTD_isError(ret)) { /* handle error */ }
    
    /* Flush output buffer if full */
    if (output.pos == output.size) {
        fwrite(outBuff, 1, output.pos, fout);
        output.pos = 0;
    }
}

```

When the return value is non-zero, internal buffers contain data that must be cleared by providing more output space. Continue calling the function until `input.pos == input.size`, indicating all input has been consumed.

### Finalizing the Compressed Frame

After processing the last input chunk, you must close the zstd frame to write the epilogue and checksum. Invoke `ZSTD_compressStream2()` with the `ZSTD_e_end` directive (lines 832-836), repeating the call until it returns 0, which signals the frame is completely flushed.

```c
size_t remaining;
do {
    remaining = ZSTD_compressStream2(
        cstream, &output, &input, ZSTD_e_end);
    
    if (ZSTD_isError(remaining)) { /* handle error */ }
    
    fwrite(outBuff, 1, output.pos, fout);
    output.pos = 0;
} while (remaining != 0);

```

## Complete Working Example

The following C program demonstrates compressing an unbounded file stream using the exact API patterns found in [`tests/zstreamtest.c`](https://github.com/facebook/zstd/blob/main/tests/zstreamtest.c) and [`zlibWrapper/zstd_zlibwrapper.c`](https://github.com/facebook/zstd/blob/main/zlibWrapper/zstd_zlibwrapper.c).

```c
/* Compile with: gcc -o stream_compress stream_compress.c -lzstd */

#include <stdio.h>
#include <stdlib.h>
#include "zstd.h"

int main(void) {
    /* 1. Create and initialize streaming context */
    ZSTD_CStream *cstream = ZSTD_createCStream();
    if (!cstream) { perror("ZSTD_createCStream"); return 1; }
    
    if (ZSTD_isError(ZSTD_initCStream(cstream, 3))) {
        fprintf(stderr, "Init failed\n");
        return 1;
    }
    
    /* 2. Allocate recommended buffer sizes */
    size_t const inSize = ZSTD_CStreamInSize();
    size_t const outSize = ZSTD_CStreamOutSize();
    void *const inBuff = malloc(inSize);
    void *const outBuff = malloc(outSize);
    
    FILE *fin = fopen("large_input.bin", "rb");
    FILE *fout = fopen("output.zst", "wb");
    
    ZSTD_inBuffer input = { inBuff, 0, 0 };
    ZSTD_outBuffer output = { outBuff, outSize, 0 };
    
    /* 3. Main compression loop */
    while (1) {
        /* Refill input buffer when empty */
        if (input.pos == input.size) {
            input.size = fread(inBuff, 1, inSize, fin);
            input.pos = 0;
            if (input.size == 0) break;  /* EOF */
        }
        
        size_t const ret = ZSTD_compressStream2(
            cstream, &output, &input, ZSTD_e_continue);
        
        if (ZSTD_isError(ret)) {
            fprintf(stderr, "Compress error: %s\n", ZSTD_getErrorName(ret));
            return 1;
        }
        
        /* Flush output when full */
        if (output.pos == output.size) {
            fwrite(outBuff, 1, output.pos, fout);
            output.pos = 0;
        }
    }
    
    /* 4. Finalize frame */
    while (1) {
        size_t const ret = ZSTD_compressStream2(
            cstream, &output, &input, ZSTD_e_end);
        
        if (ZSTD_isError(ret)) {
            fprintf(stderr, "End error: %s\n", ZSTD_getErrorName(ret));
            return 1;
        }
        
        fwrite(outBuff, 1, output.pos, fout);
        output.pos = 0;
        if (ret == 0) break;  /* Frame complete */
    }
    
    /* Cleanup */
    fclose(fin);
    fclose(fout);
    free(inBuff);
    free(outBuff);
    ZSTD_freeCStream(cstream);
    
    return 0;
}

```

## Context Reuse and Reset Strategies

A `ZSTD_CStream` context can be reused for multiple independent compression jobs without reallocation, improving performance in server applications. To reset the context while preserving compression parameters (level, dictionary, etc.), call `ZSTD_CCtx_reset()` with `ZSTD_reset_session_only`. To reset both the session and all parameters to defaults, use `ZSTD_reset_session_and_parameters`.

```c
/* Reset for new file, keep level=3 setting */
ZSTD_CCtx_reset(cstream, ZSTD_reset_session_only);
ZSTD_initCStream(cstream, 3);  /* Re-init with same or different level */

```

## Summary

- **Create** a streaming context with `ZSTD_createCStream()` and initialize it using `ZSTD_initCStream()` before compression begins.
- **Optimize** performance by allocating buffers sized according to `ZSTD_CStreamInSize()` and `ZSTD_CStreamOutSize()`.
- **Stream** data incrementally using `ZSTD_compressStream2()` with `ZSTD_e_continue`, checking return values to determine when to flush internal buffers.
- **Finalize** by calling `ZSTD_compressStream2()` with `ZSTD_e_end` until it returns 0, ensuring the zstd frame epilogue is written.
- **Reuse** contexts across jobs using reset functions to minimize allocation overhead in high-throughput applications.

## Frequently Asked Questions

### What is the difference between ZSTD_CStream and ZSTD_CCtx?

`ZSTD_CStream` is a typedef alias for `ZSTD_CCtx` defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) for backward compatibility. Both refer to the same structure handling compression state. The streaming functions operate on this context object to maintain history and parameters across incremental calls.

### How do I handle buffer overflow during streaming compression?

When `ZSTD_compressStream2()` fills the `ZSTD_outBuffer` (indicated by `output.pos == output.size`), write the buffer contents to your destination (file, socket, or network), reset `output.pos` to 0, and call the function again with the same input parameters. The function preserves unconsumed input in the `ZSTD_inBuffer` structure.

### Can I reuse a ZSTD_CStream for multiple files?

Yes. After completing a frame with `ZSTD_e_end`, reset the context using `ZSTD_CCtx_reset(cstream, ZSTD_reset_session_only)` and reinitialize with `ZSTD_initCStream()`. This avoids the overhead of `ZSTD_createCStream()` and `ZSTD_freeCStream()` in batch processing workflows.

### What compression level should I use for streaming?

The default level 3 offers an optimal balance of speed and compression ratio for most streaming applications. Levels 1-22 are supported, with higher levels providing better compression at the cost of memory usage and CPU time. According to the source code in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h), you specify this in `ZSTD_initCStream(cstream, level)` during initialization.