# How to Handle Input and Output Buffers in Zstd Streaming

> Learn to manage Zstd input and output buffers for efficient streaming compression and decompression. Use ZSTD_inBuffer and ZSTD_outBuffer structures for incremental data processing.

- Repository: [Meta/zstd](https://github.com/facebook/zstd)
- Tags: how-to-guide
- Published: 2026-09-09

---

**Zstandard (zstd) streaming compression and decompression rely on the `ZSTD_inBuffer` and `ZSTD_outBuffer` structures with cursor-based consumption, recommended buffer sizes from helper functions like `ZSTD_CStreamInSize()`, and loop-driven calls to `ZSTD_compressStream2()` or `ZSTD_decompressStream()` to process data incrementally without loading entire datasets into memory.**

The Zstandard library (`facebook/zstd`) provides a low-level streaming API that enables applications to compress or decompress data in chunks rather than requiring complete files in memory. Properly managing the lifecycle of input and output buffers—from allocation using size helpers to cursor tracking during the streaming loop—is critical for both performance and correctness. This guide covers the buffer structures defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h), the recommended allocation patterns, and the exact workflow implemented in the official examples.

## Understanding the Buffer Structures

At the core of zstd streaming are two C structs that act as sliding windows for data transfer between your application and the library.

### ZSTD_inBuffer for Input Data

**`ZSTD_inBuffer`** manages the source data flowing into the compressor or decompressor. Defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) (lines 701-705), the structure contains:

- `src`: A `const void*` pointer to the input data array
- `size`: The total number of bytes available in the buffer
- `pos`: A cursor that the library increments as it consumes bytes

The library updates `pos` to indicate how many bytes have been read, allowing you to track partial consumption without modifying the underlying data.

### ZSTD_outBuffer for Output Data

**`ZSTD_outBuffer`** receives the processed result. Declared in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) (lines 707-711), this structure contains:

- `dst`: A `void*` pointer to the destination memory region
- `size`: The allocated capacity of the buffer in bytes
- `pos`: A write cursor that the library updates as it produces output

After each streaming call, `output.pos` indicates how many valid bytes are ready to be written to your storage or network interface.

## Selecting Optimal Buffer Sizes

Zstd provides specific helper functions to determine buffer sizes that minimize internal copying and guarantee that at least one full compressed block can be processed per call. These are declared in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) (lines 842-844).

For compression streaming:
- Use **`ZSTD_CStreamInSize()`** to get the recommended input buffer size
- Use **`ZSTD_CStreamOutSize()`** to get the recommended output buffer size

For decompression streaming:
- Use **`ZSTD_DStreamInSize()`** to get the recommended input buffer size
- Use **`ZSTD_DStreamOutSize()`** to get the recommended output buffer size

Allocating buffers according to these values ensures that `ZSTD_compressStream2()` and `ZSTD_decompressStream()` operate efficiently without requiring excessive internal buffering.

## Streaming Compression Workflow

The compression implementation follows a pattern demonstrated in [`examples/streaming_compression.c`](https://github.com/facebook/zstd/blob/main/examples/streaming_compression.c) (lines 31-34 and 71-85). The workflow involves creating a context, allocating properly sized buffers, and looping until all input is consumed and flushed.

1. Create a compression context with `ZSTD_createCCtx()`
2. Allocate buffers using `ZSTD_CStreamInSize()` and `ZSTD_CStreamOutSize()`
3. Populate the input buffer, set `input.pos` to 0, and set `input.size` to the bytes read
4. Call `ZSTD_compressStream2()` with the appropriate **ZSTD_EndDirective**
5. Write `output.pos` bytes to the destination
6. Continue until `input.pos` equals `input.size` and the flush is complete

The **`ZSTD_EndDirective`** parameter controls frame boundaries:
- **`ZSTD_e_continue`**: Indicates more data is coming; the library may buffer internally for better compression
- **`ZSTD_e_end`**: Signals the final chunk, forcing the library to flush all internal buffers and complete the frame

```c
/* Compression example using recommended buffer sizes */
size_t inSize  = ZSTD_CStreamInSize();
size_t outSize = ZSTD_CStreamOutSize();
void* inBuf  = malloc(inSize);
void* outBuf = malloc(outSize);

ZSTD_CCtx* cctx = ZSTD_createCCtx();
ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 3);

while ((read = fread(inBuf, 1, inSize, fin)) > 0) {
    ZSTD_inBuffer input = { inBuf, read, 0 };
    int lastChunk = (read < inSize);
    ZSTD_EndDirective mode = lastChunk ? ZSTD_e_end : ZSTD_e_continue;

    do {
        ZSTD_outBuffer output = { outBuf, outSize, 0 };
        size_t remaining = ZSTD_compressStream2(cctx, &output, &input, mode);
        fwrite(outBuf, 1, output.pos, fout);
    } while (input.pos < input.size);
}
ZSTD_freeCCtx(cctx);
free(inBuf);
free(outBuf);

```

## Streaming Decompression Workflow

Decompression uses a similar pattern with `ZSTD_createDCtx()` and `ZSTD_decompressStream()`, as shown in [`examples/streaming_decompression.c`](https://github.com/facebook/zstd/blob/main/examples/streaming_decompression.c) (lines 20-24 and 46-60). The process continues until the function returns `0`, indicating the frame is fully decoded.

1. Create a decompression context with `ZSTD_createDCtx()`
2. Allocate buffers using `ZSTD_DStreamInSize()` and `ZSTD_DStreamOutSize()`
3. Fill the input buffer with compressed data
4. Call `ZSTD_decompressStream()` in an inner loop while `input.pos < input.size`
5. Write `output.pos` bytes to the destination
6. Continue until the function returns `0` (frame complete)

```c
/* Decompression example with proper buffer handling */
size_t inSize  = ZSTD_DStreamInSize();
size_t outSize = ZSTD_DStreamOutSize();
void* inBuf  = malloc(inSize);
void* outBuf = malloc(outSize);

ZSTD_DCtx* dctx = ZSTD_createDCtx();

while ((read = fread(inBuf, 1, inSize, fin)) > 0) {
    ZSTD_inBuffer input = { inBuf, read, 0 };
    while (input.pos < input.size) {
        ZSTD_outBuffer output = { outBuf, outSize, 0 };
        size_t ret = ZSTD_decompressStream(dctx, &output, &input);
        fwrite(outBuf, 1, output.pos, fout);
        if (ZSTD_isError(ret)) { /* handle error */ }
    }
}
ZSTD_freeDCtx(dctx);
free(inBuf);
free(outBuf);

```

## Buffer Management Best Practices

Proper management of buffer lifecycles ensures memory efficiency and prevents data corruption in multi-threaded environments.

**Buffer Reuse**: After each call to `ZSTD_compressStream2()` or `ZSTD_decompressStream()`, reuse the same memory allocations by resetting the `pos` fields to zero. This eliminates allocation overhead during large file processing.

**Partial Consumption Handling**: When the output buffer fills before all input is consumed, the streaming function returns a positive value indicating remaining work. You must drain the output buffer (write `output.pos` bytes), reset `output.pos` to zero, and call the function again with the same input buffer—which retains its `pos` cursor—before advancing to new input.

**Thread Safety**: Each thread must maintain its own `ZSTD_CCtx` or `ZSTD_DCtx`. While the compression and decompression contexts are not thread-safe, the buffer memory itself has no thread affinity; you may allocate separate buffers per thread or reuse heap memory with proper synchronization.

**Error Detection**: All streaming APIs return `size_t` values that represent either the number of bytes still pending or an error code. Always validate returns using `ZSTD_isError()` before proceeding.

## Summary

- Use **`ZSTD_inBuffer`** and **`ZSTD_outBuffer`** with `pos` cursors to track byte-level consumption without data copying
- Allocate buffers using **`ZSTD_CStreamInSize()`** / **`ZSTD_CStreamOutSize()`** for compression or **`ZSTD_DStreamInSize()`** / **`ZSTD_DStreamOutSize()`** for decompression
- Implement compression loops with **`ZSTD_compressStream2()`**, using **`ZSTD_e_end`** for the final chunk to flush the frame
- Implement decompression loops with **`ZSTD_decompressStream()`**, continuing until the function returns `0` indicating frame completion
- Validate all return values with **`ZSTD_isError()`** and handle partial output by looping until `input.pos` reaches `input.size`
- Refer to [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h) for struct definitions and [`examples/streaming_compression.c`](https://github.com/facebook/zstd/blob/main/examples/streaming_compression.c) for production-ready implementation patterns

## Frequently Asked Questions

### What happens if the output buffer is too small during zstd streaming?

If the output buffer fills before the operation completes, the library pauses and returns a positive value indicating remaining work. The `output.pos` field contains the number of valid bytes ready for writing. You must write those bytes to your destination, reset `output.pos` to zero, and call the streaming function again with the same input buffer (which retains its `pos` cursor) to continue processing the remaining data.

### Can I reuse the same buffers across multiple zstd operations?

Yes. After each call to `ZSTD_compressStream2()` or `ZSTD_decompressStream()`, you can reuse the same memory allocations. Simply reset the `pos` field of both buffers to zero before the next operation. This pattern minimizes memory allocation overhead during batch processing or large file streaming.

### How do I know when a zstd decompression frame is complete?

The `ZSTD_decompressStream()` function returns exactly `0` when the current frame has been completely decoded and all buffered output has been flushed. Until you receive this return value, continue providing compressed input and emptying the output buffer. Use `ZSTD_isError()` to distinguish between successful completion and error conditions.

### Do compression and decompression require different buffer sizes?

Yes. Compression and decompression use different internal block sizes and window constraints. For compression streaming, use **`ZSTD_CStreamInSize()`** and **`ZSTD_CStreamOutSize()`**. For decompression streaming, use **`ZSTD_DStreamInSize()`** and **`ZSTD_DStreamOutSize()`**. These functions ensure your buffers meet the library's minimum requirements for efficient block processing as defined in [`lib/zstd.h`](https://github.com/facebook/zstd/blob/main/lib/zstd.h).