What Do ZSTD_CONTENTSIZE_UNKNOWN and ZSTD_CONTENTSIZE_ERROR Mean in Zstandard?

ZSTD_CONTENTSIZE_UNKNOWN and ZSTD_CONTENTSIZE_ERROR are sentinel values returned by the Zstandard (zstd) API to distinguish between frames with unspecified decompressed sizes and frames that are corrupted or unreadable.

These constants are defined in lib/zstd.h within the facebook/zstd repository and appear throughout the decompression logic to communicate frame state. Understanding how to handle ZSTD_CONTENTSIZE_UNKNOWN versus ZSTD_CONTENTSIZE_ERROR is critical for implementing robust streaming compression and error handling in production applications.

Sentinel Value Definitions

In lib/zstd.h, both symbols are defined as extreme unsigned 64-bit integers to ensure they cannot represent valid content sizes:

• ZSTD_CONTENTSIZE_UNKNOWN — Defined as (0ULL-1) (the maximum unsigned 64-bit value). This indicates that the decompressed size was not stored in the frame header, typically occurring in streaming operations where the source size cannot be determined upfront.
• ZSTD_CONTENTSIZE_ERROR — Defined as (0ULL-2) (one less than the maximum unsigned 64-bit value). This signals that an error occurred while attempting to read the frame header, such as corruption, invalid magic numbers, or insufficient input buffer size.

The relationship ZSTD_CONTENTSIZE_ERROR < ZSTD_CONTENTSIZE_UNKNOWN enables simple validation logic, allowing callers to check for error conditions before handling unknown sizes.

Detecting Frame States in Decompression

The function ZSTD_findDecompressedSize(), implemented in lib/decompress/zstd_decompress.c (line 566), returns these sentinels to report the status of a compressed frame.

Checking for Corrupted Frames

When validating input data, always check for ZSTD_CONTENTSIZE_ERROR first to avoid processing malformed frames. This pattern appears in tests/fuzzer.c (line 1097):

U64 frameSize = ZSTD_findDecompressedSize(compressed, cSize);
if (frameSize == ZSTD_CONTENTSIZE_ERROR) {
    /* Handle corrupted or incomplete frame */
    fprintf(stderr, "Invalid frame header\n");
}

Handling Streaming Data

If the size is unknown, the frame requires streaming decompression or dynamic buffer allocation. The test suite in tests/zstreamtest.c (line 601) demonstrates this verification:

U64 size = ZSTD_findDecompressedSize(buf, bufSize);
if (size == ZSTD_CONTENTSIZE_ERROR) {
    fprintf(stderr, "Corrupted frame!\n");
    exit(1);
}
if (size == ZSTD_CONTENTSIZE_UNKNOWN) {
    printf("Original size not stored – proceeding with streaming logic.\n");
}

Using ZSTD_CONTENTSIZE_UNKNOWN During Compression

When compressing data where the total size is not known in advance, explicitly pass ZSTD_CONTENTSIZE_UNKNOWN to ZSTD_CCtx_setPledgedSrcSize(). This configures the compression context for streaming mode without requiring a predefined content length, as shown in tests/zstreamtest.c (line 1460):

ZSTD_CCtx *cctx = ZSTD_createCCtx();
size_t ret = ZSTD_CCtx_setPledgedSrcSize(cctx, ZSTD_CONTENTSIZE_UNKNOWN);
/* CHECK_Z(ret) verifies the operation succeeded */

This approach prevents the compressor from attempting to optimize for a specific frame size when operating on unbounded streams.

Summary

• ZSTD_CONTENTSIZE_UNKNOWN ((0ULL-1)) represents frames where the decompressed size is not stored in the header, requiring dynamic allocation or streaming logic.
• ZSTD_CONTENTSIZE_ERROR ((0ULL-2)) indicates frame corruption, invalid headers, or truncated input that prevents reading the size field.
• Both constants are defined in lib/zstd.h and returned by functions like ZSTD_findDecompressedSize() to communicate specific frame conditions.
• Always validate against ZSTD_CONTENTSIZE_ERROR before decompression to prevent crashes with malformed data.

Frequently Asked Questions

What is the numeric value of ZSTD_CONTENTSIZE_UNKNOWN?

ZSTD_CONTENTSIZE_UNKNOWN equals (0ULL-1), which evaluates to 18446744073709551615 (the maximum value of an unsigned 64-bit integer). This value is mathematically impossible as a valid frame size in practical applications, making it a safe sentinel.

How do I check if a frame size returned by ZSTD_findDecompressedSize is valid?

Compare the return value against both sentinels in descending order of severity. If the value equals ZSTD_CONTENTSIZE_ERROR, the frame is corrupted. If it equals ZSTD_CONTENTSIZE_UNKNOWN, the size is valid but unspecified. Any other unsigned 64-bit value represents the exact decompressed size in bytes.

Can ZSTD_CONTENTSIZE_ERROR indicate a buffer that is too small?

Yes. ZSTD_CONTENTSIZE_ERROR is returned whenever the frame header cannot be parsed correctly, including cases where the input buffer is truncated or the magic number is invalid. It serves as a general error indicator for any failure to extract size information from the frame header.

Should I treat ZSTD_CONTENTSIZE_UNKNOWN as an error condition?

No. An unknown content size is a valid state for streaming compression where the total input size cannot be determined before processing begins. Unlike ZSTD_CONTENTSIZE_ERROR, this sentinel indicates successful parsing of a frame that simply lacks size metadata, requiring the application to use streaming decompression APIs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}