# How to Handle Zstandard Compressed Traces Directly in libCacheSim

> Effortlessly handle Zstandard compressed traces in libCacheSim. Stream decompressed data on-the-fly with the SUPPORT_ZSTD_TRACE flag. No code changes needed.

- Repository: [Juncheng Yang/libcachesim](https://github.com/1a1a11a/libcachesim)
- Tags: how-to-guide
- Published: 2026-02-23

---

**libCacheSim automatically detects `.zst` file extensions and streams decompressed trace data on-the-fly when the `SUPPORT_ZSTD_TRACE` compile-time flag is enabled, requiring no code changes to existing simulation workflows.**

libCacheSim, the high-performance cache simulation library from the 1a1a11a/libcachesim repository, provides native support for handling Zstandard (ZSTD) compressed traces directly without manual decompression. This capability allows researchers to store large trace datasets in compressed form while maintaining efficient streaming read performance during cache simulations.

## Enabling ZSTD Support at Compile Time

ZSTD support is controlled by the `OPT_SUPPORT_ZSTD_TRACE` CMake option, which defaults to `ON` in the root [`CMakeLists.txt`](https://github.com/1a1a11a/libcachesim/blob/main/CMakeLists.txt):

```cmake
option(OPT_SUPPORT_ZSTD_TRACE "whether support zstd trace" ON)
if (OPT_SUPPORT_ZSTD_TRACE)
    add_compile_definitions(SUPPORT_ZSTD_TRACE=1)
endif()

```

*Source:* [[`CMakeLists.txt`](https://github.com/1a1a11a/libcachesim/blob/main/CMakeLists.txt)](https://github.com/1a1a11a/libcachesim/blob/develop/CMakeLists.txt#L26-L41)

To explicitly disable ZSTD support, configure the build with:

```bash
cmake -DOPT_SUPPORT_ZSTD_TRACE=OFF ..

```

## How libCacheSim Detects and Processes ZSTD Files

When `SUPPORT_ZSTD_TRACE` is defined, the trace reader infrastructure automatically handles `.zst` files through a streaming decompression pipeline implemented across the reader subsystem.

### File Type Detection in reader.c

In [`libCacheSim/traceReader/reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/reader.c), the `setup_reader` function inspects the file name suffix to determine compression type:

```c
if (strlen(trace_path) > 4 && 
    strcmp(trace_path + strlen(trace_path) - 4, ".zst") == 0) {
    reader->is_zstd_file = true;
}

```

*Source:* [[`reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/reader.c) lines 53-62](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L53-L62)

### Initializing the ZSTD Decompression Stream

When a ZSTD file is detected, `setup_reader` invokes `create_zstd_reader(trace_path)` from [`libCacheSim/traceReader/generalReader/zstdReader.c`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/generalReader/zstdReader.c). This function:

1. Opens the file in binary mode using `fopen`
2. Allocates input and output buffers sized to `ZSTD_DStreamInSize()` and `ZSTD_DStreamOutSize()` respectively
3. Instantiates a ZSTD decompression stream via `ZSTD_createDStream`

*Source:* [[`zstdReader.c`](https://github.com/1a1a11a/libcachesim/blob/main/zstdReader.c) lines 17-45](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/generalReader/zstdReader.c#L17-L45)

### Streaming Decompression During Trace Processing

During trace processing, `read_one_req` dispatches to format-specific readers such as `binary_read_one_req`. For ZSTD files, the binary utilities in [`libCacheSim/traceReader/customizedReader/binaryUtils.h`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/customizedReader/binaryUtils.h) switch to ZSTD-aware read functions:

```c
static inline size_t _read_bytes_zstd(reader_t *reader, void *buf, size_t len) {
    // Reads from the ZSTD decompression buffer, refilling from
    // the compressed stream when necessary
    return zstd_reader_read_bytes(reader, buf, len);
}

```

*Source:* [[`binaryUtils.h`](https://github.com/1a1a11a/libcachesim/blob/main/binaryUtils.h) lines 26-33](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/customizedReader/binaryUtils.h#L26-L33)

For line-oriented formats, `zstd_reader_read_line` extracts complete lines (including trailing newlines) from the decompressed output, handling cases where lines span multiple ZSTD frames.

*Source:* [[`zstdReader.c`](https://github.com/1a1a11a/libcachesim/blob/main/zstdReader.c) lines 26-71](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/generalReader/zstdReader.c#L26-L71)

### Reader Reset and Cleanup

To support multiple simulation passes, `reset_zstd_reader` rewinds the underlying file pointer and reinitializes the ZSTD stream. This is invoked by `reset_reader` in [`reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/reader.c) when `reader->is_zstd_file` is true.

*Source:* [[`reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/reader.c) lines 86-99](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L86-L99)

When the simulation completes, `close_reader` calls `free_zstd_reader` to release the ZSTD context and buffers.

*Source:* [[`reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/reader.c) lines 76-80](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L76-L80)

## Practical Examples for Handling ZSTD Traces

### C API Example

The following program demonstrates how to handle ZSTD compressed traces directly using the libCacheSim C API. The code is identical to uncompressed trace handling; the library automatically detects the `.zst` extension:

```c
#include "libCacheSim/reader.h"
#include "libCacheSim/simulator.h"

int main() {
    /* Build a reader for a ZSTD-compressed binary trace */
    reader_t *r = setup_reader(
        "data/twitter_cluster52_10m.csv.zst",   // .zst file
        BIN_TRACE,                              // binary trace type
        NULL);                                  // default init params

    /* Configure a simple LRU cache */
    cache_config_t cfg = {
        .cache_size = 1024 * 1024 * 1024,       // 1 GiB
        .block_size = 64 * 1024,                // 64 KiB
        .admission_algo = ADMISSION_NONE,
        .eviction_algo = EVICTION_LRU
    };
    cache_t *c = cache_create(&cfg);

    /* Run the simulator */
    simulator(r, c, NULL);

    /* Clean up */
    free_cache(c);
    close_reader(r);
    return 0;
}

```

### Command-Line Usage

The `cachesim` binary installed by the project automatically handles ZSTD compression:

```bash

# Build with default options (ZSTD support is ON)

mkdir build && cd build
cmake .. && ninja            # or make

sudo ninja install          # installs cachesim

# Run a simulation on a compressed trace

cachesim -t data/twitter_cluster52_10m.csv.zst \
         -c LRU \
         -s 1G \
         -b 64K

```

The `-t` option accepts any file path; if the extension is `.zst`, the tool transparently decompresses the stream during processing.

### Generating ZSTD Compressed Traces

To compress an existing trace for use with libCacheSim, use the Python `zstandard` library:

```python
import zstandard as zstd

in_path = "raw_trace.csv"
out_path = "raw_trace.csv.zst"

cctx = zstd.ZstdCompressor()
with open(in_path, "rb") as fin, open(out_path, "wb") as fout:
    cctx.copy_stream(fin, fout)

```

The resulting `.zst` file can be passed directly to libCacheSim without modification.

## Key Implementation Files and Functions

| File | Role | Key Functions |
|------|------|---------------|
| [`libCacheSim/traceReader/reader.c`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/reader.c) | Central entry point for trace reading | `setup_reader`, `read_one_req`, `reset_reader`, `close_reader` |
| [`libCacheSim/traceReader/generalReader/zstdReader.c`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/generalReader/zstdReader.c) | ZSTD-specific streaming implementation | `create_zstd_reader`, `free_zstd_reader`, `reset_zstd_reader`, `zstd_reader_read_line` |
| [`libCacheSim/traceReader/generalReader/zstdReader.h`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/generalReader/zstdReader.h) | Public API for ZSTD reader | Header declarations for all zstdReader functions |
| [`libCacheSim/traceReader/customizedReader/binaryUtils.h`](https://github.com/1a1a11a/libcachesim/blob/main/libCacheSim/traceReader/customizedReader/binaryUtils.h) | Binary read utilities with ZSTD support | `_read_bytes_zstd` |
| [`CMakeLists.txt`](https://github.com/1a1a11a/libcachesim/blob/main/CMakeLists.txt) | Build configuration | `OPT_SUPPORT_ZSTD_TRACE` option |

## Limitations of ZSTD Trace Handling

While libCacheSim provides robust ZSTD support, be aware of these constraints:

- **Sequential access only**: Because ZSTD does not support efficient seeking, `reader_set_read_pos` is unavailable for compressed traces. The library falls back to sequential reads and provides `reset_zstd_reader` to rewind the stream to the beginning for multiple simulation passes.
- **Binary format optimization**: While line-oriented reads work via `zstd_reader_read_line`, the highest performance is typically achieved with binary trace formats that use the `_read_bytes_zstd` helper for efficient byte-level access.

## Summary

- **libCacheSim** automatically handles Zstandard compressed traces when the `SUPPORT_ZSTD_TRACE` compile-time flag is enabled (default).
- The library detects `.zst` file extensions in `setup_reader` and initializes a streaming decompressor via `create_zstd_reader`.
- Binary and line-oriented trace formats are supported through specialized helpers like `_read_bytes_zstd` and `zstd_reader_read_line`.
- No API changes are required—existing C code and command-line tools work identically with compressed or uncompressed traces.
- Sequential access is required for ZSTD files; random seeking is not supported due to compression format limitations.

## Frequently Asked Questions

### Does libCacheSim require manual decompression before running simulations?

No. When built with the default `OPT_SUPPORT_ZSTD_TRACE=ON` CMake option, libCacheSim detects `.zst` file extensions automatically and streams decompressed data on-the-fly. You can pass compressed files directly to the `cachesim` CLI or `setup_reader` C API exactly as you would with uncompressed files.

### What CMake flag controls ZSTD support in libCacheSim?

The `OPT_SUPPORT_ZSTD_TRACE` option in the root [`CMakeLists.txt`](https://github.com/1a1a11a/libcachesim/blob/main/CMakeLists.txt) controls this feature. It defaults to `ON`, defining `SUPPORT_ZSTD_TRACE` at compile time. To disable ZSTD support and reduce binary size, configure with `-DOPT_SUPPORT_ZSTD_TRACE=OFF`.

### Can I seek to arbitrary positions within a ZSTD compressed trace?

No. Due to the nature of Zstandard compression, libCacheSim does not support random access seeking via `reader_set_read_pos` for `.zst` files. The library falls back to sequential reads and provides `reset_zstd_reader` to rewind the stream to the beginning for multiple simulation passes.

### Which trace formats work with ZSTD compression in libCacheSim?

Both binary and line-oriented trace formats are supported. Binary traces use the `_read_bytes_zstd` helper in [`binaryUtils.h`](https://github.com/1a1a11a/libcachesim/blob/main/binaryUtils.h) for efficient byte-level reads, while text-based formats rely on `zstd_reader_read_line` to extract complete lines from the decompression buffer. The highest performance is typically achieved with binary trace formats.