How to Handle Zstandard Compressed Traces Directly in libCacheSim

libCacheSim automatically detects .zst file extensions and streams decompressed trace data on-the-fly when the SUPPORT_ZSTD_TRACE compile-time flag is enabled, requiring no code changes to existing simulation workflows.

libCacheSim, the high-performance cache simulation library from the 1a1a11a/libcachesim repository, provides native support for handling Zstandard (ZSTD) compressed traces directly without manual decompression. This capability allows researchers to store large trace datasets in compressed form while maintaining efficient streaming read performance during cache simulations.

Enabling ZSTD Support at Compile Time

ZSTD support is controlled by the OPT_SUPPORT_ZSTD_TRACE CMake option, which defaults to ON in the root CMakeLists.txt:

option(OPT_SUPPORT_ZSTD_TRACE "whether support zstd trace" ON)
if (OPT_SUPPORT_ZSTD_TRACE)
    add_compile_definitions(SUPPORT_ZSTD_TRACE=1)
endif()

Source: [CMakeLists.txt](https://github.com/1a1a11a/libcachesim/blob/develop/CMakeLists.txt#L26-L41)

To explicitly disable ZSTD support, configure the build with:

cmake -DOPT_SUPPORT_ZSTD_TRACE=OFF ..

How libCacheSim Detects and Processes ZSTD Files

When SUPPORT_ZSTD_TRACE is defined, the trace reader infrastructure automatically handles .zst files through a streaming decompression pipeline implemented across the reader subsystem.

File Type Detection in reader.c

In libCacheSim/traceReader/reader.c, the setup_reader function inspects the file name suffix to determine compression type:

if (strlen(trace_path) > 4 && 
    strcmp(trace_path + strlen(trace_path) - 4, ".zst") == 0) {
    reader->is_zstd_file = true;
}

Source: [reader.c lines 53-62](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L53-L62)

Initializing the ZSTD Decompression Stream

When a ZSTD file is detected, setup_reader invokes create_zstd_reader(trace_path) from libCacheSim/traceReader/generalReader/zstdReader.c. This function:

1. Opens the file in binary mode using fopen
2. Allocates input and output buffers sized to ZSTD_DStreamInSize() and ZSTD_DStreamOutSize() respectively
3. Instantiates a ZSTD decompression stream via ZSTD_createDStream

Source: [zstdReader.c lines 17-45](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/generalReader/zstdReader.c#L17-L45)

Streaming Decompression During Trace Processing

During trace processing, read_one_req dispatches to format-specific readers such as binary_read_one_req. For ZSTD files, the binary utilities in libCacheSim/traceReader/customizedReader/binaryUtils.h switch to ZSTD-aware read functions:

static inline size_t _read_bytes_zstd(reader_t *reader, void *buf, size_t len) {
    // Reads from the ZSTD decompression buffer, refilling from
    // the compressed stream when necessary
    return zstd_reader_read_bytes(reader, buf, len);
}

Source: [binaryUtils.h lines 26-33](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/customizedReader/binaryUtils.h#L26-L33)

For line-oriented formats, zstd_reader_read_line extracts complete lines (including trailing newlines) from the decompressed output, handling cases where lines span multiple ZSTD frames.

Source: [zstdReader.c lines 26-71](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/generalReader/zstdReader.c#L26-L71)

Reader Reset and Cleanup

To support multiple simulation passes, reset_zstd_reader rewinds the underlying file pointer and reinitializes the ZSTD stream. This is invoked by reset_reader in reader.c when reader->is_zstd_file is true.

Source: [reader.c lines 86-99](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L86-L99)

When the simulation completes, close_reader calls free_zstd_reader to release the ZSTD context and buffers.

Source: [reader.c lines 76-80](https://github.com/1a1a11a/libcachesim/blob/develop/libCacheSim/traceReader/reader.c#L76-L80)

Practical Examples for Handling ZSTD Traces

C API Example

The following program demonstrates how to handle ZSTD compressed traces directly using the libCacheSim C API. The code is identical to uncompressed trace handling; the library automatically detects the .zst extension:

#include "libCacheSim/reader.h"
#include "libCacheSim/simulator.h"

int main() {
    /* Build a reader for a ZSTD-compressed binary trace */
    reader_t *r = setup_reader(
        "data/twitter_cluster52_10m.csv.zst",   // .zst file
        BIN_TRACE,                              // binary trace type
        NULL);                                  // default init params

    /* Configure a simple LRU cache */
    cache_config_t cfg = {
        .cache_size = 1024 * 1024 * 1024,       // 1 GiB
        .block_size = 64 * 1024,                // 64 KiB
        .admission_algo = ADMISSION_NONE,
        .eviction_algo = EVICTION_LRU
    };
    cache_t *c = cache_create(&cfg);

    /* Run the simulator */
    simulator(r, c, NULL);

    /* Clean up */
    free_cache(c);
    close_reader(r);
    return 0;
}

Command-Line Usage

The cachesim binary installed by the project automatically handles ZSTD compression:

# Build with default options (ZSTD support is ON)

mkdir build && cd build
cmake .. && ninja            # or make

sudo ninja install          # installs cachesim

# Run a simulation on a compressed trace

cachesim -t data/twitter_cluster52_10m.csv.zst \
         -c LRU \
         -s 1G \
         -b 64K

The -t option accepts any file path; if the extension is .zst, the tool transparently decompresses the stream during processing.

Generating ZSTD Compressed Traces

To compress an existing trace for use with libCacheSim, use the Python zstandard library:

import zstandard as zstd

in_path = "raw_trace.csv"
out_path = "raw_trace.csv.zst"

cctx = zstd.ZstdCompressor()
with open(in_path, "rb") as fin, open(out_path, "wb") as fout:
    cctx.copy_stream(fin, fout)

The resulting .zst file can be passed directly to libCacheSim without modification.

Key Implementation Files and Functions

File	Role	Key Functions
libCacheSim/traceReader/reader.c	Central entry point for trace reading	setup_reader, read_one_req, reset_reader, close_reader
libCacheSim/traceReader/generalReader/zstdReader.c	ZSTD-specific streaming implementation	create_zstd_reader, free_zstd_reader, reset_zstd_reader, zstd_reader_read_line
libCacheSim/traceReader/generalReader/zstdReader.h	Public API for ZSTD reader	Header declarations for all zstdReader functions
libCacheSim/traceReader/customizedReader/binaryUtils.h	Binary read utilities with ZSTD support	_read_bytes_zstd
CMakeLists.txt	Build configuration	OPT_SUPPORT_ZSTD_TRACE option

Limitations of ZSTD Trace Handling

While libCacheSim provides robust ZSTD support, be aware of these constraints:

• Sequential access only: Because ZSTD does not support efficient seeking, reader_set_read_pos is unavailable for compressed traces. The library falls back to sequential reads and provides reset_zstd_reader to rewind the stream to the beginning for multiple simulation passes.
• Binary format optimization: While line-oriented reads work via zstd_reader_read_line, the highest performance is typically achieved with binary trace formats that use the _read_bytes_zstd helper for efficient byte-level access.

Summary

• libCacheSim automatically handles Zstandard compressed traces when the SUPPORT_ZSTD_TRACE compile-time flag is enabled (default).
• The library detects .zst file extensions in setup_reader and initializes a streaming decompressor via create_zstd_reader.
• Binary and line-oriented trace formats are supported through specialized helpers like _read_bytes_zstd and zstd_reader_read_line.
• No API changes are required—existing C code and command-line tools work identically with compressed or uncompressed traces.
• Sequential access is required for ZSTD files; random seeking is not supported due to compression format limitations.

Frequently Asked Questions

Does libCacheSim require manual decompression before running simulations?

No. When built with the default OPT_SUPPORT_ZSTD_TRACE=ON CMake option, libCacheSim detects .zst file extensions automatically and streams decompressed data on-the-fly. You can pass compressed files directly to the cachesim CLI or setup_reader C API exactly as you would with uncompressed files.

What CMake flag controls ZSTD support in libCacheSim?

The OPT_SUPPORT_ZSTD_TRACE option in the root CMakeLists.txt controls this feature. It defaults to ON, defining SUPPORT_ZSTD_TRACE at compile time. To disable ZSTD support and reduce binary size, configure with -DOPT_SUPPORT_ZSTD_TRACE=OFF.

Can I seek to arbitrary positions within a ZSTD compressed trace?

No. Due to the nature of Zstandard compression, libCacheSim does not support random access seeking via reader_set_read_pos for .zst files. The library falls back to sequential reads and provides reset_zstd_reader to rewind the stream to the beginning for multiple simulation passes.

Which trace formats work with ZSTD compression in libCacheSim?

Both binary and line-oriented trace formats are supported. Binary traces use the _read_bytes_zstd helper in binaryUtils.h for efficient byte-level reads, while text-based formats rely on zstd_reader_read_line to extract complete lines from the decompression buffer. The highest performance is typically achieved with binary trace formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}