Creating Reproducible ZIP Archives with Deterministic Timestamps in sdata

The ZipFile class in sdata generates byte-for-byte reproducible ZIP archives by defaulting deterministic=True in to_zip(), which forces all entry timestamps to the fixed tuple (1980, 1, 1, 0, 0, 0) instead of the current filesystem time.

Creating reproducible ZIP archives with deterministic timestamps is essential for scientific data sharing and version control, yet Python’s standard zipfile module automatically embeds the current filesystem modification time into every archive entry. The sdata library solves this through its ZipFile implementation in sdata/sclass/zipfile.py, ensuring that identical input data always produces a binary-identical output archive regardless of when the bundle is created.

Why Standard ZIP Archives Fail at Reproducibility

By default, the Python zipfile module stores the current system time in the metadata of each compressed file. Because these timestamps change with every execution, two archives created from the same dataset minutes apart will have different binary hashes, breaking checksum-based verification and complicating scientific reproducibility. This nondeterminism occurs even when the underlying file contents and their SHA-3-256 hashes remain completely unchanged.

How sdata Implements Deterministic Timestamps

The deterministic Flag in ZipFile.to_zip()

In sdata/sclass/zipfile.py, the ZipFile.to_zip() method provides a deterministic parameter that defaults to True. When enabled, the method manually constructs a zipfile.ZipInfo object for every entry rather than relying on the automatic timestamp collection. As implemented in the source, the code explicitly sets the date_time attribute to a constant value:

info = zipfile.ZipInfo(
    arcname,
    date_time=(1980, 1, 1, 0, 0, 0)   # fixed timestamp

)
info.compress_type = compression

This fixed metadata eliminates the variable timestamp that typically causes binary drift, while the actual file contents—stored as JSON-serialized FileReference objects via to_json()—remain cryptographically intact.

Preserving Scientific Integrity

Because the deterministic path only modifies the archive’s metadata headers, the internal representation of each FileReference (including its SHA-3-256 hash stored under _sdata_sha3_256) remains unchanged. The method writes these JSON payloads using zf.writestr(info, json_str), ensuring that the scientific integrity of the data is preserved while guaranteeing that two runs produce identical ZIP files.

Creating and Verifying Deterministic Archives

Basic Deterministic Export

To create a reproducible archive, initialize a ZipFile instance, add your files, and call to_zip():

from sdata.sclass.zipfile import ZipFile
from pathlib import Path

# Initialize the container

z = ZipFile(name="experiment-archive")

# Add FileReferences (any local path)

z.add("/data/experiment_01.csv")
z.add("/data/metadata.json")

# Generate deterministic archive (default behavior)

zip_bytes = z.to_zip(filepath=Path("out/experiment.zip"))

The resulting out/experiment.zip will be byte-identical across every execution with the same inputs.

Disabling Determinism for Original Timestamps

If you require the original filesystem timestamps (for example, to preserve audit trails), set deterministic=False:

zip_bytes = z.to_zip(deterministic=False)   # Uses current system timestamps

Loading Archives Back into sdata

The from_zip() classmethod reconstructs the ZipFile object and its FileReference collection from either a file path or an in-memory buffer:

from sdata.sclass.zipfile import ZipFile

# Load from disk

z_loaded = ZipFile.from_zip("out/experiment.zip")

# Or load from BytesIO buffer

z_loaded = ZipFile.from_zip(zip_bytes)

# Access stored references and their hashes

for fr in z_loaded.get_filereferences():
    print(fr.sname, fr.filetype, fr.metadata.get("_sdata_sha3_256").value)

Verifying Byte-For-Byte Reproducibility

You can programmatically confirm that two archives are identical using standard hash functions:

import hashlib
import pathlib

def sha256_of_file(p: pathlib.Path) -> str:
    return hashlib.sha256(p.read_bytes()).hexdigest()

# Create two archives from identical inputs

z1 = ZipFile()
z1.add("sample.txt")
z1.to_zip(filepath=pathlib.Path("out/run1.zip"))

z2 = ZipFile()
z2.add("sample.txt")
z2.to_zip(filepath=pathlib.Path("out/run2.zip"))

# Verify binary equality

assert sha256_of_file(pathlib.Path("out/run1.zip")) == \
       sha256_of_file(pathlib.Path("out/run2.zip")), "Archives differ!"

Summary

• sdata/sclass/zipfile.py contains the ZipFile class with deterministic archive capabilities via to_zip().
• The deterministic=True default forces all ZIP entries to the fixed timestamp (1980, 1, 1, 0, 0, 0), eliminating binary drift from metadata.
• File contents remain unchanged through JSON serialization of FileReference objects, preserving SHA-3-256 hashes and scientific data integrity.
• Archives can be reconstructed using from_zip(), which parses the stored JSON metadata back into FileReference instances.
• Reproducibility can be verified using standard checksums, as identical inputs always yield identical binary outputs.

Frequently Asked Questions

What timestamp does sdata use for deterministic archives?

When deterministic=True (the default), sdata sets the ZIP entry metadata to the fixed tuple (1980, 1, 1, 0, 0, 0) for every file. This constant date-time value corresponds to the earliest representable DOS date, ensuring that the archive header remains identical across all creation times.

Does deterministic mode affect the file contents or their hashes?

No. The deterministic flag only modifies the ZIP container’s metadata headers (specifically the date_time field in ZipInfo). The actual file payloads are JSON-serialized FileReference objects containing the original SHA-3-256 hashes, which remain cryptographically unchanged and identical to the source data.

How do I load a deterministic ZIP back into sdata for inspection?

Use the ZipFile.from_zip() classmethod, which accepts either a file path string or a BytesIO buffer. This method parses the JSON metadata within the ZIP and reconstructs the original FileReference objects, allowing you to access file names, types, and verification hashes via get_filereferences().

Can I create non-deterministic archives if I need original timestamps?

Yes. Pass deterministic=False to the to_zip() method. This bypasses the custom ZipInfo creation and allows the standard zipfile module to record the current filesystem modification times, which is useful when audit trails requiring original creation times are necessary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}