How the Artifact Format and Zstd Compression Work in Codebase-Memory-MCP

Codebase-Memory-MCP packages SQLite databases into Zstandard-compressed artifacts with JSON metadata and Git-safe attributes, using atomic writes and frame-size validation to prevent corruption and resource exhaustion.

The Codebase-Memory-MCP project enables teams to share code intelligence through a portable artifact format that encapsulates entire SQLite databases. This system leverages Zstandard (zstd) compression to minimize storage footprint while maintaining fast export and import operations. Understanding how the artifact structures its compressed payload, metadata, and Git integration is essential for teams sharing codebases across repositories.

Anatomy of the Artifact Format

Each artifact consists of three distinct components that work together to ensure portability and safety.

The Compressed Payload (*.zst)

The core of the artifact is a Zstandard-compressed snapshot of the SQLite database. In src/pipeline/artifact.c, the prepare_snapshot_db() function (lines 443-475) creates this snapshot using VACUUM INTO to generate a clean, compact database copy. For "best" quality exports, the system drops user-created indexes to improve compressibility before compression begins.

JSON Metadata (artifact.json)

Alongside the compressed file sits artifact.json, generated by write_metadata() (lines 31-72 in artifact.c). This JSON document stores critical context including the schema version, original uncompressed size, compression level, Git HEAD hash, timestamps, and node/edge counts. This metadata enables version validation and integrity checks during import.

Git Configuration (.gitattributes)

The ensure_gitattributes() function (lines 76-100 in artifact.c) generates a .gitattributes entry marking the artifact as binary and forcing the merge=ours driver. This prevents Git from attempting to merge conflicting artifact versions, automatically keeping the local copy during merge operations.

Export Pipeline: Creating Artifacts

The export process follows a rigorous pipeline to ensure atomic, verifiable artifacts.

Database Snapshot Preparation

The export begins with prepare_snapshot_db() copying the live database to a temporary file. This isolation ensures the compression operates on a consistent point-in-time snapshot without locking the production database.

Compression Strategy and Levels

Compression is handled by the thin wrapper functions in internal/cbm/zstd_store.c. The cbm_zstd_compress_bound() function (lines 34-36) calculates the maximum compressed size using ZSTD_compressBound(), ensuring adequate buffer allocation. The actual compression occurs in cbm_zstd_compress() (lines 10-16), which supports two quality levels defined in artifact.c (lines 10-13): ART_ZSTD_FAST (level 3) for speed and ART_ZSTD_BEST (level 9) for maximum compression.

Atomic File Operations

Both the compressed payload and metadata are written using write_file_atomic() (lines 59-89 in artifact.c). This helper ensures that partially written files never appear on disk, preventing repository corruption if the export process is interrupted.

Import Pipeline: Restoring Artifacts

Importing artifacts prioritizes security through frame validation and bounded resource allocation.

Frame-Size Validation and Safety Bounds

Rather than trusting the original_size field in metadata, the importer uses cbm_zstd_frame_content_size() (lines 26-32 in zstd_store.c) to read the content-size directly from the zstd frame header. This value must be non-zero, must not exceed ART_MAX_DECOMPRESSED_BYTES (approximately 64 GB, defined at lines 23-24 in artifact.c), and must match the metadata record. This design prevents maliciously crafted archives from triggering memory exhaustion attacks.

Decompression and Integrity Verification

The cbm_zstd_decompress() function (lines 18-23 in zstd_store.c) performs the actual decompression into a precisely allocated buffer sized to the validated frame content. After decompression, cbm_store_check_integrity_deep() (lines 704-708 in artifact.c) executes a full SQLite integrity check, rejecting any corrupted artifacts before they enter the cache.

Security and Safety Design

The artifact format implements multiple defense layers. Deterministic sizing via frame headers eliminates reliance on attacker-controlled metadata fields. Bounded resources through ART_MAX_DECOMPRESSED_BYTES caps memory allocation. Atomic operations guarantee transactional file writes, while the Git attributes configuration prevents merge conflicts that could corrupt the artifact.

Code Examples

Exporting an Artifact (Fast)

// Export a DB snapshot with fast (zstd‑3) compression
int rc = cbm_artifact_export(
    "/path/to/db.sqlite",          // db_path
    "/path/to/repo",               // repo_path
    "my-project",                  // project_name
    CBM_ARTIFACT_FAST);            // quality flag (fast)

if (rc != 0) {
    fprintf(stderr, "Export failed: %s\n",
            cbm_artifact_export_last_error());
}

Importing an Artifact

// Import the artifact into the cache DB
int rc = cbm_artifact_import(
    "/path/to/repo",          // repo_path containing *.zst + artifact.json
    "/path/to/cache.db");     // cache_db_path (final DB location)

if (rc != 0) {
    fprintf(stderr, "Import failed\n");
}

Direct Zstd Wrapper Usage

// Simple wrapper usage – compress a buffer
size_t bound = cbm_zstd_compress_bound(src_len);
char *dst = malloc(bound);
int compressed_len = cbm_zstd_compress(src, src_len, dst, bound, 9); // level 9

// Decompress back
size_t orig_size = cbm_zstd_frame_content_size(dst, compressed_len);
char *orig = malloc(orig_size);
int64_t decompressed_len = cbm_zstd_decompress(dst, compressed_len, orig, orig_size);

Summary

• The artifact format combines a Zstandard-compressed payload (*.zst), JSON metadata, and Git attributes configuration for safe team sharing.
• Export uses prepare_snapshot_db() and cbm_zstd_compress() with configurable levels (3 or 9), writing atomically via write_file_atomic().
• Import validates frame sizes using cbm_zstd_frame_content_size() against ART_MAX_DECOMPRESSED_BYTES (64 GB limit) before decompression.
• Integrity verification runs cbm_store_check_integrity_deep() to ensure the SQLite database is corruption-free.
• The .gitattributes entry with merge=ours prevents Git merge conflicts on binary artifacts.

Frequently Asked Questions

What compression levels does Codebase-Memory-MCP support?

The system supports two compression levels defined in src/pipeline/artifact.c: ART_ZSTD_FAST (level 3) for quick exports and ART_ZSTD_BEST (level 9) for maximum space savings. The "best" mode also drops user-created indexes before compression to improve the compression ratio.

How does the importer protect against maliciously crafted artifacts?

Instead of trusting the metadata's original_size field, the importer reads the content size directly from the zstd frame header using cbm_zstd_frame_content_size(). It validates this against ART_MAX_DECOMPRESSED_BYTES (approximately 64 GB) and the metadata record, preventing resource exhaustion attacks that would otherwise allocate excessive memory.

Why does the artifact include a .gitattributes entry?

The ensure_gitattributes() function generates an entry marking the artifact as binary with merge=ours, which forces Git to always keep the local version during merges. This prevents accidental merge conflicts that could corrupt the compressed database or create invalid hybrid artifacts.

Are file operations atomic to prevent corruption?

Yes. Both the compressed payload and JSON metadata are written using write_file_atomic() (lines 59-89 in artifact.c), which ensures that files appear on disk only when fully written. This guarantees that partial writes or interrupted exports never leave the repository in an inconsistent state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}