# How prepare-dspark-model-cache.sh Handles Checkpoint Distribution in DeepSeek-V4-Flash

> Learn how prepare-dspark-model-cache.sh distributes DeepSeek-V4-Flash checkpoints efficiently. Discover its NFS and SCP copy methods for optimal model deployment.

- Repository: [Mia's AI Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark)
- Tags: how-to-guide
- Published: 2026-09-09

---

**The [`prepare-dspark-model-cache.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/prepare-dspark-model-cache.sh) script downloads the DeepSeek-V4-Flash checkpoint once on the head node, verifies all `*.safetensors` shards, then distributes the model to DSPARK workers via either NFS shared storage or explicit SCP copy based on the `DSPARK_WORKER_HF_NFS` environment flag.**

The [`prepare-dspark-model-cache.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/prepare-dspark-model-cache.sh) script serves as the entry point for model preparation in the MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark repository. This Bash orchestrator manages the complete checkpoint distribution pipeline, ensuring that the 157 GiB model weights are available consistently across your cluster while supporting both gated "abliterated" variants and official releases.

## Understanding Checkpoint Resolution Logic

Before downloading begins, the script must determine which model variant to fetch. The `resolve_checkpoint()` function (lines 66-95 in [`prepare-dspark-model-cache.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/prepare-dspark-model-cache.sh)) evaluates CLI flags (`--official`, `--abliterated`) and the `ABLITERATED` value stored in `.env.dspark` to select the appropriate Hugging Face repository.

When `ABLITERATED=1` is set, the script handles two distinct download phases:

- **Primary checkpoint**: The 157 GiB base model remains on the public Hub
- **Gated artifacts**: The `run_gated_ablit_artifacts()` function (lines 31-48) fetches only the 18 KiB direction tensor and terms file from the private gated repository, avoiding redundant downloads of the full weights

## Head Node Download and Verification Pipeline

The script strictly enforces a **download-once, verify-then-distribute** pattern on the head node.

First, `run_download()` (lines 34-41) executes a Docker container running `huggingface_hub.snapshot_download` to pull the selected revision into the local `HF_CACHE` directory. The script supports custom revisions via the `DSPARK_REVISION` environment variable for pinning specific commit SHAs.

After download completion, `verify_cache()` (lines 84-89) performs integrity validation by checking for the presence of all required `*.safetensors` shards. If any shard is missing, the script aborts immediately with a non-zero exit code, preventing corrupted checkpoints from propagating to workers.

## Checkpoint Distribution Strategies

The script implements two distinct modes for making the checkpoint available to DSPARK workers, controlled by the `DSPARK_WORKER_HF_NFS` environment variable checked at lines 58-62.

### NFS-Mounted Shared Storage

When `DSPARK_WORKER_HF_NFS=1`, the script **skips remote copying** and keeps the checkpoint exclusively on the head node. Workers mount the same `HF_CACHE` directory via NFS, eliminating storage redundancy across the cluster. This mode requires that all worker nodes have the NFS mount configured at the same path as the head node's cache directory.

### Explicit SCP Copy to Workers

By default (`DSPARK_WORKER_HF_NFS=0`), the script pushes the checkpoint to each worker explicitly:

1. **Directory preparation**: Creates the target `HF_CACHE` path on the worker via SSH
2. **Image verification**: Runs `verify_worker_image` to ensure the worker has the required `vllm-dspark-runtime` Docker image
3. **Script propagation**: Uses `scp` to transfer [`prepare-dspark-model-cache.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/prepare-dspark-model-cache.sh) and `.env.dspark` to the worker
4. **Worker execution**: Invokes the script remotely with `PREPARE_WORKER=0` and `--yes` flags

Setting `PREPARE_WORKER=0` (lines 56-73) signals the script to **skip the interactive prompt and download phase**, forcing it to reuse the checkpoint already present in the worker's local `HF_CACHE` after the SCP transfer completes.

## Multi-Worker Support and Environment Propagation

For dual-DGX configurations, the script automatically detects `WORKER2_HOST` and repeats the distribution logic (lines 73-84), ensuring both GPU nodes receive identical checkpoint states.

Authentication tokens propagate securely via `WORKER_HF_TOKEN_ENV` (lines 19-25). The script forwards the `HF_TOKEN` to workers through Docker `-e` arguments, allowing workers to resolve any missing shards without manual re-authentication. This ensures uninterrupted model loading even if the initial SCP transfer encounters network interruptions requiring partial redownloads.

## Practical Usage Examples

```bash

# Download official checkpoint with interactive confirmation

./prepare-dspark-model-cache.sh

# Prepare gated "abliterated" variant non-interactively

./prepare-dspark-model-cache.sh --abliterated --yes

# Use NFS shared storage (workers mount head node cache)

export DSPARK_WORKER_HF_NFS=1
./prepare-dspark-model-cache.sh

# Distribute to specific revision across two workers

export DSPARK_REVISION=86f746b3
export WORKER2_HOST=worker2.dgx.local
./prepare-dspark-model-cache.sh --official

```

## Summary

- The [`prepare-dspark-model-cache.sh`](https://github.com/MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark/blob/main/prepare-dspark-model-cache.sh) script in MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark serves as the single entry point for model preparation across the cluster.
- **Checkpoint resolution** via `resolve_checkpoint()` supports both official and gated abliterated variants without downloading redundant data.
- **Integrity verification** through `verify_cache()` validates all `*.safetensors` shards before distribution begins.
- **Dual distribution modes**: Set `DSPARK_WORKER_HF_NFS=1` for NFS-mounted shared storage, or use the default SCP-based push to copy weights explicitly to each worker.
- **Worker idempotency**: The `PREPARE_WORKER=0` flag ensures workers skip downloads and reuse pre-staged checkpoints, preventing redundant 157 GiB transfers.

## Frequently Asked Questions

### What happens if a worker already has a partial checkpoint?

When `PREPARE_WORKER=0` is set during the remote execution phase, the worker runs `verify_cache()` to check for existing `*.safetensors` shards. If shards are missing or corrupted, the Hugging Face token forwarded via `WORKER_HF_TOKEN_ENV` allows the worker to download only the specific missing files rather than the full 157 GiB dataset.

### How does the script handle the gated "abliterated" model differently from the official release?

The `run_gated_ablit_artifacts()` function fetches only the small direction tensor (18 KiB) and terms file from the private repository when `ABLITERATED=1`, while pulling the base checkpoint from the public Hub. This design prevents unauthorized redistribution of gated content while keeping the large weights in the open domain.

### Can I use this script with a custom Hugging Face cache directory?

Yes. The script respects the `HF_CACHE` environment variable defined in `.env.dspark` for both the head node download and the worker target path. Ensure this directory is either NFS-mounted across nodes (when using `DSPARK_WORKER_HF_NFS=1`) or has sufficient disk space for the explicit copy mode (default).

### What is the purpose of the `DSPARK_REVISION` variable?

This environment variable pins the checkpoint to a specific Git commit SHA or tag, ensuring reproducible deployments. When set, `run_download()` passes this value to `huggingface_hub.snapshot_download` as the `revision` parameter, preventing automatic updates when new model versions are published to the Hub.