How Karukan Automatically Downloads and Caches HuggingFace Models on First Run

When Karukan starts its neural kana-kanji conversion engine for the first time, it checks the local XDG cache directory and automatically downloads the required GGUF model from HuggingFace if missing, caching it for subsequent runs.

The togatoga/karukan repository implements an intelligent input method engine that relies on quantized neural networks. When the neural conversion engine initializes for the first time, Karukan automatically downloads and caches HuggingFace models to enable local inference without manual setup.

How the Download Workflow Works

The automatic download process spans three core modules within the karukan-engine and karukan-im crates.

Step 1: Model Registry Parsing

In karukan-engine/src/kanji/model_config.rs, the engine parses the models.toml registry file to determine which quantized model to load. The load() function reads the configuration and returns the default model identifier, including the repository name and quantization level.

Step 2: Cache Validation and Streaming Download

The download_model function in karukan-engine/src/kanji/hf_download.rs manages the actual retrieval. It first checks $XDG_CACHE_HOME/karukan/models (falling back to ~/.cache/karukan/models) for an existing file. If the model is absent, it constructs a HuggingFace URL in the format https://huggingface.co/<repo>/resolve/main/<filename>.gguf and streams the file using reqwest.

The implementation writes to a temporary .part file and performs an atomic rename upon completion to prevent corruption. It also writes a JSON manifest containing the etag and file size for future validation.

Step 3: Engine Initialization

During startup in karukan-im/src/init.rs, the Engine::new constructor calls hf_download::download_model to ensure the model is present before loading it into the candle inference backend.

Cache Safety and Idempotency

Karukan implements several safeguards to ensure reliable caching:

• XDG Compliance: The cache respects the XDG_CACHE_HOME environment variable, defaulting to ~/.cache/karukan/models on standard Linux systems.
• Manifest Validation: Before downloading, the system checks a JSON manifest for etag and size matches, skipping the download if the file is current.
• Atomic Writes: Downloads use temporary .part files that are atomically renamed to the final .gguf name, preventing partial files from being loaded by concurrent processes.

Implementation Example

The following excerpts demonstrate how the components interact to download and cache models:

// In karukan-im/src/init.rs – engine start‑up
use karukan_engine::kanji::{hf_download, model_config};

pub async fn load_engine() -> Result<Engine, EngineError> {
    // 1️⃣ Read the model registry and pick the model to load
    let cfg = model_config::load()?;               // parses models.toml
    let model_id = cfg.default_model();            // e.g. "karukan/gguf-q5_k_m"

    // 2️⃣ Ensure the model file is present locally
    let model_path = hf_download::download_model(&model_id).await?;

    // 3️⃣ Open the model with Candle (the inference backend)
    let backend = candle::load_gguf(&model_path)?;
    Ok(Engine::new(backend))
}

// In karukan-engine/src/kanji/hf_download.rs – the download helper
pub async fn download_model(model_id: &str) -> Result<PathBuf, DownloadError> {
    let cache_dir = cache_dir()?;                     // XDG cache location
    let file_name = format!("{}.gguf", model_id.replace('/', "_"));
    let cached_path = cache_dir.join(&file_name);

    // Fast check – does the file already exist and match the manifest?
    if cached_path.exists() && manifest_is_valid(&cached_path).await? {
        return Ok(cached_path);
    }

    // Build the Hugging Face URL
    let url = format!(
        "https://huggingface.co/{}/resolve/main/{}.gguf",
        model_id, model_id.split('/').last().unwrap()
    );

    // Stream download into a temporary file
    let tmp_path = cache_dir.join(format!("{}.part", file_name));
    let mut resp = reqwest::get(&url).await?.error_for_status()?;
    let mut out = tokio::fs::File::create(&tmp_path).await?;
    while let Some(chunk) = resp.chunk().await? {
        out.write_all(&chunk).await?;
    }

    // Atomically replace the placeholder with the final file
    tokio::fs::rename(&tmp_path, &cached_path).await?;

    // Write a manifest (etag, size) for future fast‑path checks
    write_manifest(&cached_path, resp.headers()).await?;

    Ok(cached_path)
}

Summary

• Automatic Detection: On first run, karukan-engine detects missing models via model_config.rs and triggers a download.
• Standard Caching: Models are stored in the XDG cache directory (~/.cache/karukan/models) with manifest files for validation.
• Safe Concurrency: Atomic file operations and temporary .part files prevent corruption during parallel downloads.
• Transparent Integration: The download_model function in hf_download.rs is called during engine initialization in init.rs, making the process invisible to users.

Frequently Asked Questions

Where does Karukan store downloaded HuggingFace models?

Karukan stores models in the XDG cache directory, specifically at $XDG_CACHE_HOME/karukan/models or falling back to ~/.cache/karukan/models if the environment variable is not set. Each model is saved as a .gguf file alongside a JSON manifest containing etag and size metadata.

How does Karukan avoid re-downloading models on every startup?

The download_model function in karukan-engine/src/kanji/hf_download.rs checks for file existence and validates a stored manifest (etag and size) against remote headers. If the cached file matches the remote metadata, the download is skipped and the local path is returned immediately.

What happens if the download is interrupted?

The implementation uses temporary .part files during the streaming download process. Only after the download completes successfully does it perform an atomic rename to the final .gguf filename. This ensures that partial or corrupted files are never used by the inference engine.

Can I use a custom model or mirror instead of HuggingFace?

The current implementation in hf_download.rs constructs URLs specifically for HuggingFace repositories using the pattern https://huggingface.co/{repo}/resolve/main/{file}.gguf. While the codebase supports fallback models defined in models.toml, changing the source URL would require modifying the source code in karukan-engine/src/kanji/hf_download.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}