How to Enable and Configure Parallel Processing in ImageKit

ImageKit automatically enables parallel processing for all images using the Rayon data-parallelism library, utilizing all logical CPU cores by default, and you can configure the degree of parallelism via the RAYON_NUM_THREADS environment variable or by modifying the ThreadPoolBuilder in the source code.

ImageKit is a Rust-based CLI tool designed for high-performance batch image processing. According to the hzbd/imagekit source code, the tool processes images concurrently using Rayon’s global thread pool, which automatically scales to match your system’s logical core count. Understanding how to configure this parallel processing behavior allows you to optimize performance on shared servers or resource-constrained environments.

How Parallel Processing Works in ImageKit

In the core orchestration logic located in src/lib.rs, the run function converts the list of input image paths into a parallel iterator. The implementation uses par_iter() from the Rayon crate to distribute work across multiple threads:

// src/lib.rs – parallel loop
image_paths.par_iter().for_each(move |path| {
    let fonts_clone = Arc::clone(&fonts);
    if let Err(e) = process_image(path, &cli, &fonts_clone) {
        eprintln!("Failed to process {}: {}", path.display(), e);
    }
});

Each image path is processed concurrently by the process_image function defined in src/processor.rs (starting at line 10). The Rayon dependency is declared in Cargo.toml at line 11 as rayon = "1.8", providing the data-parallelism runtime that automatically creates a global thread pool sized to the number of logical CPUs.

Configuring Parallelism in ImageKit

While the default configuration maximizes throughput, you may need to limit CPU usage. ImageKit supports two approaches to configure the Rayon thread pool.

Using the RAYON_NUM_THREADS Environment Variable

The simplest method requires no code changes. Set the RAYON_NUM_THREADS environment variable before running the binary to cap the global thread pool size:

# Limit processing to 4 threads

RAYON_NUM_THREADS=4 imagekit -i ./photos -o ./out

Rayon reads this variable at startup and adjusts the global pool accordingly. This approach is ideal for temporary adjustments or deployment scripts.

Embedding a Custom ThreadPoolBuilder

For permanent configuration baked into the binary, modify src/lib.rs to create a dedicated thread pool using ThreadPoolBuilder. Wrap the existing parallel loop inside a pool.install() block:

use rayon::ThreadPoolBuilder;

// ...

let pool = ThreadPoolBuilder::new()
    .num_threads(4)          // set desired parallelism
    .build()
    .expect("Failed to build Rayon thread pool");

// Execute the parallel loop inside the custom pool
pool.install(|| {
    image_paths.par_iter().for_each(move |path| {
        let fonts_clone = Arc::clone(&fonts);
        if let Err(e) = process_image(path, &cli, &fonts_clone) {
            eprintln!("Failed to process {}: {}", path.display(), e);
        }
    });
});

After recompiling with cargo build --release, ImageKit will always use the specified thread count regardless of environment variables.

Summary

• ImageKit processes images concurrently using Rayon’s par_iter() in src/lib.rs (lines 64-72).
• By default, parallelism equals the number of logical CPU cores.
• Use RAYON_NUM_THREADS for quick, per-run configuration without recompiling.
• Modify src/lib.rs to use ThreadPoolBuilder and pool.install() for permanent thread pool constraints.
• The actual image processing logic resides in src/processor.rs, invoked by the parallel loop.

Frequently Asked Questions

Does ImageKit support parallel processing by default?

Yes. According to the hzbd/imagekit source code, parallel processing is automatically enabled through the Rayon library. The run function in src/lib.rs uses par_iter() to distribute images across all available logical CPU cores without requiring any configuration.

How can I limit CPU usage when running ImageKit?

Set the RAYON_NUM_THREADS environment variable to the desired number of threads before executing the binary. For example, RAYON_NUM_THREADS=2 imagekit -i ./input -o ./output restricts processing to two threads, leaving CPU resources available for other applications.

Can I configure the thread pool programmatically in ImageKit?

Yes. To hardcode a specific thread count, modify src/lib.rs to construct a ThreadPoolBuilder, specify num_threads(), and wrap the par_iter() loop inside pool.install(|| { ... }). This requires recompiling the project but ensures consistent behavior across different environments.

Where is the parallel processing logic implemented in the codebase?

The parallel orchestration occurs in src/lib.rs between lines 64-72, where image_paths.par_iter().for_each() dispatches work to worker threads. The per-image processing function process_image is defined in src/processor.rs starting at line 10, while the Rayon dependency is declared in Cargo.toml at line 11.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}