How the Whisper Engine in Meetily Loads and Manages Transcription Models with GPU Acceleration (Metal/CUDA/Vulkan)

Meetily’s Whisper engine uses a dual-layer detection strategy—compile-time Cargo features select the GPU backend (Metal, CUDA, Vulkan, or HIPBLAS), while runtime hardware probing determines device capabilities and performance tiers, automatically configuring whisper_rs with GPU acceleration parameters when loading transcription models.

The Meetily transcription pipeline, built on the whisper_rs crate, dynamically adapts to heterogeneous hardware by combining static feature flags with runtime GPU detection. Whether running on macOS with Metal, Linux with CUDA, or cross-platform Vulkan, the engine automatically selects optimal acceleration parameters, including flash attention support for compatible high-performance GPUs.

Compile-Time Backend Selection via Cargo Features

The engine first determines which GPU backend was compiled into the binary through the WhisperCompiledBackend enum in frontend/src-tauri/src/whisper_engine/acceleration.rs. The current() method inspects Cargo feature flags to identify the available acceleration target:

// frontend/src-tauri/src/whisper_engine/acceleration.rs
impl WhisperCompiledBackend {
    pub fn current() -> Self {
        if cfg!(feature = "cuda")   { Self::Cuda   }
        else if cfg!(feature = "vulkan") { Self::Vulkan }
        else if cfg!(feature = "hipblas") { Self::HipBlas }
        else if cfg!(target_os = "macos") || cfg!(feature = "metal") {
            Self::Metal
        } else { Self::Cpu }
    }
}

This compile-time check prioritizes CUDA and Vulkan features explicitly, falls back to Metal on macOS or when the metal feature is enabled, and defaults to CPU if no GPU features are active. The selected backend remains fixed for the application lifecycle.

Runtime GPU Capability Detection

After establishing the compiled backend, the engine probes the actual hardware capabilities through HardwareProfile::detect() in frontend/src-tauri/src/audio/hardware_detector.rs. This function returns a GpuType enum (Metal, Cuda, Vulkan, or None) alongside a PerformanceTier classification (Low, Medium, High, or Ultra).

The WhisperEngine::detect_gpu_acceleration() method logs the detection results:

fn detect_gpu_acceleration() -> bool {
    match WhisperCompiledBackend::current() {
        WhisperCompiledBackend::Metal => { 
            log::info!("macOS detected – enabling Metal"); 
            true 
        },
        WhisperCompiledBackend::Cuda  => { 
            log::info!("CUDA feature enabled – enabling GPU"); 
            true 
        },
        WhisperCompiledBackend::Vulkan => { 
            log::info!("Vulkan feature enabled – enabling GPU"); 
            true 
        },
        _ => { 
            log::info!("No GPU acceleration – using CPU"); 
            false 
        },
    }
}

This runtime validation ensures the engine only attempts GPU initialization when both the compiled feature and physical hardware are present.

Configuring Acceleration Parameters

The whisper_context_acceleration_for() function bridges compile-time flags and runtime detection to produce a WhisperContextAcceleration struct. Located in frontend/src-tauri/src/whisper_engine/acceleration.rs, this logic determines whether to enable GPU inference and flash attention:

// frontend/src-tauri/src/whisper_engine/acceleration.rs
pub fn whisper_context_acceleration_for(
    compiled_backend: WhisperCompiledBackend,
    runtime_detected_gpu: GpuType,
    performance_tier: PerformanceTier,
) -> WhisperContextAcceleration {
    let use_gpu = !matches!(compiled_backend, WhisperCompiledBackend::Cpu);
    let fast_tier = matches!(performance_tier, PerformanceTier::High | PerformanceTier::Ultra);
    let flash_attn = matches!(compiled_backend, WhisperCompiledBackend::Metal | WhisperCompiledBackend::Cuda) 
        && fast_tier;
    
    WhisperContextAcceleration { 
        compiled_backend, 
        runtime_detected_gpu, 
        use_gpu, 
        flash_attn, 
        gpu_device: 0 
    }
}

Flash attention—a memory-efficient attention kernel—activates only when using Metal or CUDA backends on High or Ultra tier hardware. The function defaults to device index 0 for multi-GPU systems.

Loading Models with GPU Context

When WhisperEngine::load_model() in frontend/src-tauri/src/whisper_engine/whisper_engine.rs initializes a transcription model, it orchestrates the hardware detection and parameter construction:

let hardware_profile = crate::audio::HardwareProfile::detect();
let adaptive_config = hardware_profile.get_whisper_config();
let acceleration = whisper_context_acceleration_for(
    WhisperCompiledBackend::current(),
    hardware_profile.gpu_type,
    hardware_profile.performance_tier,
);

let context_param = WhisperContextParameters {
    use_gpu: acceleration.use_gpu,
    gpu_device: acceleration.gpu_device,
    flash_attn: acceleration.flash_attn,
    ..Default::default()
};

The engine then instantiates the whisper context with these parameters:

log::info!(
    "Whisper acceleration decision: compiled_backend={} runtime_detected_gpu={:?} use_gpu={} flash_attn={} gpu_device={}",
    acceleration.compiled_backend.as_str(),
    acceleration.runtime_detected_gpu,
    acceleration.use_gpu,
    acceleration.flash_attn,
    acceleration.gpu_device,
);

// Load the model with GPU parameters
let context = WhisperContext::new_with_params(&model_path, context_param)?;

If use_gpu evaluates to false—either because the compiled backend is Cpu or runtime detection returns GpuType::None—the model runs entirely on CPU threads without GPU memory allocation.

Cross-Platform Build Configuration

Enable specific GPU backends by modifying the features array in your Cargo.toml. The engine supports conditional compilation for different operating systems:

# Cargo.toml (partial)

[features]
default = ["metal"]        # macOS default

cuda = []                  # Enable NVIDIA CUDA support

vulkan = []                # Enable Vulkan support for AMD/Intel GPUs

hipblas = []               # Enable AMD ROCm/HIP support

Compile with feature flags to target specific hardware:

# macOS with Metal (default)

cargo build

# Linux with NVIDIA CUDA

cargo build --features cuda

# Cross-platform Vulkan

cargo build --features vulkan

The model files (.ggml or .bin binaries listed in WHISPER_MODEL_CATALOG from frontend/src-tauri/src/config.rs) remain identical across all backends; only the acceleration layer changes based on compilation flags.

Summary

• Compile-time detection uses cfg! macros in acceleration.rs to identify enabled GPU features (Metal, CUDA, Vulkan, HIPBLAS).
• Runtime hardware profiling through HardwareProfile::detect() provides actual GPU availability and performance tiers.
• Acceleration resolution via whisper_context_acceleration_for() enables GPU inference and conditional flash attention for Metal/CUDA on High/Ultra tier devices.
• Model loading constructs WhisperContextParameters with use_gpu, gpu_device, and flash_attn flags before calling WhisperContext::new_with_params().
• Automatic fallback to CPU occurs when compiled features are absent or hardware detection fails.

Frequently Asked Questions

How does Meetily choose between Metal, CUDA, and Vulkan?

Meetily selects the backend at compile time through Cargo feature flags checked by WhisperCompiledBackend::current(). If multiple features are enabled, the code prioritizes CUDA, then Vulkan, then HIPBLAS, then Metal. At runtime, the engine verifies that the compiled backend matches available hardware through HardwareProfile::detect(), ensuring the engine never attempts to initialize Metal on a Linux system or CUDA on a Mac.

What triggers flash attention activation in the Whisper engine?

Flash attention activates only when two conditions are met: the compiled backend must be Metal or Cuda (enforced via pattern matching), and the runtime performance tier must be High or Ultra as determined by the hardware detector. This optimization, configured in whisper_context_acceleration_for(), reduces memory bandwidth during attention calculations but requires modern GPU architectures.

Can I force CPU-only mode even if a GPU is detected?

Yes, by compiling without GPU features or ensuring the WhisperCompiledBackend::current() returns Cpu. The engine treats WhisperCompiledBackend::Cpu as a hard override—when this variant is active, use_gpu is set to false regardless of what HardwareProfile::detect() reports. Alternatively, you can modify the build configuration to exclude all GPU features (cargo build with no GPU features enabled).

Where does Meetily store downloaded transcription models?

Models are stored in the configured models_dir path (typically a subdirectory of the application data directory) as defined in frontend/src-tauri/src/config.rs. The WHISPER_MODEL_CATALOG static map defines the supported model names (e.g., "base", "small", "large-v3") and their corresponding filenames. The same binary files work across all GPU backends; no separate model downloads are required for Metal versus CUDA inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}