How the Meetily Summary Engine Orchestrates LLM Requests for Meeting Summaries

The Meetily summary engine orchestrates LLM requests for meeting summaries through a Rust-based sidecar process that runs local GGUF models, communicating via JSON-encoded messages with integrated health monitoring, timeout protection, and cancellation support.

Meetily is an open-source meeting assistant that transforms conversation transcripts into concise AI-generated summaries. The summary engine orchestrates LLM requests for meeting summaries using a sophisticated pipeline that manages local model execution through a dedicated sidecar process, eliminating external API dependencies while maintaining complete data privacy.

Architecture Overview

The orchestration layer resides in the Tauri-based Rust backend, specifically within frontend/src-tauri/src/summary/summary_engine/. At its core, the system uses a SidecarManager to spawn and monitor a separate binary process that loads GGUF format models (such as Gemma 3). This separation isolates the LLM runtime from the main application process, ensuring that model crashes or resource exhaustion do not affect the core Meetily application.

The public entry point generate_with_builtin in client.rs encapsulates the entire workflow, from model resolution to response parsing, providing a single async interface for the rest of the application.

Step-by-Step Orchestration Workflow

Initializing the Sidecar Manager

Before any generation can occur, the engine initializes the sidecar infrastructure. The init_sidecar_manager(app_data_dir) function creates a SidecarManager instance (defined in sidecar.rs) and stores it in a lazy-global Arc<Mutex<Option<Arc<SidecarManager>>>>. This singleton pattern ensures that only one sidecar process exists throughout the application lifecycle, managing binary startup, process monitoring, and shutdown coordination.

Caching and Resolving Model Paths

To avoid repeated filesystem lookups, the engine implements a Lazy initialized RwLock<HashMap> called MODEL_PATH_CACHE. When generate_with_builtin receives a model identifier (e.g., "gemma3:1b"), it calls get_cached_model_path to either retrieve a cached absolute path or resolve it via models::get_model_path from models.rs. This caching layer significantly reduces latency for subsequent summary requests using the same model.

Formatting Prompts with Chat Templates

Raw prompts undergo formatting through models::format_prompt before transmission. This function applies model-specific chat templates, combining system instructions (e.g., "You are a concise meeting summarizer") with user content (the meeting transcript) into a single properly formatted string that respects the target model's expected input structure.

Constructing the Generation Request

The engine constructs a strongly-typed Request::Generate enum variant defined in client.rs. This request object includes:

• The formatted prompt string
• Model-specific sampling parameters (temperature, top-k, top-p) from model_def.sampling
• Context limits and maximum token counts
• The absolute filesystem path to the GGUF model file

The request serializes to JSON using serde_json::to_string before transmission to the sidecar process.

Executing Requests with Health Checks

Before sending data, manager.ensure_running(model_path) verifies that the sidecar process is alive and loaded with the requested model. If the process is down or loaded with a different model, the manager handles transparent restarts. The manager.is_healthy() method provides runtime status probing to detect zombie processes or initialization failures.

Handling Timeouts and Cancellation

The SidecarManager::send_request(request_json, timeout) method transmits the JSON payload with configurable timeout protection defined by models::GENERATION_TIMEOUT_SECS. For user-initiated aborts, the engine accepts an optional tokio_util::sync::CancellationToken. If cancellation occurs, the system immediately triggers force_shutdown_sidecar to terminate the process, preventing resource waste on discarded requests.

Parsing Responses and Error Handling

The sidecar returns JSON responses that deserialize into the Response enum with variants Response::Response { text, error } or Response::Error { message }. The orchestration layer checks for error fields, returning anyhow errors for failures while passing successful generated text back to the calling service in service.rs.

Graceful Shutdown Procedures

The engine provides shutdown_sidecar_gracefully for clean termination during application exit, allowing in-flight requests to complete before process cleanup. The force_shutdown_sidecar variant immediately kills the process, used primarily during cancellation scenarios or emergency cleanup.

Implementation Example

The following example demonstrates the complete orchestration flow from initialization to summary generation:

use meetily::summary::summary_engine::client::{
    init_sidecar_manager, generate_with_builtin, shutdown_sidecar_gracefully,
};
use tokio_util::sync::CancellationToken;
use std::path::PathBuf;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the sidecar manager (once per app run)
    let data_dir = PathBuf::from("/Users/me/Library/Application Support/Meetily");
    init_sidecar_manager(data_dir.clone()).await?;

    // Create a cancellation token for user aborts
    let cancel_token = CancellationToken::new();

    // Generate meeting summary
    let summary = generate_with_builtin(
        &data_dir,
        "gemma3:1b",                              // Model identifier
        "You are a concise meeting summarizer.",  // System prompt
        "Discuss project roadmap and next sprint.", // User prompt (transcript)
        Some(&cancel_token),                      // Optional cancellation
    )
    .await?;

    println!("Summary:\n{}", summary);

    // Clean shutdown when application exits
    shutdown_sidecar_gracefully().await?;
    Ok(())
}

Cancelling In-Flight Requests

To abort a generation programmatically:

let token = CancellationToken::new();
let handle = tokio::spawn(async move {
    generate_with_builtin(&data_dir, "gemma3:1b", "...", "...", Some(&token)).await
});

// Trigger cancellation from UI or timeout logic
token.cancel();
let result = handle.await?;

Key Source Files

The orchestration pipeline spans several Rust modules in the Meetily codebase:

• frontend/src-tauri/src/summary/summary_engine/sidecar.rs — Implements SidecarManager for process lifecycle management and inter-process communication.
• frontend/src-tauri/src/summary/summary_engine/models.rs — Contains model metadata definitions, path resolution logic, and prompt formatting utilities.
• frontend/src-tauri/src/summary/summary_engine/client.rs — Provides the public generate_with_builtin API, request construction, timeout handling, and response parsing.
• frontend/src-tauri/src/summary/service.rs — High-level coordinator that bridges UI commands to the summary engine.
• frontend/src-tauri/src/summary/llm_client.rs — Fallback implementation for remote LLM providers (Ollama, Groq) when local inference is unavailable.

Summary

• The Meetily summary engine uses a sidecar process architecture to isolate local LLM execution from the main application, implemented in Rust within the Tauri backend.
• Model path caching via MODEL_PATH_CACHE eliminates redundant filesystem operations across multiple summary requests.
• Requests are serialized to JSON using the Request::Generate enum and transmitted with configurable timeouts and cancellation token support.
• The SidecarManager handles process health checks, automatic restarts, and graceful or forced shutdown scenarios.
• All orchestration logic is encapsulated in generate_with_builtin in client.rs, providing a single async interface for meeting summary generation.

Frequently Asked Questions

What is the sidecar pattern used in Meetily's summary engine?

The sidecar pattern refers to running the LLM inference engine as a separate binary process that communicates with the main Meetily application via JSON messages over stdin/stdout. As implemented in sidecar.rs, this isolation prevents model crashes or memory leaks from affecting the core application stability while enabling the use of optimized GGUF model runners.

How does Meetily handle model loading and caching?

The engine maintains a thread-safe RwLock<HashMap> called MODEL_PATH_CACHE that stores resolved absolute paths to GGUF model files. When generate_with_builtin is called, get_cached_model_path checks this cache before calling models::get_model_path, reducing filesystem I/O and improving response times for repeated summary requests with the same model.

Can users cancel an ongoing summary generation request?

Yes, the orchestration layer supports cancellation through tokio_util::sync::CancellationToken. When passed to generate_with_builtin, the token allows immediate abortion of in-flight requests. Upon cancellation, the engine triggers force_shutdown_sidecar to terminate the LLM process instantly, freeing system resources.

What happens if the sidecar process crashes during generation?

The SidecarManager implements health monitoring via is_healthy() checks and ensure_running() guards. If a crash is detected, the manager automatically attempts to restart the sidecar process and reload the requested model. Failed health checks propagate as anyhow errors to the caller, allowing the UI to display appropriate error messages or fall back to remote LLM providers via llm_client.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}