# How the Meetily Summary Engine Orchestrates LLM Requests for Meeting Summaries

> Discover how the Meetily summary engine orchestrates LLM requests for meeting summaries using a Rust sidecar, GGUF models, and robust communication features for efficient AI note-taking.

- Repository: [Zackriya Solutions/meetily](https://github.com/Zackriya-Solutions/meetily)
- Tags: internals
- Published: 2026-07-25

---

**The Meetily summary engine orchestrates LLM requests for meeting summaries through a Rust-based sidecar process that runs local GGUF models, communicating via JSON-encoded messages with integrated health monitoring, timeout protection, and cancellation support.**

Meetily is an open-source meeting assistant that transforms conversation transcripts into concise AI-generated summaries. The **summary engine orchestrates LLM requests for meeting summaries** using a sophisticated pipeline that manages local model execution through a dedicated sidecar process, eliminating external API dependencies while maintaining complete data privacy.

## Architecture Overview

The orchestration layer resides in the Tauri-based Rust backend, specifically within `frontend/src-tauri/src/summary/summary_engine/`. At its core, the system uses a `SidecarManager` to spawn and monitor a separate binary process that loads GGUF format models (such as Gemma 3). This separation isolates the LLM runtime from the main application process, ensuring that model crashes or resource exhaustion do not affect the core Meetily application.

The public entry point `generate_with_builtin` in [`client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/client.rs) encapsulates the entire workflow, from model resolution to response parsing, providing a single async interface for the rest of the application.

## Step-by-Step Orchestration Workflow

### Initializing the Sidecar Manager

Before any generation can occur, the engine initializes the sidecar infrastructure. The `init_sidecar_manager(app_data_dir)` function creates a `SidecarManager` instance (defined in [`sidecar.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/sidecar.rs)) and stores it in a lazy-global `Arc<Mutex<Option<Arc<SidecarManager>>>>`. This singleton pattern ensures that only one sidecar process exists throughout the application lifecycle, managing binary startup, process monitoring, and shutdown coordination.

### Caching and Resolving Model Paths

To avoid repeated filesystem lookups, the engine implements a `Lazy` initialized `RwLock<HashMap>` called `MODEL_PATH_CACHE`. When `generate_with_builtin` receives a model identifier (e.g., `"gemma3:1b"`), it calls `get_cached_model_path` to either retrieve a cached absolute path or resolve it via `models::get_model_path` from [`models.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/models.rs). This caching layer significantly reduces latency for subsequent summary requests using the same model.

### Formatting Prompts with Chat Templates

Raw prompts undergo formatting through `models::format_prompt` before transmission. This function applies model-specific chat templates, combining system instructions (e.g., "You are a concise meeting summarizer") with user content (the meeting transcript) into a single properly formatted string that respects the target model's expected input structure.

### Constructing the Generation Request

The engine constructs a strongly-typed `Request::Generate` enum variant defined in [`client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/client.rs). This request object includes:

- The formatted prompt string
- Model-specific sampling parameters (temperature, top-k, top-p) from `model_def.sampling`
- Context limits and maximum token counts
- The absolute filesystem path to the GGUF model file

The request serializes to JSON using `serde_json::to_string` before transmission to the sidecar process.

### Executing Requests with Health Checks

Before sending data, `manager.ensure_running(model_path)` verifies that the sidecar process is alive and loaded with the requested model. If the process is down or loaded with a different model, the manager handles transparent restarts. The `manager.is_healthy()` method provides runtime status probing to detect zombie processes or initialization failures.

### Handling Timeouts and Cancellation

The `SidecarManager::send_request(request_json, timeout)` method transmits the JSON payload with configurable timeout protection defined by `models::GENERATION_TIMEOUT_SECS`. For user-initiated aborts, the engine accepts an optional `tokio_util::sync::CancellationToken`. If cancellation occurs, the system immediately triggers `force_shutdown_sidecar` to terminate the process, preventing resource waste on discarded requests.

### Parsing Responses and Error Handling

The sidecar returns JSON responses that deserialize into the `Response` enum with variants `Response::Response { text, error }` or `Response::Error { message }`. The orchestration layer checks for error fields, returning `anyhow` errors for failures while passing successful generated text back to the calling service in [`service.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/service.rs).

### Graceful Shutdown Procedures

The engine provides `shutdown_sidecar_gracefully` for clean termination during application exit, allowing in-flight requests to complete before process cleanup. The `force_shutdown_sidecar` variant immediately kills the process, used primarily during cancellation scenarios or emergency cleanup.

## Implementation Example

The following example demonstrates the complete orchestration flow from initialization to summary generation:

```rust
use meetily::summary::summary_engine::client::{
    init_sidecar_manager, generate_with_builtin, shutdown_sidecar_gracefully,
};
use tokio_util::sync::CancellationToken;
use std::path::PathBuf;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize the sidecar manager (once per app run)
    let data_dir = PathBuf::from("/Users/me/Library/Application Support/Meetily");
    init_sidecar_manager(data_dir.clone()).await?;

    // Create a cancellation token for user aborts
    let cancel_token = CancellationToken::new();

    // Generate meeting summary
    let summary = generate_with_builtin(
        &data_dir,
        "gemma3:1b",                              // Model identifier
        "You are a concise meeting summarizer.",  // System prompt
        "Discuss project roadmap and next sprint.", // User prompt (transcript)
        Some(&cancel_token),                      // Optional cancellation
    )
    .await?;

    println!("Summary:\n{}", summary);

    // Clean shutdown when application exits
    shutdown_sidecar_gracefully().await?;
    Ok(())
}

```

### Cancelling In-Flight Requests

To abort a generation programmatically:

```rust
let token = CancellationToken::new();
let handle = tokio::spawn(async move {
    generate_with_builtin(&data_dir, "gemma3:1b", "...", "...", Some(&token)).await
});

// Trigger cancellation from UI or timeout logic
token.cancel();
let result = handle.await?;

```

## Key Source Files

The orchestration pipeline spans several Rust modules in the Meetily codebase:

- **[`frontend/src-tauri/src/summary/summary_engine/sidecar.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/summary/summary_engine/sidecar.rs)** — Implements `SidecarManager` for process lifecycle management and inter-process communication.
- **[`frontend/src-tauri/src/summary/summary_engine/models.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/summary/summary_engine/models.rs)** — Contains model metadata definitions, path resolution logic, and prompt formatting utilities.
- **[`frontend/src-tauri/src/summary/summary_engine/client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/summary/summary_engine/client.rs)** — Provides the public `generate_with_builtin` API, request construction, timeout handling, and response parsing.
- **[`frontend/src-tauri/src/summary/service.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/summary/service.rs)** — High-level coordinator that bridges UI commands to the summary engine.
- **[`frontend/src-tauri/src/summary/llm_client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/frontend/src-tauri/src/summary/llm_client.rs)** — Fallback implementation for remote LLM providers (Ollama, Groq) when local inference is unavailable.

## Summary

- The Meetily summary engine uses a **sidecar process architecture** to isolate local LLM execution from the main application, implemented in Rust within the Tauri backend.
- **Model path caching** via `MODEL_PATH_CACHE` eliminates redundant filesystem operations across multiple summary requests.
- Requests are **serialized to JSON** using the `Request::Generate` enum and transmitted with configurable timeouts and **cancellation token support**.
- The `SidecarManager` handles process health checks, automatic restarts, and **graceful or forced shutdown** scenarios.
- All orchestration logic is encapsulated in `generate_with_builtin` in [`client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/client.rs), providing a single async interface for meeting summary generation.

## Frequently Asked Questions

### What is the sidecar pattern used in Meetily's summary engine?

The sidecar pattern refers to running the LLM inference engine as a separate binary process that communicates with the main Meetily application via JSON messages over stdin/stdout. As implemented in [`sidecar.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/sidecar.rs), this isolation prevents model crashes or memory leaks from affecting the core application stability while enabling the use of optimized GGUF model runners.

### How does Meetily handle model loading and caching?

The engine maintains a thread-safe `RwLock<HashMap>` called `MODEL_PATH_CACHE` that stores resolved absolute paths to GGUF model files. When `generate_with_builtin` is called, `get_cached_model_path` checks this cache before calling `models::get_model_path`, reducing filesystem I/O and improving response times for repeated summary requests with the same model.

### Can users cancel an ongoing summary generation request?

Yes, the orchestration layer supports cancellation through `tokio_util::sync::CancellationToken`. When passed to `generate_with_builtin`, the token allows immediate abortion of in-flight requests. Upon cancellation, the engine triggers `force_shutdown_sidecar` to terminate the LLM process instantly, freeing system resources.

### What happens if the sidecar process crashes during generation?

The `SidecarManager` implements health monitoring via `is_healthy()` checks and `ensure_running()` guards. If a crash is detected, the manager automatically attempts to restart the sidecar process and reload the requested model. Failed health checks propagate as `anyhow` errors to the caller, allowing the UI to display appropriate error messages or fall back to remote LLM providers via [`llm_client.rs`](https://github.com/Zackriya-Solutions/meetily/blob/main/llm_client.rs).