# How Magika Is Implemented in Rust: Core Library and CLI Architecture > Discover how Magika is implemented in Rust, exploring its core library and CLI architecture for efficient content inference and batch processing. Learn about its builder pattern, abstracted I/O, and post-processing layers. - Repository: [Google/magika](https://github.com/google/magika) - Tags: architecture - Published: 2026-04-16 --- **Magika's Rust implementation consists of two crates—`magika_lib` for ONNX-based content inference and `magika_cli` for async batch processing—using a builder pattern for session configuration, abstracted I/O for feature extraction, and a threshold-based post-processing layer to map raw logits to content types.** The `google/magika` repository provides a high-performance Rust implementation of Magika's content-type detection engine. Located in the `rust/` directory, this implementation splits functionality between a core library and a command-line interface, enabling both embedded use and standalone operation. Understanding how Magika is implemented in Rust reveals a clean architecture built around the ONNX Runtime, zero-copy feature extraction, and ergonomic async abstractions. ## Core Library Architecture (`magika_lib`) The core library in `rust/lib/` handles model loading, feature extraction, and inference. It is structured as a set of cohesive modules that transform raw file bytes into structured content-type predictions. ### Session Creation with the Builder Pattern In [`rust/lib/src/builder.rs`](https://github.com/google/magika/blob/main/rust/lib/src/builder.rs), Magika exposes a `Builder` struct that configures the underlying ONNX Runtime session before instantiation. This follows the standard builder pattern, allowing callers to chain configuration methods. ```rust pub struct Builder { … } impl Builder { pub fn with_inter_threads(self, n: usize) -> Self { … } pub fn with_intra_threads(self, n: usize) -> Self { … } pub fn with_optimization_level(self, lvl: GraphOptimizationLevel) -> Self { … } pub fn with_parallel_execution(self, p: bool) -> Self { … } pub fn build(self) -> Result { … } } ``` The `build()` method embeds the compiled model file using `include_bytes!("model.onnx")`, ensuring the ONNX model is baked into the binary at compile time. All session-level knobs map directly to equivalent ONNX Runtime configuration options, including thread parallelism and graph optimization levels. ### The Session API The `Session` struct in [`rust/lib/src/session.rs`](https://github.com/google/magika/blob/main/rust/lib/src/session.rs) and [`rust/lib/src/lib.rs`](https://github.com/google/magika/blob/main/rust/lib/src/lib.rs) provides the primary public interface. It wraps the `ort::session::Session` and exposes both synchronous and asynchronous methods for content identification. ```rust pub struct Session { pub(crate) session: ort::session::Session, } impl Session { pub fn new() -> Result { Builder::default().build() } pub fn identify_file_sync(&mut self, path: impl AsRef) -> Result { … } pub fn identify_content_sync(&mut self, data: impl SyncInput) -> Result { … } // async counterparts (identify_file_async, identify_content_async, …) } ``` The `Session` handles the heavy lifting of feature extraction and inference, abstracting the ONNX Runtime details from the caller. ### Feature Extraction Pipeline Feature extraction resides in [`rust/lib/src/input.rs`](https://github.com/google/magika/blob/main/rust/lib/src/input.rs). The model expects a fixed-size vector of integer tokens representing the first and last `block_size` bytes of a file. ```rust pub async fn extract(file: impl AsyncInput) -> Result { // 1️⃣ Read a prefix (beg) and a suffix (end) of the file // 2️⃣ Strip leading/trailing whitespace (strip_prefix / strip_suffix) // 3️⃣ Pad to `config.beg_size` / `config.end_size` with `config.padding_token` // 4️⃣ Return either a `Features` vector or a rule-based `ContentType` } ``` The module defines `SyncInput` and `AsyncInput` traits that abstract over `&[u8]`, `std::fs::File`, and `tokio::fs::File`. This allows the same extraction pipeline to operate in both blocking and async contexts without code duplication. ### Model Configuration and Thresholds Static model parameters live in [`rust/lib/src/model.rs`](https://github.com/google/magika/blob/main/rust/lib/src/model.rs) and [`rust/lib/src/config.rs`](https://github.com/google/magika/blob/main/rust/lib/src/config.rs). The `ModelConfig` struct holds dimensions, thresholds, and an overwrite map. ```rust pub(crate) struct ModelConfig { pub beg_size: usize, pub end_size: usize, pub min_file_size_for_dl: usize, pub padding_token: i32, pub block_size: usize, pub thresholds: Cow<'static, [f32; ContentType::SIZE]>, pub overwrite_map: Cow<'static, [ContentType; ContentType::SIZE]>, } ``` The `CONFIG` constant (generated in `rust/gen`) contains per-class confidence thresholds and an overwrite map that replaces low-confidence AI predictions with rule-based fallbacks. ### Running Inference The inference logic in [`rust/lib/src/session.rs`](https://github.com/google/magika/blob/main/rust/lib/src/session.rs) and [`rust/lib/src/future.rs`](https://github.com/google/magika/blob/main/rust/lib/src/future.rs) converts extracted features into content-type predictions. ```rust let input = Array2::from_shape_vec([features.len(), cfg.features_size()], …)?; let mut output = env::ort_session_run(&mut self.session, input).await?; let logits = output.remove("target_label").unwrap().try_extract_array()?; let file_type = FileType::convert(logits); ``` The `identify_features_batch` method constructs a 2-D tensor where each row represents a file's feature vector. It invokes `ort_session_run` (abstracted via the `Env` trait in [`future.rs`](https://github.com/google/magika/blob/main/future.rs)) and post-processes the raw logits using `FileType::convert`, which applies per-class thresholds and the overwrite map. ### Result Types and Content Mapping The final output types are defined in [`rust/lib/src/file.rs`](https://github.com/google/magika/blob/main/rust/lib/src/file.rs) and [`rust/lib/src/content.rs`](https://github.com/google/magika/blob/main/rust/lib/src/content.rs). * `FileType` is an enum with variants `Directory`, `Symlink`, and `Inferred(InferredType)`. * `InferredType` contains the AI-predicted `ContentType`, an optional overwrite reason, and the confidence score. * `ContentType` (generated in [`content.rs`](https://github.com/google/magika/blob/main/content.rs)) provides human-readable labels, MIME types, groups, and file extensions. ## Command-Line Interface (`magika_cli`) The CLI crate in `rust/cli/` provides a `tokio`-based asynchronous front-end that drives the library for batch processing. ### Async Batch Processing Architecture The entry point in [`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs) implements a producer-consumer pipeline: 1. **Feature Extraction Task**: Walks the input list, extracts features (or applies rule-based detection), and sends batches over an async channel. 2. **Worker Pool**: Spawns `num_tasks` workers, each owning a `Session` instance, to run `identify_features_batch_async` on incoming batches. 3. **Result Ordering**: Reorders results to preserve input order before printing. Key CLI features include recursive directory walking (`process_path` at lines 95-121), stdin handling (lines 97-100), and custom output formatting via placeholder parsing in `Response::format` (lines 40-90). Color-coded output maps content groups to a Tailwind color palette via `Response::color` (lines 56-74). ## Practical Usage Examples Below are minimal, copy-pasteable examples for embedding Magika in your own Rust projects. ### Synchronous File Identification ```rust use magika_lib::{Session, Result}; fn main() -> Result<()> { // Create a session with default configuration let mut magika = Session::new()?; // Identify a file on disk let ftype = magika.identify_file_sync("Cargo.toml")?; println!("Label: {}", ftype.info().label); println!("MIME: {}", ftype.info().mime_type); Ok(()) } ``` This example uses the public API from [`lib.rs`](https://github.com/google/magika/blob/main/lib.rs) and accesses file metadata via the `info()` method defined in [`file.rs`](https://github.com/google/magika/blob/main/file.rs). ### Asynchronous Content Processing ```rust use magika_lib::{Session, AsyncInput, Result}; use tokio::fs::File; #[tokio::main] async fn main() -> Result<()> { let mut magika = Session::new()?; // Identify a file by reading its content into memory let mut f = File::open("README.md").await?; let ftype = magika.identify_content_async(&mut f).await?; println!("Detected: {}", ftype.info().description); Ok(()) } ``` This relies on the `AsyncInput` trait from [`input.rs`](https://github.com/google/magika/blob/main/input.rs) and the async methods in [`session.rs`](https://github.com/google/magika/blob/main/session.rs). ### Custom Builder Configuration ```rust use magika_lib::{Builder, Session, Result}; use ort::session::builder::GraphOptimizationLevel; fn custom_session() -> Result { Builder::default() .with_inter_threads(4) // parallelism across graph nodes .with_intra_threads(2) // parallelism inside each node .with_optimization_level(GraphOptimizationLevel::Level2) .build() } ``` This advanced configuration is defined in [`builder.rs`](https://github.com/google/magika/blob/main/builder.rs) and allows fine-tuning of the ONNX Runtime execution environment. ## Summary - **Magika's Rust implementation** is split into `magika_lib` (core inference) and `magika_cli` (async batch processing front-end). - **Builder pattern** in [`rust/lib/src/builder.rs`](https://github.com/google/magika/blob/main/rust/lib/src/builder.rs) configures the ONNX Runtime session with embedded model bytes. - **Feature extraction** in [`rust/lib/src/input.rs`](https://github.com/google/magika/blob/main/rust/lib/src/input.rs) uses `SyncInput` and `AsyncInput` traits to tokenize file prefixes and suffixes for the neural network. - **Inference pipeline** in [`rust/lib/src/session.rs`](https://github.com/google/magika/blob/main/rust/lib/src/session.rs) batches features into tensors, runs them through the ONNX model, and applies per-class thresholds and overwrite maps from [`rust/lib/src/model.rs`](https://github.com/google/magika/blob/main/rust/lib/src/model.rs). - **Result types** in [`rust/lib/src/file.rs`](https://github.com/google/magika/blob/main/rust/lib/src/file.rs) and [`rust/lib/src/content.rs`](https://github.com/google/magika/blob/main/rust/lib/src/content.rs) provide structured metadata including MIME types, labels, and confidence scores. - **CLI architecture** in [`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs) implements a producer-consumer pattern with Tokio for parallel batch processing and flexible output formatting. ## Frequently Asked Questions ### How does Magika handle both synchronous and asynchronous file I/O in Rust? Magika abstracts file I/O through the `SyncInput` and `AsyncInput` traits defined in [`rust/lib/src/input.rs`](https://github.com/google/magika/blob/main/rust/lib/src/input.rs). The library implements these traits for `&[u8]`, `std::fs::File`, and `tokio::fs::File`, allowing the same feature extraction logic to run in both blocking and async contexts. When using `identify_content_async`, the session reads file chunks asynchronously without blocking the runtime, while `identify_content_sync` uses standard blocking I/O. ### Where is the ONNX model stored in the Rust binary? The compiled ONNX model is embedded directly into the library binary using `include_bytes!("model.onnx")` inside [`rust/lib/src/builder.rs`](https://github.com/google/magika/blob/main/rust/lib/src/builder.rs). This ensures that the `magika_lib` crate is self-contained and does not require external model files at runtime. When `Builder::build()` is called, it loads these bytes into an ONNX Runtime session using the `ort` crate, configuring thread counts and optimization levels based on the builder's state. ### How does Magika convert raw model outputs into final content type predictions? After inference in [`rust/lib/src/session.rs`](https://github.com/google/magika/blob/main/rust/lib/src/session.rs), raw logits are processed by `FileType::convert` in [`rust/lib/src/file.rs`](https://github.com/google/magika/blob/main/rust/lib/src/file.rs). This conversion applies per-class confidence thresholds and an overwrite map defined in [`rust/lib/src/model.rs`](https://github.com/google/magika/blob/main/rust/lib/src/model.rs). If a prediction's confidence falls below its threshold, the system checks the overwrite map to determine if a rule-based fallback should replace the AI prediction. The final `FileType` enum contains variants for `Directory`, `Symlink`, or `Inferred(InferredType)`, where `InferredType` holds the `ContentType`, confidence score, and optional overwrite reason. ### What is the architecture of the Magika CLI batch processing pipeline? The CLI in [`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs) implements a producer-consumer architecture using Tokio. A single feature extraction task walks the input list (handling recursive directories and stdin), extracts features using the `AsyncInput` trait, and sends batches over an async channel. A pool of worker tasks (`num_tasks` concurrent workers) each own a `Session` instance and call `identify_features_batch_async` to run inference. Results are reordered to preserve input sequence before formatting output as plain text, JSON, JSONL, or custom templates with color-coded content groups.