How Magika Is Implemented in Rust: Core Library and CLI Architecture

Magika's Rust implementation consists of two crates—magika_lib for ONNX-based content inference and magika_cli for async batch processing—using a builder pattern for session configuration, abstracted I/O for feature extraction, and a threshold-based post-processing layer to map raw logits to content types.

The google/magika repository provides a high-performance Rust implementation of Magika's content-type detection engine. Located in the rust/ directory, this implementation splits functionality between a core library and a command-line interface, enabling both embedded use and standalone operation. Understanding how Magika is implemented in Rust reveals a clean architecture built around the ONNX Runtime, zero-copy feature extraction, and ergonomic async abstractions.

Core Library Architecture (magika_lib)

The core library in rust/lib/ handles model loading, feature extraction, and inference. It is structured as a set of cohesive modules that transform raw file bytes into structured content-type predictions.

Session Creation with the Builder Pattern

In rust/lib/src/builder.rs, Magika exposes a Builder struct that configures the underlying ONNX Runtime session before instantiation. This follows the standard builder pattern, allowing callers to chain configuration methods.

pub struct Builder { … }
impl Builder {
    pub fn with_inter_threads(self, n: usize) -> Self { … }
    pub fn with_intra_threads(self, n: usize) -> Self { … }
    pub fn with_optimization_level(self, lvl: GraphOptimizationLevel) -> Self { … }
    pub fn with_parallel_execution(self, p: bool) -> Self { … }
    pub fn build(self) -> Result<Session> { … }
}

The build() method embeds the compiled model file using include_bytes!("model.onnx"), ensuring the ONNX model is baked into the binary at compile time. All session-level knobs map directly to equivalent ONNX Runtime configuration options, including thread parallelism and graph optimization levels.

The Session API

The Session struct in rust/lib/src/session.rs and rust/lib/src/lib.rs provides the primary public interface. It wraps the ort::session::Session and exposes both synchronous and asynchronous methods for content identification.

pub struct Session {
    pub(crate) session: ort::session::Session,
}
impl Session {
    pub fn new() -> Result<Self> { Builder::default().build() }
    pub fn identify_file_sync(&mut self, path: impl AsRef<Path>) -> Result<FileType> { … }
    pub fn identify_content_sync(&mut self, data: impl SyncInput) -> Result<FileType> { … }
    // async counterparts (identify_file_async, identify_content_async, …)
}

The Session handles the heavy lifting of feature extraction and inference, abstracting the ONNX Runtime details from the caller.

Feature Extraction Pipeline

Feature extraction resides in rust/lib/src/input.rs. The model expects a fixed-size vector of integer tokens representing the first and last block_size bytes of a file.

pub async fn extract(file: impl AsyncInput) -> Result<FeaturesOrRuled> {
    // 1️⃣ Read a prefix (beg) and a suffix (end) of the file
    // 2️⃣ Strip leading/trailing whitespace (strip_prefix / strip_suffix)
    // 3️⃣ Pad to `config.beg_size` / `config.end_size` with `config.padding_token`
    // 4️⃣ Return either a `Features` vector or a rule-based `ContentType`
}

The module defines SyncInput and AsyncInput traits that abstract over &[u8], std::fs::File, and tokio::fs::File. This allows the same extraction pipeline to operate in both blocking and async contexts without code duplication.

Model Configuration and Thresholds

Static model parameters live in rust/lib/src/model.rs and rust/lib/src/config.rs. The ModelConfig struct holds dimensions, thresholds, and an overwrite map.

pub(crate) struct ModelConfig {
    pub beg_size: usize,
    pub end_size: usize,
    pub min_file_size_for_dl: usize,
    pub padding_token: i32,
    pub block_size: usize,
    pub thresholds: Cow<'static, [f32; ContentType::SIZE]>,
    pub overwrite_map: Cow<'static, [ContentType; ContentType::SIZE]>,
}

The CONFIG constant (generated in rust/gen) contains per-class confidence thresholds and an overwrite map that replaces low-confidence AI predictions with rule-based fallbacks.

Running Inference

The inference logic in rust/lib/src/session.rs and rust/lib/src/future.rs converts extracted features into content-type predictions.

let input = Array2::from_shape_vec([features.len(), cfg.features_size()], …)?;
let mut output = env::ort_session_run(&mut self.session, input).await?;
let logits = output.remove("target_label").unwrap().try_extract_array()?;
let file_type = FileType::convert(logits);

The identify_features_batch method constructs a 2-D tensor where each row represents a file's feature vector. It invokes ort_session_run (abstracted via the Env trait in future.rs) and post-processes the raw logits using FileType::convert, which applies per-class thresholds and the overwrite map.

Result Types and Content Mapping

The final output types are defined in rust/lib/src/file.rs and rust/lib/src/content.rs.

• FileType is an enum with variants Directory, Symlink, and Inferred(InferredType).
• InferredType contains the AI-predicted ContentType, an optional overwrite reason, and the confidence score.
• ContentType (generated in content.rs) provides human-readable labels, MIME types, groups, and file extensions.

Command-Line Interface (magika_cli)

The CLI crate in rust/cli/ provides a tokio-based asynchronous front-end that drives the library for batch processing.

Async Batch Processing Architecture

The entry point in rust/cli/src/main.rs implements a producer-consumer pipeline:

1. Feature Extraction Task: Walks the input list, extracts features (or applies rule-based detection), and sends batches over an async channel.
2. Worker Pool: Spawns num_tasks workers, each owning a Session instance, to run identify_features_batch_async on incoming batches.
3. Result Ordering: Reorders results to preserve input order before printing.

Key CLI features include recursive directory walking (process_path at lines 95-121), stdin handling (lines 97-100), and custom output formatting via placeholder parsing in Response::format (lines 40-90). Color-coded output maps content groups to a Tailwind color palette via Response::color (lines 56-74).

Practical Usage Examples

Below are minimal, copy-pasteable examples for embedding Magika in your own Rust projects.

Synchronous File Identification

use magika_lib::{Session, Result};

fn main() -> Result<()> {
    // Create a session with default configuration
    let mut magika = Session::new()?;

    // Identify a file on disk
    let ftype = magika.identify_file_sync("Cargo.toml")?;
    println!("Label: {}", ftype.info().label);
    println!("MIME:  {}", ftype.info().mime_type);
    Ok(())
}

This example uses the public API from lib.rs and accesses file metadata via the info() method defined in file.rs.

Asynchronous Content Processing

use magika_lib::{Session, AsyncInput, Result};
use tokio::fs::File;

#[tokio::main]
async fn main() -> Result<()> {
    let mut magika = Session::new()?;

    // Identify a file by reading its content into memory
    let mut f = File::open("README.md").await?;
    let ftype = magika.identify_content_async(&mut f).await?;
    println!("Detected: {}", ftype.info().description);
    Ok(())
}

This relies on the AsyncInput trait from input.rs and the async methods in session.rs.

Custom Builder Configuration

use magika_lib::{Builder, Session, Result};
use ort::session::builder::GraphOptimizationLevel;

fn custom_session() -> Result<Session> {
    Builder::default()
        .with_inter_threads(4)               // parallelism across graph nodes
        .with_intra_threads(2)               // parallelism inside each node
        .with_optimization_level(GraphOptimizationLevel::Level2)
        .build()
}

This advanced configuration is defined in builder.rs and allows fine-tuning of the ONNX Runtime execution environment.

Summary

• Magika's Rust implementation is split into magika_lib (core inference) and magika_cli (async batch processing front-end).
• Builder pattern in rust/lib/src/builder.rs configures the ONNX Runtime session with embedded model bytes.
• Feature extraction in rust/lib/src/input.rs uses SyncInput and AsyncInput traits to tokenize file prefixes and suffixes for the neural network.
• Inference pipeline in rust/lib/src/session.rs batches features into tensors, runs them through the ONNX model, and applies per-class thresholds and overwrite maps from rust/lib/src/model.rs.
• Result types in rust/lib/src/file.rs and rust/lib/src/content.rs provide structured metadata including MIME types, labels, and confidence scores.
• CLI architecture in rust/cli/src/main.rs implements a producer-consumer pattern with Tokio for parallel batch processing and flexible output formatting.

Frequently Asked Questions

How does Magika handle both synchronous and asynchronous file I/O in Rust?

Magika abstracts file I/O through the SyncInput and AsyncInput traits defined in rust/lib/src/input.rs. The library implements these traits for &[u8], std::fs::File, and tokio::fs::File, allowing the same feature extraction logic to run in both blocking and async contexts. When using identify_content_async, the session reads file chunks asynchronously without blocking the runtime, while identify_content_sync uses standard blocking I/O.

Where is the ONNX model stored in the Rust binary?

The compiled ONNX model is embedded directly into the library binary using include_bytes!("model.onnx") inside rust/lib/src/builder.rs. This ensures that the magika_lib crate is self-contained and does not require external model files at runtime. When Builder::build() is called, it loads these bytes into an ONNX Runtime session using the ort crate, configuring thread counts and optimization levels based on the builder's state.

How does Magika convert raw model outputs into final content type predictions?

After inference in rust/lib/src/session.rs, raw logits are processed by FileType::convert in rust/lib/src/file.rs. This conversion applies per-class confidence thresholds and an overwrite map defined in rust/lib/src/model.rs. If a prediction's confidence falls below its threshold, the system checks the overwrite map to determine if a rule-based fallback should replace the AI prediction. The final FileType enum contains variants for Directory, Symlink, or Inferred(InferredType), where InferredType holds the ContentType, confidence score, and optional overwrite reason.

What is the architecture of the Magika CLI batch processing pipeline?

The CLI in rust/cli/src/main.rs implements a producer-consumer architecture using Tokio. A single feature extraction task walks the input list (handling recursive directories and stdin), extracts features using the AsyncInput trait, and sends batches over an async channel. A pool of worker tasks (num_tasks concurrent workers) each own a Session instance and call identify_features_batch_async to run inference. Results are reordered to preserve input sequence before formatting output as plain text, JSON, JSONL, or custom templates with color-coded content groups.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}