# Magika Training Data Size and Performance: Inside Google's Lightweight File Detection AI

> Discover Magika's training data size and performance. Google's AI uses 100M files for 99% accuracy and 5ms inference. Learn how this lightweight file detection model works.

- Repository: [Google/magika](https://github.com/google/magika)
- Tags: performance
- Published: 2026-04-16

---

**Magika was trained on roughly 100 million files spanning 200+ content types, achieves approximately 99% average precision-recall accuracy, and delivers inference in about 5 milliseconds per file on a single CPU core.**

The open-source `google/magika` project provides a production-ready neural network for content type detection that balances massive training scale with minimal runtime overhead. Understanding Magika's training data size and performance characteristics reveals why it outperforms traditional file signature approaches while maintaining consistent sub-10ms latency across diverse file formats.

## Training Dataset Scale and Accuracy

According to the project documentation in [`website-ng/src/content/docs/introduction/overview.md`](https://github.com/google/magika/blob/main/website-ng/src/content/docs/introduction/overview.md), Magika's neural network was **trained and evaluated on a dataset of approximately 100 million samples** covering **more than 200 distinct content types**, including both binary executables and textual formats. This extensive training corpus enables the model to generalize across rare file variants and novel format combinations that traditional magic-number detection often misses.

The evaluation metrics documented in both the repository's top-level [`README.md`](https://github.com/google/magika/blob/main/README.md) and the website overview confirm that the model achieves **approximately 99% average precision-recall** on the held-out test set. This high accuracy applies across the full spectrum of supported content types, from common document formats like PDF and Office files to specialized binary structures and source code dialects.

## Inference Performance Benchmarks

Once the model is loaded into memory, **inference takes approximately 5 milliseconds per file** when running on a single CPU core. Critically, this latency remains constant regardless of the input file's total size because Magika only ingests a small, fixed-size slice of each file (typically a few kilobytes) for feature extraction.

The model file itself is exceptionally compact, with the serialized ONNX graph stored at `assets/models/standard_v3_3/model.onnx` occupying only a few megabytes on disk. This small footprint enables rapid deployment in containerized environments and ensures that the one-time model loading overhead (`_init_onnx_session`) has negligible impact on overall system memory usage.

## Technical Architecture

### Model Storage and Format

The production model utilizes the **ONNX (Open Neural Network Exchange)** format, specifically stored as `model.onnx` within the `assets/models/standard_v3_3/` directory. This standardized graph representation ensures interoperability across programming languages while maintaining a minimal storage footprint of just a few megabytes—orders of magnitude smaller than comparable deep learning models for computer vision or natural language processing.

### Runtime Implementation Details

Inference execution relies on **ONNX Runtime** via two primary implementations:

- **Python API**: The `Magika` class in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py) manages the ONNX session lifecycle. The constructor calls `_init_onnx_session()` to compile the model once per process, after which subsequent predictions reuse the optimized session state to achieve the consistent ~5ms latency.

- **Rust CLI**: The command-line interface defined in [`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs) leverages the `ort` crate (Rust bindings for ONNX Runtime) to load the identical `model.onnx` file. This implementation enables parallel file processing while maintaining the same accuracy and latency characteristics as the Python version.

Both implementations extract only a limited byte window from the start of each file, feeding this fixed-size tensor to the neural network regardless of whether the source file is 1 KB or 1 GB.

## Code Implementation Examples

### Python API Integration

The high-level Python interface handles model loading automatically upon class instantiation:

```python
from magika import Magika

# Initialize once (loads assets/models/standard_v3_3/model.onnx)

magika = Magika()

# Identify a single file - ~5ms inference time

result = magika.identify_path("sample.pdf")
print(result.prediction.output.label)   # e.g., "pdf"

print(result.prediction.score)          # confidence score (0-1)

# Process raw bytes without filesystem access

with open("sample.png", "rb") as f:
    data = f.read()
result = magika.identify_bytes(data)
print(result.prediction.output.label)   # e.g., "png"

```

The `Magika` class manages the ONNX Runtime session internally, ensuring that the model compilation overhead occurs only once during initialization as implemented in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py).

### Rust CLI Execution

For batch processing or system integration, the Rust CLI provides optimized throughput via parallel execution:

```bash

# Install the Rust CLI

cargo install --locked magika-cli

# Identify files recursively with ~5ms latency per file

magika -r /path/to/directory | head

# Process specific files with JSON output

magika --json document.pdf image.png executable.exe

```

The CLI entry point in [`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs) creates the ONNX session at startup and dispatches inference tasks for each input path, reusing the compiled model weights across the entire execution lifetime.

## Summary

- **Magika's training data** consists of approximately **100 million files** representing **200+ content types**, enabling robust generalization across binary and textual formats.
- **Inference performance** averages **~5ms per file** on commodity CPUs, with latency independent of file size due to fixed-size feature extraction.
- **Model storage** utilizes a compact **ONNX format** (`assets/models/standard_v3_3/model.onnx`) requiring only a few megabytes of disk space.
- **Runtime implementations** in both Python ([`magika.py`](https://github.com/google/magika/blob/main/magika.py)) and Rust ([`main.rs`](https://github.com/google/magika/blob/main/main.rs)) use ONNX Runtime to execute predictions with approximately **99% average precision-recall accuracy**.
- **Architecture design** loads the model once per process (`_init_onnx_session`) and processes only small byte slices, optimizing for both memory efficiency and throughput.

## Frequently Asked Questions

### How large is Magika's training dataset?

Magika was trained and evaluated on a corpus of roughly **100 million samples** spanning more than **200 distinct content types**, including both binary executables and text-based formats. This dataset size is documented in [`website-ng/src/content/docs/introduction/overview.md`](https://github.com/google/magika/blob/main/website-ng/src/content/docs/introduction/overview.md) and enables the model to achieve high accuracy on rare and novel file variants.

### What is the inference latency of the Magika model?

After the one-time model loading overhead, inference consistently requires **approximately 5 milliseconds per file** when running on a single CPU core. This latency remains constant regardless of the input file's total size because the model only processes a small, fixed-size slice (a few kilobytes) from each file.

### What model format does Magika use for deployment?

Magika serializes its neural network as an **ONNX graph** stored at `assets/models/standard_v3_3/model.onnx`. This format choice enables cross-platform deployment using ONNX Runtime in both Python (via the `onnxruntime` package) and Rust (via the `ort` crate), while maintaining a minimal disk footprint of only a few megabytes.

### Does Magika require GPU acceleration for production inference?

No. The model is specifically optimized for **CPU-only inference** and achieves its ~5ms latency target on single-core processors without requiring GPU acceleration. The ONNX Runtime CPU provider handles all inference operations in both the Python API ([`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py)) and the Rust CLI ([`rust/cli/src/main.rs`](https://github.com/google/magika/blob/main/rust/cli/src/main.rs)).