What Heuristic Magika Uses for Very Small Files: Inside the 8-Byte Threshold

When a file is smaller than 8 bytes, Magika bypasses its neural network and applies a simple heuristic that classifies valid UTF-8 bytes as txt and everything else as unknown.

Google's Magika content-type detection system uses a lightweight fallback mechanism for files too small to feed into its deep-learning model. When a file contains fewer than 8 bytes—the default min_file_size_for_dl threshold defined in the Rust core—Magika triggers a heuristic that inspects the raw bytes for UTF-8 validity rather than attempting model inference.

The 8-Byte Threshold Trigger

The decision to use the heuristic starts with a size check in rust/lib/src/input.rs. Magika reads the file into a feature vector where positions correspond to byte indices. If the file length is less than min_file_size_for_dl (8 bytes by default), the value at index 7 remains the special padding token, signaling the model cannot be used.

The code checks this condition before feature extraction:

if features[config.min_file_size_for_dl - 1] != config.padding_token {
    // File is large enough → use the ML model
    return Ok(FeaturesOrRuled::Features(Features(features)));
}

When this check fails, execution falls through to the heuristic path.

The UTF-8 Validation Heuristic

For files 8 bytes or smaller, Magika applies a brutally simple content-type rule in the extract() function within rust/lib/src/input.rs. It attempts to decode the first block of bytes as UTF-8 text.

If the bytes decode successfully as UTF-8, Magika returns the content type txt, assuming the file is plain text. If decoding fails, it returns unknown, indicating binary or unrecognized data.

The implementation uses Rust's std::str::from_utf8:

let content_type = match std::str::from_utf8(&first_block) {
    Ok(_) => ContentType::Txt,      // Valid UTF-8 → plain text
    Err(_) => ContentType::Unknown, // Not UTF-8 → unknown type
};

This binary classification avoids the computational overhead of the neural model while providing a reasonable guess for extremely small files.

Configuration and Defaults

The threshold constant is defined in rust/lib/src/model.rs as part of the model configuration structure. The default value of 8 bytes (min_file_size_for_dl: 8) represents the minimum input size the deep-learning model can effectively process.

Documentation in website-ng/src/content/docs/core-concepts/how-magika-works.md confirms this behavior, stating that "if the file is too small for the model (e.g., under ~8 bytes), Magika uses simple heuristics to return a generic answer like txt or unknown."

Practical Examples

Python Client

When using the official Python client, the heuristic triggers automatically for files under 8 bytes:

from magika import Magika

magika = Magika()

# 5-byte plain-text file

with open("tiny.txt", "wb") as f:
    f.write(b"Hello")

print(magika.run("tiny.txt").output)   # → txt

# 5-byte binary file

with open("tiny.bin", "wb") as f:
    f.write(b"\x00\xff\x01\x02\x03")

print(magika.run("tiny.bin").output)   # → unknown

Rust Implementation

Directly invoking the feature extraction in Rust demonstrates the heuristic path:

use magika::input::extract_async;
use magika::input::AsyncFile;

#[tokio::main]
async fn main() {
    // File smaller than 8 bytes
    let result = extract_async(AsyncFile::open("tiny.bin").await.unwrap())
        .await
        .unwrap();

    match result {
        FeaturesOrRuled::Ruled(content_type) => {
            println!("Heuristic result: {}", content_type); 
            // Prints "txt" for UTF-8, "unknown" otherwise
        }
        FeaturesOrRuled::Features(_) => {
            println!("Model inference path (should not happen for tiny files)");
        }
    }
}

Summary

• Magika uses an 8-byte threshold (min_file_size_for_dl) to determine when a file is too small for neural-network inference.
• Files ≤8 bytes trigger a UTF-8 validation heuristic in rust/lib/src/input.rs instead of the ML model.
• Valid UTF-8 bytes are classified as txt; invalid UTF-8 bytes are classified as unknown.
• This fallback ensures Magika returns fast, deterministic results for edge-case files without loading the deep-learning model.

Frequently Asked Questions

What is the minimum file size for Magika's neural network?

The minimum file size is 8 bytes, defined by the min_file_size_for_dl constant in rust/lib/src/model.rs. Files smaller than this threshold bypass the neural network entirely and use the UTF-8 heuristic instead.

How does Magika classify a 5-byte text file?

Magika reads the 5 bytes and attempts to decode them as UTF-8. If the decoding succeeds—as it would for ASCII characters like "Hello"—the file is classified as txt. This determination happens in the extract() function within rust/lib/src/input.rs.

Why does Magika return "unknown" for some tiny files?

The unknown classification occurs when the byte sequence cannot be interpreted as valid UTF-8. This typically indicates binary data or non-text encodings. Rather than guessing incorrectly, Magika's heuristic conservatively labels these files as unknown when the UTF-8 validation fails.

Can I configure the threshold for small file detection?

The 8-byte threshold is hardcoded as a default in the model configuration, but the Rust implementation allows custom configurations when initializing the Magika instance. However, the heuristic logic itself—classifying valid UTF-8 as txt and invalid as unknown—remains fixed for any file falling below the minimum size threshold.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}