How is the Confidence Score Calculated in Magika

Magika calculates the confidence score by extracting the maximum softmax probability from the ONNX model's output vector, representing the model's certainty that the top-predicted content-type label is correct.

Magika is Google's AI-powered file content-type detection library that uses a deep neural network to classify files. When processing a file, the system generates a confidence score between 0.0 and 1.0 that quantifies the model's certainty in its prediction. Understanding how this score is derived from the raw model outputs helps developers implement appropriate thresholds in their applications.

The Softmax Output from the ONNX Model

The confidence calculation begins with the ONNX runtime inference step. Magika processes extracted byte features through a deep-learning model that outputs a probability distribution over all possible content-type labels. According to the source code in [python/src/magika/magika.py](https://github.com/google/magika/blob/main/python/src/magika/magika.py), the model returns a softmax-activated vector via the target_label output node.

The inference call occurs at lines 794–801, where Magika executes the ONNX session and retrieves the raw prediction vector:

# ONNX session execution returns logits; softmax activation produces probabilities

session = onnxruntime.InferenceSession(model_path)
outputs = session.run(None, inputs)
preds = outputs[0][0]  # Probability vector: one float per content type, summing to 1.0

Extracting the Maximum Probability

Once the softmax vector is available, Magika identifies the predicted label by locating the index with the highest probability value. Rather than converting to external array libraries, the implementation uses Python's built-in max function with a custom key for efficient index retrieval.

At lines 536–537 of magika.py, the code performs the extraction:

target_label_idx = max(range(len(preds)), key=preds.__getitem__)
score = preds[target_label_idx]

The score variable now contains the confidence value—a float between 0.0 and 1.0—representing the probability that the file matches the content type at target_label_idx.

Threshold-Based Decision Making

The raw confidence score determines whether Magika returns the specific predicted label or falls back to a generic classification. The system compares the score against per-content-type thresholds stored in the model configuration file [config.min.json](https://github.com/google/magika/blob/main/python/src/magika/models/standard_v3_3/config.min.json).

• High-confidence mode (default): Returns the predicted label only if score exceeds the type-specific threshold; otherwise returns txt or unknown
• Best-guess mode: Returns the predicted label regardless of score, but the score still reflects actual model certainty

The final score is stored in MagikaResult.prediction.score as defined in [magika_prediction.py](https://github.com/google/magika/blob/main/python/src/magika/types/magika_result.py).

Implementation Consistency Across Languages

While the Python implementation in magika.py serves as the reference, the Rust implementation ensures identical behavior for performance-critical applications. The Rust code in [rust/lib/src/lib.rs](https://github.com/google/magika/blob/main/rust/lib/src/lib.rs) mirrors the same selection logic: identifying the maximum probability in the softmax output and returning it as the confidence score. This parity ensures that the command-line client (implemented in [magika_client.py](https://github.com/google/magika/blob/main/python/src/magika/cli/magika_client.py)) and Python library produce identical scores for the same input file.

Example: Retrieving Confidence Scores Programmatically

You can inspect the confidence score after initializing the Magika client. The score is always available via the prediction.score attribute, regardless of prediction mode:

from magika import Magika

# Initialize with default model

magika = Magika()

# Detect file and access confidence metrics

result = magika.detect_path("example.pdf")
print(f"Content type: {result.prediction.label}")
print(f"Confidence: {result.prediction.score:.4f}")  # e.g., 0.9872

# Compare modes—score remains the underlying probability

result_strict = magika.detect_path("ambiguous.txt")
result_best = magika.detect_path("ambiguous.txt", prediction_mode="best_guess")
print(f"Same score in both modes: {result_strict.prediction.score == result_best.prediction.score}")

Summary

• Confidence scores in Magika represent the softmax probability of the top-predicted content-type label, ranging from 0.0 (no confidence) to 1.0 (certainty).
• The calculation is performed in magika.py lines 536–537 by selecting the maximum value from the model's probability distribution vector.
• Scores are evaluated against thresholds in config.min.json to determine whether to return the specific label or downgrade to txt/unknown.
• Both Python and Rust implementations use identical algorithms, ensuring consistent scoring across the google/magika repository.

Frequently Asked Questions

What does a confidence score of 1.0 mean in Magika?

A score of 1.0 indicates the model assigned 100% of the probability mass to a single content-type label in the softmax output. While theoretically possible, practical scores typically fall in the 0.9–0.99 range for high-confidence predictions; values above 0.95 generally indicate strong model certainty.

How does Magika handle low-confidence predictions?

In the default high-confidence mode, Magika compares the calculated score against per-type thresholds defined in the model configuration. If the score falls below the threshold for the predicted content type, Magika downgrades the result to txt (for text-like content) or unknown (for binary content), signaling that the specific classification is unreliable.

Can I use the confidence score to implement custom filtering?

Yes. The MagikaResult object exposes the raw score via result.prediction.score, allowing you to implement application-specific thresholds. For example, you might require score > 0.8 before trusting automatic file routing, or use the score to prioritize files for manual review when classifications are uncertain.

Is the confidence score calculation different between Python and Rust implementations?

No. Both implementations follow the identical mathematical procedure: extracting the maximum value from the softmax probability vector returned by the ONNX model. The Rust implementation in rust/lib/src/lib.rs mirrors the Python logic in magika.py, ensuring consistent confidence scores whether using the Python library, CLI tool, or Rust bindings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}