How Magika Uses Confidence Thresholds for File Type Predictions: A Deep Dive

Magika applies configurable confidence thresholds defined in ModelConfig to determine whether to trust the deep-learning model's output or fall back to generic labels like TXT or UNKNOWN, with three prediction modes controlling the strictness of these validations.

Google's open-source Magika library uses a deep-learning model to identify file types, but raw model outputs aren't always reliable. To handle uncertainty, Magika implements a sophisticated confidence threshold system that filters predictions based on user-selected strictness levels. This mechanism ensures that low-confidence guesses are replaced with safer generic alternatives rather than potentially incorrect specific labels.

Model Configuration and Threshold Storage

When Magika loads a model, it constructs a ModelConfig object from a JSON configuration file located in python/src/magika/types/model.py (lines 44-55). This configuration contains three critical fields that govern confidence handling:

• medium_confidence_threshold – A global float value (e.g., 0.5) that serves as the default threshold for MEDIUM_CONFIDENCE mode and the fallback for per-type checks in HIGH_CONFIDENCE mode.
• thresholds – An optional dictionary mapping content-type labels to specific high-confidence thresholds (e.g., 0.95 for PDF), allowing stricter validation for specific file types.
• overwrite_map – A dictionary that can remap raw deep-learning labels to different labels before the confidence check is applied.

The model configuration is loaded during initialization via Magika._load_model_config in python/src/magika/magika.py (lines 7-27).

Prediction Modes and Confidence Handling

Magika supports three mutually exclusive prediction modes defined in python/src/magika/types/prediction_mode.py (lines 25-34). These modes determine how strictly the library applies confidence thresholds before accepting a prediction.

BEST_GUESS Mode

In PredictionMode.BEST_GUESS, Magika always returns the raw deep-learning label (after applying any overwrite maps), regardless of the confidence score. This mode trusts the model completely and never falls back to TXT or UNKNOWN, making it suitable when you need a specific type guess even for ambiguous files.

MEDIUM_CONFIDENCE Mode

PredictionMode.MEDIUM_CONFIDENCE requires that the model's score meet or exceed the global medium_confidence_threshold defined in the model config. If the score falls below this threshold, Magika replaces the output with a generic fallback—TXT if the original label represents a text type, or UNKNOWN otherwise. This mode provides a balanced approach between specificity and reliability.

HIGH_CONFIDENCE Mode

PredictionMode.HIGH_CONFIDENCE uses per-type thresholds from the thresholds dictionary when available, falling back to the global medium threshold only when a specific type isn't listed. This allows critical file types like PDF or executables to require higher confidence scores (e.g., 0.95) while other types use the standard threshold.

The Decision Logic in magika.py

The core confidence validation occurs in Magika._get_output_label_from_dl_label_and_score within python/src/magika/magika.py (lines 93-118 for confidence checks, lines 120-132 for fallback logic). The method executes the following decision tree:

1. Applies the overwrite map to transform the raw DL label into an intermediate output_label.
2. Evaluates the selected prediction mode against the score:
   • BEST_GUESS: Accepts the label immediately.
   • HIGH_CONFIDENCE: Accepts only if score >= thresholds.get(dl_label, medium_confidence_threshold).
   • MEDIUM_CONFIDENCE: Accepts only if score >= medium_confidence_threshold.
3. If no confidence check passes, sets overwrite_reason to LOW_CONFIDENCE and falls back to TXT or UNKNOWN.

The fallback logic differentiates between text and binary content types, ensuring that text files receive a TXT label while binary files receive UNKNOWN when confidence is insufficient.

Understanding Prediction Results and Overwrite Reasons

The final MagikaResult object exposes the confidence handling through three key fields:

• prediction.dl – The raw deep-learning output containing the original label and score.
• prediction.output – The final label after confidence threshold checks and potential fallbacks.
• prediction.overwrite_reason – An enum value explaining any deviation from the DL label: OVERWRITE_MAP (label remapped), LOW_CONFIDENCE (fallback applied), or NONE (original label kept).

When using the CLI, users receive explicit warnings for low-confidence downgrades. The client code in python/src/magika/cli/magika_client.py (lines 99-108) checks for OverwriteReason.LOW_CONFIDENCE and displays messages like:

[Low-confidence model best-guess: <dl description> (<dl group>), score=0.73]

Practical Code Examples

Using Different Prediction Modes

from magika import Magika, PredictionMode

# Default instance uses MEDIUM_CONFIDENCE

magika = Magika()

# BEST_GUESS - never falls back

best_guess = Magika(prediction_mode=PredictionMode.BEST_GUESS)
result = best_guess.identify_path("suspicious.bin")
print(result.prediction.output.label)  # Always returns DL label regardless of score

# HIGH_CONFIDENCE - uses per-type thresholds

strict = Magika(prediction_mode=PredictionMode.HIGH_CONFIDENCE)
result = strict.identify_path("document.pdf")

# Only returns PDF if score >= PDF-specific threshold (e.g., 0.95)

Inspecting Confidence Fallbacks

from magika import Magika, PredictionMode, OverwriteReason

magika = Magika(prediction_mode=PredictionMode.MEDIUM_CONFIDENCE)
result = magika.identify_path("ambiguous.dat")

if result.prediction.overwrite_reason == OverwriteReason.LOW_CONFIDENCE:
    print(f"Score {result.prediction.score:.2f} below threshold; "
          f"fallback to {result.prediction.output.label}")

Customizing Thresholds

import json
from pathlib import Path
from magika import Magika, PredictionMode

# Load and modify the default model configuration

config_path = Path(Magika._DEFAULT_MODEL_DIR) / "model_config.json"
config = json.loads(config_path.read_text())

# Raise global medium threshold

config["medium_confidence_threshold"] = 0.7

# Require higher confidence for JavaScript files

config["thresholds"]["javascript"] = 0.9

# Save and use custom config

custom_path = config_path.with_name("custom_config.json")
custom_path.write_text(json.dumps(config))

magika = Magika(model_dir=custom_path.parent, 
                prediction_mode=PredictionMode.HIGH_CONFIDENCE)

Summary

• Magika uses a ModelConfig object stored in python/src/magika/types/model.py to manage confidence thresholds, including a global medium_confidence_threshold and optional per-type thresholds.
• Three prediction modes control validation strictness: BEST_GUESS (no filtering), MEDIUM_CONFIDENCE (global threshold), and HIGH_CONFIDENCE (per-type thresholds).
• The decision logic lives in Magika._get_output_label_from_dl_label_and_score within python/src/magika/magika.py, which falls back to TXT or UNKNOWN when scores are insufficient.
• Results expose an overwrite_reason field that indicates whether LOW_CONFIDENCE triggered a fallback, enabling programmatic handling of uncertain predictions.
• The CLI surfaces low-confidence warnings through python/src/magika/cli/magika_client.py, providing transparency when the model is uncertain.

Frequently Asked Questions

What is the difference between MEDIUM_CONFIDENCE and HIGH_CONFIDENCE modes in Magika?

MEDIUM_CONFIDENCE mode uses the single global medium_confidence_threshold value for all file types, while HIGH_CONFIDENCE mode consults the per-type thresholds dictionary first, falling back to the global threshold only when a specific content type isn't listed. HIGH_CONFIDENCE allows critical file types like PDF or executables to require higher certainty (e.g., 0.95) while maintaining standard thresholds for others.

How can I customize confidence thresholds for specific file types?

Modify the thresholds dictionary in your model's JSON configuration file, which maps content-type labels (like "pdf" or "javascript") to float values between 0 and 1. Load this configuration via the model_dir parameter when initializing Magika, or edit the default config at Magika._DEFAULT_MODEL_DIR / "model_config.json" before instantiation.

Why does Magika return TXT or UNKNOWN for some files instead of the predicted type?

When the deep-learning model's confidence score falls below the threshold defined by the current prediction mode, Magika applies a fallback mechanism in python/src/magika/magika.py (lines 120-132). If the original prediction was a text content type, it returns TXT; otherwise, it returns UNKNOWN. This prevents high-confidence false positives on ambiguous or unfamiliar file content.

Where does Magika store the confidence threshold configuration?

Thresholds are stored in a JSON file named model_config.json within the model directory, parsed into a ModelConfig dataclass defined in python/src/magika/types/model.py (lines 44-55). This file contains medium_confidence_threshold (float), thresholds (dictionary of content-type-specific values), and overwrite_map (label remappings).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}