# How Magika Extracts Features From Files for Its Model: Implementation Deep Dive

> Learn how Magika extracts file features for its deep learning model. Discover its byte reading, integer conversion, and vector padding techniques.

- Repository: [Google/magika](https://github.com/google/magika)
- Tags: deep-dive
- Published: 2026-04-16

---

**Magika extracts features by reading at most `block_size` bytes from the beginning and end of a seekable file, stripping whitespace, converting the remaining bytes to integers (0-255), and padding with a configurable token to produce a fixed-length numeric vector that the deep-learning model consumes.**

Magika is Google's open-source AI-powered file type detection tool that uses neural networks to identify content types with high accuracy. Unlike traditional signature-based scanners that read entire files, **Magika feature extraction** employs a lightweight, deterministic pipeline that works on any seekable input—whether a file on disk or a `BytesIO` buffer—without loading the full content into memory.

## The Core Feature Extraction Pipeline

The extraction process follows identical logic across all language implementations (Python, Rust, and Go), differing only in I/O APIs. The pipeline converts raw binary data into a structured integer vector through five distinct steps.

### Determining the Window Size and Position

Magika begins by calculating a small read window based on `block_size`, a parameter defined in the model configuration (defaulting to approximately 4 KB). For any given file, the system reads at most this many bytes from offset `0` (the beginning) and the same amount from the end of the file. This operation occurs in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py) within the `_extract_features_from_seekable` method, and in [`rust/lib/src/input.rs`](https://github.com/google/magika/blob/main/rust/lib/src/input.rs) via the `extract_features_async` function.

### Normalizing Content with Whitespace Stripping

After reading the raw bytes, Magika normalizes the data by stripping insignificant padding. Leading whitespace is removed from the beginning chunk using `strip_prefix` (Rust) or equivalent logic, while trailing whitespace is removed from the end chunk using `strip_suffix`. In the Go implementation at [`go/magika/features.go`](https://github.com/google/magika/blob/main/go/magika/features.go), this uses `bytes.TrimLeft` and `bytes.TrimRight` against a space character set (`\t`, `\n`, `\v`, `\f`, `\r`, ` `). This normalization ensures that files with varying amounts of header or footer whitespace generate consistent feature vectors.

### Slicing and Padding to Fixed Dimensions

The cleaned byte arrays are then sliced to exact specifications: `beg_size` bytes are retained from the beginning, and `end_size` bytes from the end. If a file is smaller than the requested window, or if whitespace stripping removed significant content, the remainder is padded with `padding_token` (normally `0`). As implemented in the configuration structs (referenced across [`rust/lib/src/config.rs`](https://github.com/google/magika/blob/main/rust/lib/src/config.rs) and analogous Python/Go files), this guarantees every file produces a feature vector of exactly `features_size` length, regardless of actual file size.

### Converting Bytes to Integer Vectors

Finally, each retained byte is cast to an integer value between 0 and 255. The beginning and end vectors are concatenated (and in Go, optionally combined with a middle window) to form the final input tensor. This numeric representation allows the ONNX model to perform content classification without parsing file formats or decoding text.

### Legacy Offset Features

Older model versions supported reading eight-byte blocks at fixed hexadecimal offsets (0x8000, 0x8800, etc.), but the current standard model disables this feature by setting `use_inputs_at_offsets` to `False` in the configuration.

## Language-Specific Implementation Details

While the logic remains consistent, each implementation optimizes for its runtime environment.

### Python Implementation

In [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py) (lines 40-71), the `_extract_features_from_seekable` function handles synchronous extraction. It accepts a seekable wrapper providing `size` and `read_at` methods, then orchestrates the window reading, stripping, and padding sequence using the model's `beg_size`, `end_size`, `mid_size`, `padding_token`, and `block_size` parameters.

### Rust Async Implementation

The Rust library at [`rust/lib/src/input.rs`](https://github.com/google/magika/blob/main/rust/lib/src/input.rs) provides `extract_features_async` for non-blocking I/O:

```rust
let buffer_size = std::cmp::min(config.block_size as u64, file_len) as usize;
let mut content_beg = vec![0; buffer_size];
file.read_at(&mut content_beg, 0).await?;
let beg = strip_prefix(&content_beg);

let mut end = vec![0; buffer_size];
file.read_at(&mut end, file_len - buffer_size as u64).await?;
let end = strip_suffix(&end);

let mut features = vec![config.padding_token; config.features_size()];
let split_features = config.split_features(&mut features);
copy_features(split_features.beg, beg, 0);
copy_features(split_features.end, end, 1);

```

This async pattern uses `tokio` for efficient file reading while maintaining the same deterministic output as the synchronous Python version.

### Go Implementation

Go's [`go/magika/features.go`](https://github.com/google/magika/blob/main/go/magika/features.go) offers a synchronous `ExtractFeatures` function that additionally supports a middle window extraction:

```go
beg := er.readAt(0, cfg.BlockSize)
mid := er.readAt((size-cfg.MidSize)/2, cfg.MidSize)
end := er.readAt(size-cfg.BlockSize, cfg.BlockSize)

f := buildFeatures(cfg, beg, mid, end)
...
func buildFeatures(cfg Config, beg, mid, end []byte) Features {
    spaces := string([]rune{'\t', '\n', '\v', '\f', '\r', ' '})
    beg = bytes.TrimLeft(beg, spaces)
    end = bytes.TrimRight(end, spaces)
    beg = safeSlice(beg, 0, cfg.BegSize)
    end = safeSlice(end, len(end)-cfg.EndSize, len(end))

    return Features{
        Beg: padInt32(cfg, beg, 0, cfg.BegSize),
        Mid: padInt32(cfg, mid, (cfg.MidSize-len(mid))/2, cfg.MidSize),
        End: padInt32(cfg, end, cfg.EndSize-len(end), cfg.EndSize),
    }
}

```

The `padInt32` helper ensures consistent integer conversion and padding across all three windows.

## Code Examples

### Extracting Features in Python

```python
from pathlib import Path
from magika.magika import Magika, ModelFeatures

# Load the Magika singleton (downloads the ONNX model on first use)

magika = Magika()

# Open a file as a seekable object

with open("example.pdf", "rb") as f:
    # The wrapper provides `size` and `read_at`

    seekable = magika._make_seekable(f)      # internal helper

    feats: ModelFeatures = Magika._extract_features_from_seekable(
        seekable,
        beg_size=magika._model_config.beg_size,
        mid_size=magika._model_config.mid_size,
        end_size=magika._model_config.end_size,
        padding_token=magika._model_config.padding_token,
        block_size=magika._model_config.block_size,
        use_inputs_at_offsets=False,
    )
print(feats.beg[:10], feats.end[:10])   # first 10 ints of each part

```

### Async Feature Extraction in Rust

```rust
use magika_lib::Magika;
use tokio::fs::File;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Build the Magika instance (loads the ONNX model)
    let mut magika = Magika::new().await?;

    // Open a file
    let mut f = File::open("example.png").await?;

    // Extract features (returns either Features or a ruled ContentType)
    match magika.extract_async(&mut f).await? {
        FeaturesOrRuled::Features(feat) => {
            println!("beg: {:?}", &feat.beg[0..5]);
            println!("end: {:?}", &feat.end[0..5]);
        }
        FeaturesOrRuled::Ruled(ct) => {
            println!("File ruled as: {:?}", ct);
        }
    }
    Ok(())
}

```

### Standalone Feature Extraction in Go

```go
package main

import (
    "fmt"
    "os"

    "github.com/google/magika/go/magika"
)

func main() {
    // Load the model configuration (the same one used for inference)
    cfg := magika.DefaultConfig // or load a custom Config

    f, _ := os.Open("example.docx")
    defer f.Close()
    stat, _ := f.Stat()

    // Extract features
    feats, err := magika.ExtractFeatures(cfg, f, int(stat.Size()))
    if err != nil {
        panic(err)
    }
    fmt.Printf("beg: %v\n", feats.Beg[:5])
    fmt.Printf("end: %v\n", feats.End[:5])
}

```

## Design Rationale and Performance

This extraction architecture provides three critical advantages for production deployment:

- **Speed** – Reading only a few kilobytes makes feature extraction orders of magnitude faster than scanning entire files, enabling high-throughput analysis.
- **Memory Efficiency** – By operating on seekable streams and pre-allocating fixed-size buffers, Magika processes multi-gigabyte files with minimal RAM usage.
- **Robustness** – The combination of whitespace stripping, configurable padding, and fixed-size output ensures the model receives valid input even for empty files, tiny text files, or malformed binaries.

## Summary

- **Magika feature extraction** converts files into fixed-size numeric vectors by sampling `block_size` windows from file boundaries.
- Core functions `_extract_features_from_seekable` (Python), `extract_features_async` (Rust), and `ExtractFeatures` (Go) implement identical logic for cross-language consistency.
- The pipeline strips whitespace using language-specific trim functions, slices content to `beg_size` and `end_size`, and pads with `padding_token` to maintain exact dimensions.
- This seekable-stream approach never loads full files into memory, supporting efficient content type detection on files of any size.
- Configuration parameters controlling extraction reside in [`rust/lib/src/config.rs`](https://github.com/google/magika/blob/main/rust/lib/src/config.rs) and analogous structures, ensuring synchronized behavior across the Python, Rust, and Go implementations.

## Frequently Asked Questions

### Does Magika read the entire file into memory to extract features?

No. Magika deliberately reads at most `block_size` bytes (typically 4 KB) from the beginning and end of the file using seek operations. This design, implemented in `_extract_features_from_seekable` and `extract_features_async`, ensures that multi-gigabyte files can be analyzed with constant memory usage.

### What determines the size of the feature vector in Magika?

The feature vector length is determined by the sum of `beg_size`, `end_size`, and optionally `mid_size` parameters defined in the model configuration files (such as those in [`rust/lib/src/config.rs`](https://github.com/google/magika/blob/main/rust/lib/src/config.rs)). The system pads shorter inputs with `padding_token` (default `0`) to ensure every file produces a vector of exactly `features_size` length required by the neural network.

### Why does Magika strip whitespace from file content?

Whitespace stripping (performed via `strip_prefix`/`strip_suffix` in Rust, `bytes.TrimLeft`/`TrimRight` in Go, and equivalent Python logic) normalizes input data so that insignificant padding bytes do not affect model predictions. This ensures that files with varying amounts of leading or trailing whitespace generate identical feature vectors.

### Is the feature extraction consistent across Python, Rust, and Go?

Yes. While the I/O APIs differ—Python uses synchronous `_extract_features_from_seekable`, Rust uses async `extract_features_async`, and Go uses synchronous `ExtractFeatures`—all implementations follow identical algorithms for window selection, whitespace removal, and padding. The project uses golden test files to verify that all three languages produce bitwise-identical feature vectors for the same inputs.