# How Magika Avoids Loading Entire Large Files Into Memory: Incremental Feature Extraction

> Discover how Magika uses incremental feature extraction to process large files efficiently without loading them entirely into memory. Learn about its seekable abstraction for scalable file analysis.

- Repository: [Google/magika](https://github.com/google/magika)
- Tags: internals
- Published: 2026-04-16

---

**Magika processes files incrementally by reading only small fixed-size blocks from the beginning and end of files using a `Seekable` abstraction, ensuring large files never consume excessive RAM.**

The Google Magika library identifies file content types using deep learning, yet it must remain memory-efficient when processing multi-gigabyte archives or logs. Instead of slurping entire files into Python memory, Magika extracts only the minimal features necessary for model inference. This article examines the exact source code mechanisms—from the `Seekable` wrapper in [`python/src/magika/types/seekable.py`](https://github.com/google/magika/blob/main/python/src/magika/types/seekable.py) to the feature extraction logic in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py)—that enable bounded memory usage regardless of file size.

## The Memory Challenge in File Type Detection

Traditional file inspection tools often read complete file contents to calculate hashes, inspect magic numbers, or extract metadata. For large binary datasets or sequentially accessed streams, this approach exhausts available RAM and increases latency. Magika solves this by treating file access as a **random-access sampling problem** rather than a bulk loading operation.

## The Seekable Abstraction: Random Access Without Full Reads

At the heart of Magika's memory efficiency lies the `Seekable` class, which wraps any binary stream and exposes a slice-based reading interface.

### Implementing Offset-Based Reads

Located in [`python/src/magika/types/seekable.py`](https://github.com/google/magika/blob/main/python/src/magika/types/seekable.py), the `Seekable` class provides the `read_at` method:

```python
def read_at(self, offset: int, size: int) -> bytes:
    self._stream.seek(offset)
    return self._stream.read(size)

```

This method guarantees that exactly `size` bytes are read starting at `offset`, never the entire file. The caller defines precise windows—typically 2048 bytes—ensuring RAM usage remains constant regardless of file dimensions.

## Feature Extraction Without Bulk Loading

Magika's model requires only two small integer vectors representing the start and end of a file. The `_extract_features_from_seekable` function in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py) orchestrates this partial reading strategy.

### Reading Fixed Windows from Start and End

The extraction routine calculates how many bytes to read based on the `block_size` parameter (default 2048 bytes) and the file's total size:

```python
bytes_num_to_read = min(block_size, seekable.size)

# Read from beginning

beg_bytes = seekable.read_at(0, bytes_num_to_read)

# Read from end

end_bytes = seekable.read_at(seekable.size - bytes_num_to_read, bytes_num_to_read)

```

After reading, the function strips whitespace, pads the sequences to fixed lengths (`beg_size` and `end_size` from the model configuration), and converts them to integer arrays. **The deep learning model never sees the full file contents**—only these small, fixed-size feature vectors.

### Early Exit for Small Files

Magika includes shortcuts for files below the `min_file_size_for_dl` threshold. In `_get_result_or_features_from_seekable`, if the file is smaller than this limit, Magika reads the entire file (which is inexpensive for small sizes) and may bypass the neural model entirely. For empty files, it returns an instant result without any reading.

## Step-by-Step Flow for Large File Processing

When processing a multi-gigabyte file, Magika executes the following memory-bound workflow:

1. **Wrap the stream** in a `Seekable` object, which knows the total size but does not buffer content.
2. **Check size against thresholds** (`min_file_size_for_dl`) to determine if feature extraction is needed.
3. **Calculate read windows** based on `block_size` (typically 2048 bytes) and file size.
4. **Execute two precise reads** via `seekable.read_at()`—one at offset 0 for the start, one at `size - block_size` for the end.
5. **Process features** by stripping whitespace, padding to model input dimensions, and converting to integer tensors.
6. **Run inference** on the fixed-size feature vectors, producing a content type prediction without ever loading the full file.

This design ensures RAM usage remains roughly **4 KB per file** (two blocks of 2048 bytes) regardless of whether the file is 10 KB or 10 GB.

## Configuration Parameters Controlling Memory Usage

Magika's memory bounds are defined in [`python/src/magika/config/content_types_kb.min.json`](https://github.com/google/magika/blob/main/python/src/magika/config/content_types_kb.min.json) and exposed through `ModelConfig` in [`python/src/magika/types/model.py`](https://github.com/google/magika/blob/main/python/src/magika/types/model.py):

- **`block_size`**: The maximum bytes read from disk (default 2048).
- **`beg_size`**: The fixed input dimension for the "beginning" feature vector.
- **`end_size`**: The fixed input dimension for the "end" feature vector.
- **`min_file_size_for_dl`**: Threshold below which the entire file is read (and considered small enough to not warrant incremental processing).

These constants ensure the `_extract_features_from_seekable` function operates within predictable memory constraints.

## Practical Code Examples

### Identify a Large File Without Loading It

```python
from magika import Magika

mag = Magika()
result = mag.identify_path("/path/to/4gb.iso")
print(result.prediction.output.label)  # e.g., ContentTypeLabel.ISO

```

Behind the scenes, this reads only 4096 bytes total (2048 from start, 2048 from end).

### Identify Content from an HTTP Stream

```python
import urllib.request
from magika import Magika

resp = urllib.request.urlopen("https://example.com/large-dataset.bin")
stream = resp.raw  # A BufferedReader

result = Magika().identify_stream(stream)
print(result.prediction.output.label)

```

The `Seekable` wrapper handles the stream without buffering the entire HTTP response.

### Process Raw Bytes (Already in Memory)

```python
data = b"#!/usr/bin/env python\nprint('hello')"
result = mag.identify_bytes(data)
print(result.prediction.output.label)  # ContentTypeLabel.PYTHON

```

For bytes already loaded, Magika skips the `Seekable` wrapper and processes the buffer directly.

## Summary

- **Incremental reading**: Magika uses the `Seekable` abstraction in [`python/src/magika/types/seekable.py`](https://github.com/google/magika/blob/main/python/src/magika/types/seekable.py) to read precise byte ranges without loading full files.
- **Bounded feature extraction**: The `_extract_features_from_seekable` function in [`python/src/magika/magika.py`](https://github.com/google/magika/blob/main/python/src/magika/magika.py) reads at most `block_size` bytes (default 2048) from the start and end of any file.
- **Constant memory usage**: Whether processing a 10 KB text file or a 100 GB binary, Magika's RAM usage stays roughly 4 KB per file during feature extraction.
- **Smart shortcuts**: Files smaller than `min_file_size_for_dl` are read entirely (inexpensive), while empty files return instantly without I/O.

## Frequently Asked Questions

### How much memory does Magika use when analyzing a 10 GB file?

Magika uses approximately 4 KB of memory to analyze a 10 GB file. It reads exactly 2048 bytes from the beginning and 2048 bytes from the end using the `Seekable.read_at` method, ignoring the remaining gigabytes of content. The deep learning model only processes these small feature vectors, not the raw file data.

### What is the `Seekable` class in Magika?

The `Seekable` class is a lightweight wrapper defined in [`python/src/magika/types/seekable.py`](https://github.com/google/magika/blob/main/python/src/magika/types/seekable.py) that provides random-access reading capabilities for any binary stream. It exposes a `read_at(offset, size)` method that seeks to a specific byte position and returns exactly the requested number of bytes, enabling Magika to sample specific file regions without buffering the entire stream.

### Does Magika ever read the entire file contents?

Yes, but only for small files. When a file's size is less than `min_file_size_for_dl` (as defined in the model configuration), Magika reads the entire file because the memory cost is negligible and allows for direct content analysis. For empty files, Magika returns a result immediately without reading any bytes. For all larger files, it strictly uses partial reads.

### Can I use Magika with streaming data or HTTP responses?

Yes. Magika's `identify_stream` method accepts any file-like object (such as `BufferedReader` from HTTP responses). The method wraps the stream in a `Seekable` object, allowing the same incremental reading strategy to work on network streams without downloading the entire content first. This makes Magika suitable for analyzing large remote files on the fly.