# How LlamaFactory Handles Large Datasets Using Streaming Mode: A Complete Technical Guide

> Discover how LlamaFactory handles large datasets with streaming mode. Learn to train on massive corpora without loading everything into RAM. A complete technical guide.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory handles large datasets using streaming mode by switching to HuggingFace's iterable datasets, which lazily load samples one at a time with a configurable shuffle buffer, enabling training on infinite or terabyte-scale corpora without materializing the full collection in RAM.**

When training large language models on massive corpora, memory constraints often prevent loading entire datasets into system RAM. LlamaFactory solves this through its streaming mode implementation, which transforms the data pipeline to process samples on-the-fly. This article examines the source code architecture behind how LlamaFactory handles large datasets using streaming mode, from argument validation in `DataArguments` to the lazy iteration mechanisms in `DataEngine`.

## What Is Streaming Mode in LlamaFactory?

Streaming mode in LlamaFactory is a data loading strategy that treats datasets as unbounded streams rather than finite indexed collections. When activated via the `streaming` flag, the library delegates to HuggingFace's `datasets` library with `streaming=True`, returning `IterableDataset` objects that yield samples sequentially from disk or network storage.

### Key Benefits for Large-Scale Training

- **Constant Memory Usage**: Only the shuffle buffer (default 16,384 samples) and current batch reside in memory, regardless of total dataset size.
- **Infinite Dataset Support**: Enables training on continuous data streams without predefined boundaries.
- **Zero-Copy Indexing**: Avoids materializing full sample lists through iterator-based access patterns.

## Core Architecture: How LlamaFactory Implements Streaming

The streaming implementation spans several components, with `DataEngine` in [`src/llamafactory/v1/core/data_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/data_engine.py) serving as the central coordinator.

### Step 1: Activating Streaming via DataArguments

Streaming is controlled through the `DataArguments` class in [`src/llamafactory/hparams/data_args.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/data_args.py). When `streaming=True` is passed via CLI (`--streaming`) or config, the `__post_init__` method validates compatibility by forbidding `max_samples` and requiring integer `val_size`:

```python

# From src/llamafactory/hparams/data_args.py

def __post_init__(self):
    if self.streaming:
        require_version("datasets>=2.0.0", "To fix: pip install datasets>=2.0.0")
        if self.max_samples is not None:
            raise ValueError("`max_samples` is incompatible with streaming mode.")
        if self.val_size is not None and not isinstance(self.val_size, int):
            raise ValueError("`val_size` must be an integer in streaming mode.")

```

### Step 2: Loading Iterable Datasets

The `DataEngine._load_dataset()` method detects streaming requirements and calls HuggingFace's `load_dataset()` with `streaming=True`. For local file sources, the `DataLoaderPlugin` in [`src/llamafactory/v1/plugins/data_plugins/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/plugins/data_plugins/loader.py) wraps results with `to_iterable_dataset()`:

```python

# From src/llamafactory/v1/core/data_engine.py

def _load_dataset(self, dataset_info: "DatasetInfo", model_name: str):
    # ...

    dataset = load_dataset(
        dataset_info.path,
        split=dataset_info.split,
        streaming=self.streaming,  # True when streaming mode active

        # ...

    )

```

### Step 3: Handling Indexing and Length Constraints

Because iterable datasets cannot support random access, `DataEngine` implements guardrails in [`src/llamafactory/v1/core/data_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/data_engine.py). The `__len__` method returns `-1` to signal that the dataset has unknown or infinite length, while `__getitem__` raises an explicit error:

```python

# From src/llamafactory/v1/core/data_engine.py

def __len__(self):
    if self.streaming:
        return -1  # Indicates infinite or unknown length

    return len(self.data_index)

def __getitem__(self, idx):
    if self.streaming:
        raise IndexError("Cannot index a streaming dataset.")
    # ... regular indexing logic

```

To maintain API compatibility with components expecting indexable structures, `_build_data_index()` creates a placeholder list of 1000 arbitrary entries when streaming is active, satisfying interface requirements without materializing actual data.

## Practical Examples: Configuring LlamaFactory Streaming Mode

### Command-Line Interface

Enable streaming mode when launching training via `llamafactory-cli`:

```bash
llamafactory-cli train \
  --dataset_dir data \
  --dataset large_corpus.yaml \
  --streaming true \
  --buffer_size 32768 \
  --output_dir output/streaming_model

```

The `--buffer_size` parameter controls the shuffle buffer for the iterable dataset (default 16,384).

### YAML Dataset Configuration

Define streaming behavior in your dataset configuration file:

```yaml

# data/large_corpus.yaml

default:
  path: "bigscience/P3"
  split: "train"
  streaming: true        # Critical flag for large datasets

  buffer_size: 65536

```

Setting `streaming: true` in the YAML triggers the iterable dataset path in `DataEngine._load_dataset()`.

### Programmatic Usage with DataEngine

For custom scripts using LlamaFactory's internal APIs:

```python
from llamafactory.v1.core.data_engine import DataEngine
from llamafactory.hparams.data_args import DataArguments

# Configure arguments with streaming enabled

data_args = DataArguments(
    dataset="bigscience/P3",
    streaming=True,
    buffer_size=65536,
    val_size=1000  # Must be integer in streaming mode

)

# Initialize engine

engine = DataEngine(
    dataset_path="bigscience/P3",
    data_args=data_args
)

# Iterate through samples (memory-efficient streaming)

for i, sample in enumerate(engine):
    if i >= 1000:
        break
    print(f"Sample {i}: {sample['text'][:50]}...")

```

## Key Source Files and Implementation Details

The streaming functionality is distributed across these critical files:

| File | Responsibility |
|------|----------------|
| [`src/llamafactory/hparams/data_args.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/data_args.py) | Defines `DataArguments.streaming` and validates incompatible options (forbids `max_samples`, requires integer `val_size`). |
| [`src/llamafactory/v1/core/data_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/data_engine.py) | Central `DataEngine` class – detects streaming, loads datasets, builds placeholder index, guards length/indexing. |
| [`src/llamafactory/v1/plugins/data_plugins/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/plugins/data_plugins/loader.py) | `DataLoaderPlugin` that loads local files; adds `to_iterable_dataset()` when `streaming=True`. |
| [`src/llamafactory/data/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/data/loader.py) | High-level helper functions that invoke `load_dataset(..., streaming=True)`. |
| [`src/llamafactory/v1/utils/types.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/utils/types.py) | Type definitions including `DatasetInfo` with the `streaming` boolean field. |

These files together enable LlamaFactory to treat massive datasets as **streams**, keeping memory usage low while still exposing a familiar dataset interface to downstream training and inference components.

## Summary

- **LlamaFactory streaming mode** enables training on datasets larger than system memory by switching to HuggingFace's iterable datasets.
- Activation occurs via `DataArguments.streaming=True` (CLI `--streaming`), which triggers validation against incompatible options like `max_samples`.
- The `DataEngine` class in [`src/llamafactory/v1/core/data_engine.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/core/data_engine.py) manages streaming state, returning `-1` for `__len__` and raising errors for random indexing attempts.
- Local file sources receive special handling through `DataLoaderPlugin.to_iterable_dataset()` to ensure uniform streaming behavior across hub and local data.
- Users control memory usage through the `buffer_size` parameter, which configures the shuffle buffer for the iterable dataset (default 16,384).

## Frequently Asked Questions

### What is the maximum dataset size supported in LlamaFactory streaming mode?

There is no fixed maximum. LlamaFactory streaming mode supports **infinite or unbounded datasets** because samples are loaded lazily from disk or network. The only memory constraints come from the shuffle buffer (default 16,384 samples) and the model parameters, not the total dataset size. This architecture enables training on terabyte-scale corpora or continuous data streams.

### Can I use max_samples with streaming mode enabled?

No. The `DataArguments` validator in [`src/llamafactory/hparams/data_args.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/hparams/data_args.py) explicitly raises a `ValueError` if `max_samples` is set while `streaming=True`. This prevents confusion because streaming datasets cannot be randomly sampled or truncated at load time using standard indexing operations. To limit training steps, use the trainer's `max_steps` argument instead.

### How does shuffling work when using LlamaFactory streaming mode?

Shuffling occurs within a fixed-size buffer rather than across the entire dataset. The `buffer_size` parameter (default 16,384) controls how many samples are held in memory for shuffling. As the iterator advances, new samples fill the buffer and random selection occurs within this window, providing approximate global shuffling for large streams while maintaining constant memory usage.

### Why does len() return -1 when streaming is enabled?

The `DataEngine.__len__` method returns `-1` to signal that the dataset has **unknown or infinite length**. This follows Python conventions for unbounded iterables and prevents downstream components from attempting epoch-based training that requires knowing the total step count in advance. When using LlamaFactory streaming mode, configure training with `max_steps` rather than `num_train_epochs`.