# What Persistence Backends Does Pathway Support for State Recovery and Fault Tolerance?

> Discover Pathway's state recovery and fault tolerance options. Explore support for Filesystem, Amazon S3, Azure Blob Storage, and a Mock backend for robust applications.

- Repository: [Pathway/pathway](https://github.com/pathwaycom/pathway)
- Tags: api-reference
- Published: 2026-03-06

---

**Pathway supports four persistence backends for fault-tolerant state recovery: local Filesystem, Amazon S3 (and MinIO-compatible services), Azure Blob Storage, and an in-memory Mock backend for testing.**

Pathway, the open-source stream processing framework maintained by `pathwaycom/pathway`, provides a configurable checkpointing layer that stores **operator state**, **frontiers**, and **metadata** in a pluggable key-value storage layer. This architecture enables exactly-once semantics and automatic recovery from failures across both single-node and distributed deployments.

## The Four Persistence Backends

The persistence layer in `src/persistence/backends/` implements four concrete storage backends, all adhering to the `PersistenceBackend` trait defined in [`src/persistence/backends/mod.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/mod.rs).

### Filesystem Backend

The **Filesystem** backend stores snapshots and metadata in a local directory, making it suitable for development, single-node runs, or environments with shared network drives. Internally, Pathway uses `FilesystemKVStorage` (defined in [`src/persistence/backends/file.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/file.rs)) to map key-value operations to files on disk. This backend requires no external dependencies and provides the lowest latency for local recovery.

### Amazon S3 and Compatible Storage

The **S3** backend writes snapshots as objects to Amazon S3 or compatible services like MinIO. Implemented in [`src/persistence/backends/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs) via `S3KVStorage`, this backend guarantees durability across clusters and supports multi-node deployments. Pathway organizes snapshot data under a configurable prefix within the bucket, separating metadata from per-operator state files.

### Azure Blob Storage

For Azure-native deployments, Pathway provides the **Azure** backend (`AzureKVStorage` in [`src/persistence/backends/azure.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/azure.rs)). This backend offers identical semantics to S3 but targets Azure Blob Storage accounts, integrating with Azure's identity and networking models for enterprise cloud environments.

### Mock Backend for Testing

The **Mock** backend provides an in-memory stub (`MockKVStorage` in [`src/persistence/backends/mock.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/mock.rs)) used exclusively for unit tests or when users explicitly require a no-op persistence layer. This backend does not survive process restarts but eliminates I/O overhead during testing.

## Technical Architecture: The PersistenceBackend Trait

All four backends implement the `PersistenceBackend` trait, which defines the core key-value operations required by the checkpointing system: `list_keys`, `get_value`, `put_value`, and `remove_key`.

The enum `PersistentStorageConfig` (defined in [`src/persistence/config.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/config.rs) at lines 46-59) selects the concrete implementation at runtime. When a pipeline starts, Pathway creates a `PersistenceManagerOuterConfig` containing the chosen backend, snapshot interval, and access flags. Internally, `PersistenceManagerConfig` transforms this configuration into three distinct storage objects:

- **`create_metadata_storage`** – A single KV store per worker for pipeline metadata.
- **`create_snapshot_writer` and `create_snapshot_readers`** – Per-operator KV stores for state and offset snapshots.
- **`create_cached_object_storage`** – An optional local cache for large objects when using remote backends.

Pathway automatically adjusts the **snapshot merging interval** based on whether the backend is local or remote. Remote backends trigger less frequent merging through `adjusted_snapshot_merging_interval` to minimize API call costs and latency.

## Configuring Persistence in Python

Pathway exposes backend selection through `pw.persistence.Backend` methods in the Python API. The bridge function `construct_persistent_storage_config` in [`src/python_api.rs`](https://github.com/pathwaycom/pathway/blob/main/src/python_api.rs) maps string identifiers (`"fs"`, `"s3"`, `"azure"`, `"mock"`) to the appropriate `PersistentStorageConfig` variants.

### Filesystem Configuration

```python
import pathway as pw

backend = pw.persistence.Backend.filesystem("./Pathway-Cache")

config = pw.PersistenceConfig(
    snapshot_interval=60000,  # 60 seconds

    backend=backend,
    snapshot_access=pw.persistence.SnapshotAccess.All,
    persistence_mode=pw.persistence.PersistenceMode.OperatorPersisting,
)

```

### S3 or MinIO Configuration

```python
import pathway as pw
from pathway.io.s3 import AwsS3Settings

s3_settings = AwsS3Settings(
    bucket_name="my-bucket",
    region="eu-central-1",
    access_key="****",
    secret_access_key="****",
)

backend = pw.persistence.Backend.s3(s3_settings, "pipeline-state/")
config = pw.PersistenceConfig(
    snapshot_interval=120000,
    backend=backend,
    snapshot_access=pw.persistence.SnapshotAccess.ReadWrite,
    persistence_mode=pw.persistence.PersistenceMode.OperatorPersisting,
)

```

## Configuring Persistence in Rust

When using Pathway's Rust API directly, instantiate `PersistentStorageConfig` variants and wrap them in `PersistenceManagerOuterConfig`:

```rust
use pathway_engine::persistence::config::{
    PersistentStorageConfig, PersistenceManagerOuterConfig, SnapshotAccess, PersistenceMode
};
use std::path::PathBuf;
use std::time::Duration;

let fs_cfg = PersistentStorageConfig::Filesystem(PathBuf::from("/tmp/pw-state"));
let outer = PersistenceManagerOuterConfig::new(
    Duration::from_secs(60),  // snapshot interval
    fs_cfg,
    SnapshotAccess::ReadWrite,
    PersistenceMode::OperatorPersisting,
    true,   // enable compression
    false,  // disable async snapshots
    Duration::from_secs(30),  // merging interval
);

```

## Summary

- Pathway provides **four persistence backends**: Filesystem, Amazon S3 (MinIO-compatible), Azure Blob Storage, and Mock.
- All backends implement the `PersistenceBackend` trait with standard KV operations (`list_keys`, `get_value`, `put_value`, `remove_key`).
- The `PersistentStorageConfig` enum in [`src/persistence/config.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/config.rs) controls runtime backend selection.
- Python users configure backends via `pw.persistence.Backend.filesystem()`, `Backend.s3()`, or `Backend.azure()`.
- Remote backends automatically trigger less frequent snapshot merging to optimize API costs.
- Core implementations reside in [`src/persistence/backends/file.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/file.rs), [`s3.rs`](https://github.com/pathwaycom/pathway/blob/main/s3.rs), [`azure.rs`](https://github.com/pathwaycom/pathway/blob/main/azure.rs), and [`mock.rs`](https://github.com/pathwaycom/pathway/blob/main/mock.rs).

## Frequently Asked Questions

### How do I choose between filesystem and cloud storage for Pathway persistence?

Use the **Filesystem** backend for local development, single-node deployments, or when operating in air-gapped environments with shared network storage. Choose **S3** or **Azure** for distributed clusters requiring durability across machine failures or when running in Kubernetes across multiple availability zones. Cloud backends ensure state survives node termination but introduce network latency during snapshot operations.

### Can I use MinIO or other S3-compatible storage with Pathway?

Yes. The S3 backend uses standard S3 API calls implemented in [`src/persistence/backends/s3.rs`](https://github.com/pathwaycom/pathway/blob/main/src/persistence/backends/s3.rs), making it compatible with MinIO, Ceph, and other S3-compatible object stores. Configure the endpoint URL and credentials through `AwsS3Settings` when initializing `pw.persistence.Backend.s3()`.

### What is the Mock backend used for in Pathway?

The **Mock** backend provides an in-memory hash map implementation of `PersistenceBackend` for unit testing and CI pipelines. It eliminates disk I/O and cloud dependencies, allowing tests to verify checkpointing logic without external infrastructure. Never use Mock in production, as all state disappears when the process exits.

### How does Pathway handle snapshot intervals with different persistence backends?

Pathway adjusts the snapshot merging behavior based on backend latency. For remote backends like S3 and Azure, the system increases the `adjusted_snapshot_merging_interval` to reduce the number of ListObjects and PutObject API calls, lowering costs and preventing throttling. Local filesystem backends use more aggressive merging intervals since disk I/O is cheaper and faster.