How to Optimize Prefect Flow Performance with Result Storage Tier Resolution

You can optimize Prefect flow performance by configuring the result storage tier resolution to prefer local filesystem storage for low-latency access, while using remote blocks only when necessary for persistence or sharing.

Prefect stores flow and task outputs in result storage, selecting the optimal backend through a tier-resolution process that balances speed and durability. This article explains how to leverage the PrefectHQ/prefect source code to minimize storage latency and eliminate unnecessary network overhead.

Understanding Result Storage Tier Resolution

The system evaluates three configuration layers to determine where results are written. According to the source code in src/prefect/results.py, the resolver checks these tiers in strict order of precedence:

1. Explicit flow- or task-level setting – A result_storage argument passed directly to a flow or task decorator.
2. Server-wide default – A block ID configured in the Prefect server settings.
3. Local fallback – A LocalFileSystem block rooted at settings.results.local_storage_path (defaults to $PREFECT_HOME/storage).

This hierarchy ensures that developers can override behavior locally, administrators can set organization-wide defaults, and flows always have a functional fallback.

How Tier Resolution Works

The resolution logic is implemented in prefect/results.py by the _aget_default_result_storage and _get_default_result_storage functions.

Resolution Logic

The functions first examine settings.results.default_storage_block. If a block slug exists, the resolver uses that block (_DefaultResultStorageSource.SETTINGS). If no setting is present, it queries the server for a default block ID (_DefaultResultStorageSource.SERVER). Otherwise, it instantiates a local filesystem block (_DefaultResultStorageSource.LOCAL_STORAGE_PATH).

# prefect/results.py (lines 29-46)

default_block = settings.results.default_storage_block
basepath = settings.results.local_storage_path
server_default_block_id = (
    None if default_block is not None else await _aread_server_default_result_storage_block_id()
)

Caching Mechanism

To avoid repeated lookups, the resolver caches the computed WritableFileSystem in a module-level dictionary named _default_storages. Reusing the same storage block across multiple tasks eliminates redundant block initialization and API calls.

Performance Impact by Storage Tier

Each tier introduces different latency characteristics that directly affect flow runtime:

• Local (filesystem): Ideal for small-to-medium workloads and short-lived flows. Typical latency ranges from microseconds to milliseconds (disk I/O).
• Remote block (S3, Azure Blob, GCS): Necessary for large payloads or cross-region sharing, but introduces network overhead of approximately 10-100ms per operation.
• Server-wide default: Provides centralized governance but requires an extra API call to fetch the block ID, adding latency before the first result write.

When flows generate many intermediate results, remote write operations accumulate significant overhead. Configuring local storage as the default tier minimizes this latency for the common case.

Optimization Strategies

Set a Local Default Storage Block

Configure the PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK environment variable or use the CLI to specify a local filesystem block. This ensures the resolver selects the fastest tier without code changes.

prefect config set results.default_storage_block="local-file-system/my-results"

Disable Automatic Persistence

When results do not need to persist beyond the current run, disable persistence entirely. The persist_by_default flag is defined in src/prefect/settings/models/results.py (line 27).

from prefect import flow

@flow(persist_result=False)
def my_flow():
    # Results remain in memory only

    pass

Setting persist_result=False skips the resolve-and-write steps, eliminating storage latency.

Inherit Storage Configuration

Avoid specifying result_storage on individual tasks unless necessary. Let tasks inherit the flow's storage configuration to prevent redundant tier-resolution calls for each task invocation.

Verify Active Configuration

Use the CLI to inspect which tier is currently active. The implementation resides in src/prefect/cli/result_storage.py (lines 27-39).

prefect result-storage inspect --output json

This command reveals whether the system uses a remote block ID or the local fallback.

Implementation Examples

The following examples demonstrate practical configurations for optimizing performance.

Configure Local Default Storage

Register a local filesystem block and set it as the default:

import subprocess

# Register the block (run once)

subprocess.run([
    "prefect", "result-storage", "set", 
    "--id", "00000000-1111-2222-3333-444444444444"
], check=True)

Define a Flow with Implicit Local Storage

Flows without explicit result_storage arguments automatically use the configured local tier:

from prefect import flow, task

@task
def heavy_computation(x):
    return x * x

@flow
def fast_flow():
    # Uses local tier via resolver

    return heavy_computation(42)

fast_flow()

Disable Persistence for Transient Workflows

For workflows where intermediate results are disposable:

from prefect import flow

@flow(persist_result=False)
def transient_flow():
    # Skips result storage entirely

    pass

transient_flow()

Key Source Files

Understanding these components helps diagnose performance issues:

File	Role
src/prefect/results.py	Implements tier-resolution logic (_aget_default_result_storage, caching).
src/prefect/settings/models/results.py	Defines configuration settings (default_storage_block, local_storage_path, persist_by_default).
src/prefect/cli/result_storage.py	Provides CLI commands to inspect and configure default storage.
src/prefect/flows.py	Flow constructor that forwards result_storage to the resolver.
src/prefect/tasks.py	Task constructor that inherits flow storage unless overridden.

Summary

• Tier resolution follows a strict order: explicit settings, server defaults, then local filesystem fallback.
• Local storage provides the lowest latency (microseconds to milliseconds) compared to remote blocks (10-100ms network overhead).
• Disable persistence with persist_result=False when results are transient to eliminate storage operations entirely.
• Cache reuse is automatic via _default_storages, but minimizing per-task overrides prevents unnecessary resolution calls.
• Configuration verification via prefect result-storage inspect ensures your optimization settings are active.

Frequently Asked Questions

What is result storage tier resolution in Prefect?

Result storage tier resolution is the process by which Prefect determines where to persist flow and task outputs. It evaluates three tiers in order: explicit flow/task settings, server-wide defaults, and local filesystem fallback. This mechanism ensures optimal backend selection based on configuration and availability.

How do I configure local result storage for better performance?

Set the PREFECT_RESULTS_DEFAULT_STORAGE_BLOCK environment variable or run prefect config set results.default_storage_block="local-file-system/my-block". This directs the resolver in src/prefect/results.py to use local disk I/O instead of remote network calls, reducing latency from milliseconds to microseconds.

When should I use remote storage instead of local storage?

Use remote storage blocks (S3, Azure Blob, GCS) when you need cross-region result sharing, durability beyond a single worker node, or when processing large payloads that exceed local disk capacity. Remote storage adds 10-100ms of network latency per write, making it unsuitable for high-frequency intermediate results in performance-critical paths.

Does disabling result persistence improve flow performance?

Yes. Setting persist_result=False on a flow or task completely skips the tier-resolution and write operations. According to src/prefect/settings/models/results.py, this overrides the persist_by_default setting and keeps results in memory only, eliminating disk I/O or network overhead for transient computations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}