# How Prefect Caching Works: Cache Policies and Cache Key Functions Explained

> Understand Prefect caching with policies and key functions. Prefect uses context to create cache keys for idempotent task execution, reusing prior results.

- Repository: [Prefect/prefect](https://github.com/PrefectHQ/prefect)
- Tags: deep-dive
- Published: 2026-07-13

---

**Prefect's caching mechanism uses composable cache policies that convert task runtime context into deterministic cache keys, enabling idempotent execution by short-circuiting tasks when matching prior results exist.**

Prefect's caching system centers on policy-driven architecture that determines when task results should be reused across flow runs. By configuring cache policies and custom cache key functions in the PrefectHQ/prefect repository, developers gain precise control over result persistence, storage backends, and computational efficiency. The implementation spans [`src/prefect/cache_policies.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/cache_policies.py) and [`src/prefect/tasks.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/tasks.py), providing both high-level decorators and low-level key computation hooks.

## How Cache Policies Determine the Effective Caching Strategy

When a task is instantiated, Prefect resolves the effective caching strategy through a specific precedence order defined in `Task.__init__`.

### Task Constructor Resolution Logic

In [`src/prefect/tasks.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/tasks.py) (lines 511-525), the constructor inspects both `cache_policy` and `cache_key_fn` parameters. If a user supplies a custom `cache_key_fn`, it takes precedence and is automatically wrapped in a `CacheKeyFnPolicy` via `CachePolicy.from_cache_key_fn`. This policy object standardizes the interface between custom functions and the broader policy ecosystem.

If no explicit policy is provided and result persistence is enabled, Prefect falls back to the `DEFAULT` policy (lines 55-57). However, if `persist_result=False` is set, the policy is forced to `NO_CACHE` (a `_None` policy) regardless of other settings, and a warning is emitted to indicate the conflict (lines 49-55).

## The Default Cache Policy Composition

The `DEFAULT` cache policy is not a monolithic object but a composed strategy defined in [`src/prefect/cache_policies.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/cache_policies.py) (lines 12-13) as:

```python
DEFAULT = INPUTS + TASK_SOURCE + RUN_ID

```

The `+` operator creates a `CompoundCachePolicy`, which validates that component policies have compatible `storage`, `isolation`, and `lock` settings before merging them (lines 20-55). This validation ensures that combined policies can safely share cache storage without configuration conflicts.

## Built-in Cache Policy Types and Their Hashing Behavior

Individual policies contribute specific deterministic components to the final cache key through their `compute_key` implementations.

### Inputs Policy

The `Inputs` policy hashes the runtime inputs passed to the task. It supports excluding specific keys via the `exclude` parameter and applies stable transforms for known mutable types like pandas DataFrames to ensure deterministic hashing. If inputs are empty or unhashable, it returns `None`. The implementation resides in `Inputs.compute_key` (lines 58-86).

### TaskSource Policy

`TaskSource` ensures cache invalidation when task logic changes by hashing the function's source code. It captures the source at task creation time using `inspect.getsource` and handles the hashing in `TaskSource.compute_key` (lines 88-106).

### RunId Policy

The `RunId` policy scopes cache keys to specific execution contexts by returning the current flow-run ID, or the task-run ID if no flow context exists. This prevents cache hits across unrelated flow runs. See `RunId.compute_key` (lines 108-122).

### CacheKeyFnPolicy

When users provide a custom `cache_key_fn`, Prefect wraps it in a `CacheKeyFnPolicy`. This policy calls the user function with the `TaskRunContext` and raw inputs dictionary, expecting a string or `None` return value. The implementation is in `CacheKeyFnPolicy.compute_key` (lines 70-78).

### NO_CACHE Policy

The `_None` policy (exposed as `NO_CACHE`) always yields `None` from its `compute_key` method (lines 62-68), effectively disabling caching by ensuring no cache key is ever generated or matched.

### CompoundCachePolicy

When policies are combined using the `+` operator, `CompoundCachePolicy` coordinates key generation. It computes each sub-policy's key, discards any `None` values, sorts the remaining keys, and hashes the final tuple to produce a single deterministic string. This logic is implemented in `CompoundCachePolicy.compute_key` (lines 90-108).

## The Cache Key Generation Flow

During task execution setup, specifically within `Task.create_run`, Prefect invokes the resolved policy's `compute_key` method, passing the `TaskRunContext`, resolved inputs, and flow parameters. The resulting string is stored on the task run object.

The engine then queries for a previously completed task state matching this key. If a match exists, the task execution is short-circuited and the prior result is restored without re-running the task function. This lookup occurs before the task enters the `RUNNING` state, ensuring true idempotency.

## Customizing Cache Behavior

### Implementing Custom Cache Key Functions

To override policy-based hashing entirely, provide a function to the `cache_key_fn` parameter:

```python
from prefect import task, TaskRunContext

def my_key(ctx: TaskRunContext, inputs: dict) -> str | None:
    # Cache only on the first argument, ignore everything else

    return f"my-task-{inputs.get('first_arg')}"

@task(cache_key_fn=my_key)
def expensive_op(first_arg: int, noisy_arg: str):
    # `noisy_arg` will never affect caching

    return first_arg ** 2

```

The function receives the `TaskRunContext` and the raw input dictionary, allowing complete control over key generation logic.

### Excluding Volatile Inputs

For cases where specific arguments change frequently but should not invalidate the cache, use the subtraction operator or explicit exclusion:

```python
from prefect import task
from prefect.cache_policies import INPUTS

# Method 1: Subtract from existing policy

my_policy = INPUTS - "timestamp"

@task(cache_policy=my_policy)
def compute(value: int, timestamp: datetime.datetime):
    # `timestamp` changes each run → excluded from cache key

    return value * 2

```

### Storage and Locking Configuration

Policies support advanced configuration for distributed scenarios. You can specify `key_storage` (accepting block references or filesystem paths) and configure `lock_manager` or `isolation_level` parameters to guarantee atomicity when multiple workers may write the same cache key (src/prefect/cache_policies.py lines 72-82).

### Disabling Caching

To completely disable caching for a task, either set `persist_result=False` or explicitly pass `cache_policy=NO_CACHE`. When disabled, the engine ignores any cache key even if a custom `cache_key_fn` is supplied.

## Stable Transforms for Deterministic Hashing

Some Python types, notably pandas DataFrames, produce non-deterministic byte streams when serialized with `cloudpickle`. Prefect addresses this by registering deterministic transforms in `_register_stable_transforms` (src/prefect/cache_policies.py lines 33-47). These transforms ensure that identical data structures generate identical hash values across different Python processes and runs, preventing unnecessary cache misses.

## Summary

- **Policy resolution** follows a strict order: custom `cache_key_fn` takes precedence, followed by explicit `cache_policy`, with `DEFAULT` as the fallback when persistence is enabled.
- **Default composition** combines `INPUTS + TASK_SOURCE + RUN_ID` to create robust cache keys that invalidate when inputs, code, or run context change.
- **Individual policies** handle specific concerns: input hashing, source code validation, run scoping, or custom logic via user-provided functions.
- **Key generation** occurs in `Task.create_run` and drives the engine's lookup of prior completed states to enable short-circuit execution.
- **Customization** supports input exclusion, custom key functions, storage backends, and distributed locking for atomic cache writes.

## Frequently Asked Questions

### What is the difference between `cache_policy` and `cache_key_fn` in Prefect?

The `cache_policy` parameter accepts a policy object (like `INPUTS`, `TASK_SOURCE`, or `NO_CACHE`) that defines how the cache key is computed using Prefect's built-in rules. The `cache_key_fn` parameter accepts a custom Python function that receives the task context and inputs, allowing you to define the key string manually. When `cache_key_fn` is provided, it is wrapped in a `CacheKeyFnPolicy` and takes precedence over any `cache_policy` setting.

### How does Prefect handle caching when `persist_result` is set to `False`?

When `persist_result=False` is configured on a task, Prefect forces the cache policy to `NO_CACHE` (the `_None` policy) regardless of any other configuration, as implemented in [`src/prefect/tasks.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/tasks.py) (lines 49-55). The system emits a warning to indicate that the explicit `persist_result=False` setting is overriding caching behavior, ensuring no cache keys are generated or stored.

### Can I exclude specific function arguments from the cache key calculation?

Yes, you can exclude volatile or irrelevant arguments using the subtraction operator on the `INPUTS` policy (e.g., `INPUTS - "param_name"`) or by instantiating `Inputs(exclude=[...])` with a list of keys to omit. This is useful for timestamps, random seeds, or other arguments that change between runs but should not trigger cache invalidation.

### What happens when multiple cache policies are combined with the `+` operator?

Combining policies with `+` creates a `CompoundCachePolicy` that validates compatibility between storage, isolation, and lock settings before merging. During key generation, it computes keys for each component policy, filters out `None` values, sorts the remaining keys, and hashes them as a tuple to produce the final cache key. This ensures deterministic behavior while allowing flexible policy composition.