# Difference Between Prefect's Server-Based Orchestration and Task-Level State Management

> Understand the difference between Prefect's server-based orchestration and task-level state management. Learn how Prefect separates flow runs from task execution for reliable workflows.

- Repository: [Prefect/prefect](https://github.com/PrefectHQ/prefect)
- Tags: deep-dive
- Published: 2026-07-13

---

**Prefect separates durable, server-side flow orchestration handled by `FlowRunEngine` from lightweight, local task execution managed by `TaskRunEngine`, enabling reliable distributed workflows with minimal overhead for individual tasks.**

Prefect is a Python-native workflow orchestration framework that distinguishes between global flow coordination and local task execution. Understanding the difference between Prefect's server-based orchestration and task-level state management is essential for debugging failures, optimizing performance, and designing resilient data pipelines. This architectural separation ensures that flow-run state remains durable even when individual tasks fail or workers crash.

## Core Architectural Components

### Server-Based Orchestration with FlowRunEngine

The server-based orchestration layer centers on the `FlowRun` object, a durable record stored in the Prefect server (or Prefect Cloud) that represents an entire flow execution. According to the PrefectHQ/prefect source code, the `FlowRunEngine` in [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py) serves as the primary orchestrator, maintaining the **single source of truth** for flow execution state.

All state transitions flow through the server. When the engine changes state, `FlowRunEngine.set_state` calls `propose_state_sync` to communicate with the server via the `PrefectClient`, which validates and records the transition before updating the local `self.flow_run.state`. This ensures that the UI, API, and external agents always observe consistent flow status, even if the executing worker crashes.

### Task-Level State Management with TaskRunEngine

In contrast, task-level state management operates through the `TaskRunEngine` (implemented as both `SyncTaskRunEngine` and `AsyncTaskRunEngine` in [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py)). The primary object here is the `TaskRun`, which represents a single task execution.

For synchronous runs, state changes are applied directly to an in-memory `TaskRun` object created via `_create_task_run_locally`. The engine updates denormalized fields after every transition without immediate server communication, reducing latency for high-frequency task operations. Only when the task completes does the engine persist the final state to the server.

## State Transition Mechanisms

The two engines handle state persistence differently to balance durability against performance.

**Flow-run state transitions** require server validation. In [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py) (lines 335-348), the `set_state` method implements a synchronous proposal pattern: the engine submits the desired state change to the orchestration API, awaits confirmation, and then updates local objects. This strict coordination prevents race conditions and enables server-side policy enforcement.

**Task-run state transitions** prioritize speed. As implemented in [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py) (lines 56-78), `SyncTaskRunEngine.set_state` modifies the local `TaskRun` object immediately for synchronous execution, emitting `TaskRunStateChange` events via `_emit_task_run_state_change_event` defined in [`src/prefect/utilities/engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/utilities/engine.py). Async task runs use `SyncPrefectClient` or `PrefectClient` to communicate state changes, but still operate with less orchestration overhead than flow runs.

## Heartbeats, Retries, and Failure Handling

Operational resilience features differ significantly between the two layers.

### Flow-Run Heartbeats and Crash Recovery

The flow engine maintains liveness through a background heartbeat thread. The `_send_heartbeats` method in [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py) (lines 108-131) emits periodic `prefect.flow-run.heartbeat` events, allowing the server to detect stalled or crashed workers. If a flow crashes, `FlowRunEngine.handle_crash` (lines 724-739) creates a `Crashed` state on the server and finalizes the flow run, ensuring the failure is recorded durably even if the worker process terminates unexpectedly.

### Task-Level Retry Logic

Task engines do **not** emit heartbeats; they rely on the parent flow's heartbeat for liveness detection. Instead, they focus on local failure handling. The `SyncTaskRunEngine.handle_retry` method (lines 688-724 in [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py)) manages retry policies locally, transitioning tasks through `AwaitingRetry` or `Retrying` states without server round-trips for each attempt. This allows rapid retry loops for transient failures while keeping orchestration overhead minimal.

When tasks encounter unrecoverable errors, `SyncTaskRunEngine.handle_crash` (lines 445-453) translates exceptions to `Crashed` or `Failed` states locally. The task may retry based on its configured `retries` policy before the flow engine observes the final state.

## Context Propagation and Persistence

Metadata flows through distinct context objects that bridge the two layers.

The `FlowRunEngine.setup_run_context` method (lines 508-541 in [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py)) establishes a `FlowRunContext` that propagates flow-run metadata—including the run ID, tags, and parameters—to all child tasks. This context serves as the parent scope for task execution.

Within this scope, `SyncTaskRunEngine.setup_run_context` (lines 560-588 in [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py)) creates a `TaskRunContext` carrying task-specific metadata such as the task key, run ID, and result store configuration. This hierarchical context structure allows tasks to access flow-level configuration while maintaining isolated state management.

Persistence strategies reflect the separation of concerns. Flow-run state is always persisted to the server via the orchestration API (`client.set_flow_run_name`, `client.update_flow_run`), as seen in `FlowRunEngine.begin_run` (lines 265-284). Task runs, however, use a lazy persistence model: synchronous tasks create `TaskRun` objects locally via `_create_task_run_locally` (lines 33-66) and only write to the server upon completion, or remain entirely in-memory for short-lived operations.

## How the Two Layers Interact During Execution

Understanding the runtime interaction between server-based orchestration and task-level state management clarifies how Prefect balances durability with performance.

1. **Flow Initialization**: When execution begins, `FlowRunEngine.initialize_run` (lines 506-531) creates or reads a `FlowRun` record on the server, establishing the orchestration baseline.

2. **Task Invocation**: Inside the flow context, each `@task` call invokes a `TaskRunEngine`. For synchronous tasks, the engine instantiates a local `TaskRun` via `_create_task_run_locally`. For asynchronous tasks, the engine calls `client.create_task_run` before execution begins.

3. **State Aggregation**: As tasks execute, they emit `TaskRunStateChange` events. The flow engine listens for these events through the orchestration client and aggregates them into the overall flow state, updating the server's `FlowRun` record when significant transitions occur.

4. **Orchestration Decisions**: The server may issue pause, cancel, or retry commands based on global policies. These decisions propagate through the control channel to the `FlowRunEngine`, which then coordinates task-level actions such as `cancel_all_tasks`.

This architecture ensures that **server-based orchestration** provides durable, observable coordination while **task-level state management** delivers fast, low-overhead execution for individual operations.

## Practical Example: Retry Handling

Consider a flow with a fragile task that requires local retries:

```python

# example_flow.py

from prefect import flow, task

@task(retries=2, retry_delay_seconds=[1, 2])
def fragile_task(x: int) -> int:
    """A task that may fail and will be retried locally."""
    if x < 0:
        raise ValueError("negative values not allowed")
    return x * 2

@flow
def my_flow(value: int):
    """A flow that is orchestrated by the Prefect server."""
    result = fragile_task.submit(value)
    return result.wait()

if __name__ == "__main__":
    my_flow(5)   # succeeds, flow run state becomes Completed

    my_flow(-1)  # task retries → fails, flow run ends in Failed

```

When `my_flow(-1)` executes:

- The **FlowRunEngine** creates a `FlowRun` on the server via `client.create_flow_run`.
- The **SyncTaskRunEngine** creates a local `TaskRun` via `_create_task_run_locally` and begins execution.
- Upon raising `ValueError`, the engine invokes `handle_retry`, emits an `AwaitingRetry` state, and sleeps before retrying.
- After exhausting retries, `handle_exception` sets the task state to `Failed`.
- The flow engine receives the final task state, updates the `FlowRun` to `Failed`, and persists this status to the server.

This demonstrates how task-level state management handles transient failures locally while server-based orchestration records the ultimate outcome durably.

## Summary

- **Server-based orchestration** maintains the global source of truth for flow execution through `FlowRunEngine`, persisting all major state transitions to the Prefect server and emitting heartbeats for liveness detection.
- **Task-level state management** operates locally via `TaskRunEngine`, handling retries, caching, and fine-grained state transitions with minimal server communication to reduce latency.
- The separation enables high reliability (server knows final state even if workers die) while supporting fast, low-overhead task execution for synchronous workflows.
- Key files implementing this architecture include [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py) for orchestration logic and [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py) for task execution management.

## Frequently Asked Questions

### Does Prefect task-level state management require a server connection?

For synchronous task runs, task-level state management operates entirely in-memory using `_create_task_run_locally` and only persists the final state to the server when the task completes. Asynchronous tasks or tasks running in distributed environments communicate state changes via `PrefectClient`, but still handle retry logic locally before notifying the server.

### How does Prefect handle flow crashes versus task crashes?

When a flow crashes, `FlowRunEngine.handle_crash` in [`src/prefect/flow_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/flow_engine.py) creates a `Crashed` state on the server immediately, ensuring durability even if the worker process terminates. Task crashes are handled by `SyncTaskRunEngine.handle_crash` in [`src/prefect/task_engine.py`](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/task_engine.py), which translates exceptions to terminal states locally and may trigger retry loops before the flow engine observes the failure.

### Why don't Prefect tasks emit heartbeats like flows do?

Tasks rely on the parent flow's heartbeat for liveness detection to reduce network overhead. While the flow engine runs `_send_heartbeats` as a background thread to emit `prefect.flow-run.heartbeat` events, individual tasks only emit `TaskRunStateChange` events when transitioning between states, making task execution lighter-weight and more suitable for high-frequency operations.

### Can task retries occur without the server knowing?

Yes. The `SyncTaskRunEngine.handle_retry` method manages retry delays and state transitions (such as `AwaitingRetry`) locally for synchronous tasks. The server is only notified of the final terminal state (Completed, Failed, or Crashed) after all retry attempts are exhausted, allowing rapid retry loops without orchestration latency.