# What Is Deterministic Replay in Duroxide and Why pg_durable Orchestrations Require Deterministic Code

> Understand deterministic replay in Duroxide and why pg_durable orchestrations need deterministic code for crash recovery and exact-once guarantees.

- Repository: [Microsoft/pg_durable](https://github.com/microsoft/pg_durable)
- Tags: deep-dive
- Published: 2026-06-08

---

**Deterministic replay in duroxide is the crash-recovery process that re-executes persisted activities from `duro_history` to resume pg_durable orchestrations after a crash, requiring purely deterministic orchestration code because any nondeterminism would cause replayed executions to diverge from the original history and break exactly-once guarantees.**

`pg_durable` is the open-source Microsoft extension that brings durable task orchestration directly into PostgreSQL. Its runtime is built on `duroxide`, a Rust-based durable-task framework that relies on **deterministic replay** to resume orchestrations after a crash or restart. Every line of code inside an orchestration must be purely deterministic, because replay re-executes the same sequence of logic and expects bit-for-bit identical results.

## How Deterministic Replay Works in Duroxide

When an orchestration runs inside pg_durable, each unit of work is executed as a **duroxide activity**. After an activity finishes, its result is persisted to the `duro_history` table. If the PostgreSQL server or the background worker crashes, the runtime can **replay** the workflow by re-executing the activity sequence from the last persisted checkpoint.

In [`src/worker.rs`](https://github.com/microsoft/pg_durable/blob/main/src/worker.rs), the `duroxide_worker_main` function starts the runtime and initiates this replay logic. Because the framework reconstructs execution history rather than snapshotting full memory state, every replayed step must match the original run exactly.

## Why Deterministic Code Is Required for pg_durable Orchestrations

Because replay re-executes the same source code, any nondeterminism would cause the new execution to diverge from the original history. Sources such as `Utc::now()`, random number generators, or unregistered external I/O would yield different values on restart, breaking correctness and potentially violating ordering constraints.

The guarantees that depend on strict determinism include:

- **Crash-recovery** – The workflow resumes from any checkpoint without losing work. Nondeterministic steps would produce different results on replay, corrupting the persisted state model.
- **Exactly-once execution** – Each activity runs once and its result is cached in `duro_history`; replay reuses that cached result. If an activity could return a different value on a second execution, the cache would be meaningless.
- **Deterministic ordering** – The DAG of futures is traversed in a fixed order. In [`src/types.rs`](https://github.com/microsoft/pg_durable/blob/main/src/types.rs), `DUROXIDE_SCHEMA` uses a `BTreeMap` for deterministic iteration, ensuring activity IDs are generated in a repeatable sequence.
- **Testability** – pg_durable’s regression tests rely on deterministic output files. Flaky behavior would make the `*_out` verification files useless.

## Deterministic vs Nondeterministic Orchestration Patterns

The source code in [`src/orchestrations/mod.rs`](https://github.com/microsoft/pg_durable/blob/main/src/orchestrations/mod.rs) contains explicit comments and assertions that forbid nondeterministic sources inside orchestration logic. Below are concrete patterns that illustrate the boundary between valid and invalid code.

### Valid Deterministic Orchestration

The following SQL DSL workflow is safe because every side-effect is expressed as a registered **duroxide activity**:

```sql
SELECT durable.start(
    durable.sql('SELECT count(*) AS total FROM users') => 'users'
    ~> durable.sql('SELECT count(*) AS total FROM orders') => 'orders'
    ~> durable.func('store_stats', '{"users": $users.rows[0].total, "orders": $orders.rows[0].total}')
);

```

All three steps—[`durable.sql`](https://github.com/microsoft/pg_durable/blob/main/durable.sql) and `durable.func`—are activities recorded by the runtime. The orchestration itself never calls `now()`, `random()`, or any other nondeterministic primitive, so replay after a crash is safe.

### Invalid Nondeterministic Orchestration

This workflow is invalid because it embeds nondeterministic SQL directly in the orchestration:

```sql
SELECT durable.start(
    durable.sql('SELECT now()') => 'now'
    ~> durable.sql('SELECT random()') => 'rand'
    ~> durable.func('store', '{"ts": $now.rows[0].now, "r": $rand.rows[0].rand}')
);

```

If the worker restarts, `now()` and `random()` would return different values, causing the replayed execution to diverge from the history stored in `duro_history`.

### Wrapping Nondeterministic Work in an Activity

The correct way to capture the current time is to route the call through a registered activity so the result is persisted:

```sql
SELECT durable.start(
    durable.func('fetch_current_time', '{}') => 'now'
    ~> durable.func('store', '{"ts": $now.rows[0].now}')
);

```

Under the hood, `fetch_current_time` is implemented as a duroxide activity. Its timestamp is saved to `duro_history`, so on replay the cached value is reused instead of recomputed.

## Key Source Files That Enforce the Deterministic Contract

The deterministic guarantee is not merely a coding convention; it is embedded in the architecture and enforced by specific modules:

- **[`src/worker.rs`](https://github.com/microsoft/pg_durable/blob/main/src/worker.rs)** – Contains `duroxide_worker_main`, the background worker entry point that starts the duroxide runtime and upholds the deterministic contract at startup.
- **[`src/types.rs`](https://github.com/microsoft/pg_durable/blob/main/src/types.rs)** – Defines `DUROXIDE_SCHEMA` and documents deterministic serialization order through the use of `BTreeMap`.
- **[`src/orchestrations/mod.rs`](https://github.com/microsoft/pg_durable/blob/main/src/orchestrations/mod.rs)** – Carries a top-level comment that explicitly prohibits random numbers, current time, and other nondeterministic sources inside orchestrations.
- **[`src/orchestrations/execute_function_graph.rs`](https://github.com/microsoft/pg_durable/blob/main/src/orchestrations/execute_function_graph.rs)** – Sorts keys for deterministic logging and iterates the DAG with a deterministic clock.
- **[`docs/pg_durable_spec.md`](https://github.com/microsoft/pg_durable/blob/main/docs/pg_durable_spec.md)** – States that the runtime is powered by duroxide and that orchestrations are deterministic and survive crashes via replay.
- **[`docs/ARCHITECTURE.md`](https://github.com/microsoft/pg_durable/blob/main/docs/ARCHITECTURE.md)** – Provides a visual diagram of the durable runtime and notes the deterministic ordering of DAG nodes.

## Summary

- **Deterministic replay** is the duroxide mechanism that rebuilds workflow state by replaying persisted activities from `duro_history` after a crash.
- **Deterministic code** is mandatory because replay re-executes the same orchestration logic; any deviation would invalidate cached results and break exactly-once semantics.
- pg_durable enforces this contract at compile-time through module comments in `src/orchestrations/` and at runtime through `duroxide_worker_main` in [`src/worker.rs`](https://github.com/microsoft/pg_durable/blob/main/src/worker.rs).
- All side-effects—timers, SQL, HTTP calls—must be expressed as **duroxide activities** so their outputs are recorded and replayed identically.

## Frequently Asked Questions

### What is deterministic replay in duroxide?

Deterministic replay is the crash-recovery strategy used by the duroxide runtime to resume pg_durable orchestrations. After each activity completes, its result is written to the `duro_history` table; if PostgreSQL restarts, duroxide replays the activity sequence from the last checkpoint to reconstruct the workflow state exactly as it was.

### Why is nondeterministic code forbidden in pg_durable orchestrations?

Nondeterministic code is forbidden because duroxide replays orchestration logic during recovery. If an instruction such as `now()` or `random()` produced a different value on replay, the execution path would diverge from the persisted history, breaking correctness and exactly-once guarantees. All side-effects must be routed through activities whose results are cached.

### How does the runtime enforce deterministic execution?

The runtime enforces determinism architecturally and procedurally. The source in [`src/orchestrations/mod.rs`](https://github.com/microsoft/pg_durable/blob/main/src/orchestrations/mod.rs) documents the prohibition on nondeterministic sources, [`src/types.rs`](https://github.com/microsoft/pg_durable/blob/main/src/types.rs) mandates deterministic iteration order with `BTreeMap`, and [`src/worker.rs`](https://github.com/microsoft/pg_durable/blob/main/src/worker.rs) starts the duroxide engine through `duroxide_worker_main` only after validating that the execution environment meets the deterministic contract, including a check that `max_duroxide_connections` is configured above the required minimum.

### Are all SQL queries inside an orchestration automatically deterministic?

No. Only SQL queries that are routed through registered duroxide activities—such as [`durable.sql`](https://github.com/microsoft/pg_durable/blob/main/durable.sql) or `durable.func`—are safe, because their results are persisted. Raw or dynamic SQL that invokes nondeterministic functions like `now()` or `random()` inside the orchestration logic itself violates the deterministic requirement and will break replay.