Epoch Sentinel Mechanism in pg_durable: How the Background Worker Detects Extension Drop-and-Recreate Races

The epoch sentinel mechanism in microsoft/pg_durable writes a UUID heartbeat row into df._worker_epoch so the background worker can detect when the extension has been dropped or recreated and restart its Duroxide runtime before operating on stale object references.

The pg_durable extension hosts a Rust-based background worker (BGW) that runs the Duroxide runtime inside PostgreSQL. Because the BGW periodically polls the pg_extension catalog to confirm the extension is still present, a rapid DROP EXTENSION ... CASCADE followed immediately by CREATE EXTENSION can occur between two poll ticks and leave the worker holding dangling pointers to objects that no longer exist. The epoch sentinel mechanism closes this detection gap by maintaining a unique sentinel row that is automatically destroyed when the extension is removed.

Why the Epoch Sentinel Mechanism Is Needed

The background worker uses catalog polling to know when pg_durable is installed. If the extension is dropped and recreated inside a single polling window, a simple extension-exists check can return a false positive because the new installation reuses the same extension name. In that scenario, the worker continues running with a stale Duroxide runtime, cached query plans, and schema references that point to the old, deleted catalog objects. As described in docs/extension_lifecycle.md, the sentinel removes this blind spot by relying on a tangible row in the df schema that DROP EXTENSION ... CASCADE forcibly removes.

How the Epoch Sentinel Mechanism Works

Once the Duroxide runtime is successfully initialized, the worker generates a fresh UUID and inserts it as an epoch sentinel into the df._worker_epoch table defined in migration scripts such as sql/pg_durable--0.2.0--0.2.1.sql. While the worker remains active, it periodically updates the last_seen_at timestamp for that same UUID and verifies the row is still present. If the row disappears, the schema is gone, or the UUID changes, the worker treats the sentinel as lost.

Initializing the Sentinel in src/worker.rs

The write_epoch_sentinel function in src/worker.rs (line 91) generates a new UUID, deletes any previous epoch row, and inserts a fresh record into df._worker_epoch. It returns the epoch_id so the polling loop can track it.

// After the Duroxide runtime has started
let epoch_id = write_epoch_sentinel(&mgmt_pool).await?;
log!("pg_durable: epoch sentinel written ({})", epoch_id);

Validating the Sentinel on Every Poll Tick

During each iteration of the main loop, the worker invokes check_epoch_sentinel in src/worker.rs (line 55). This function updates last_seen_at for the stored UUID and returns true only if the row still exists in df._worker_epoch.

loop {
    if let Some(eid) = epoch_id {
        let still_valid = check_epoch_sentinel(&mgmt_pool, eid).await;
        if !still_valid {
            log!("pg_durable: epoch sentinel gone — extension dropped or recreated");
            break; // trigger shutdown / re-init
        }
    }
    // sleep until next poll tick …
}

Fallback to Extension-Exists Polling

If the worker cannot write the sentinel—for example, if the df schema is not yet ready—it falls back to basic extension-exists polling. This branch is handled inside run_until_extension_dropped_or_shutdown in src/worker.rs (line 88).

let epoch_id = match write_epoch_sentinel(&mgmt_pool).await {
    Ok(id) => Some(id),
    Err(e) => {
        log!("pg_durable: failed to write epoch sentinel: {} — falling back to extension-exists polling", e);
        None
    }
};

Safety and Correctness Guarantees Provided by the Sentinel

The epoch sentinel mechanism delivers three concrete guarantees according to the microsoft/pg_durable source code:

• Safety. When the sentinel disappears, the background worker shuts down the Duroxide runtime immediately. This prevents crashes or writes to catalog objects that were removed by DROP EXTENSION ... CASCADE.
• Correctness. A lost sentinel forces a full teardown and fresh initialization of the runtime against the newly created extension schema. All cached plans and internal schema caches are rebuilt so the worker never uses stale metadata.
• Robustness. Because the sentinel lives inside the df schema, it is destroyed the moment the extension is dropped. This catches drop-and-recreate sequences that happen faster than the worker’s poll interval, leaving no window for silent operation on obsolete state.

Summary

• The pg_durable background worker polls the pg_extension catalog, but that alone cannot catch a rapid DROP EXTENSION ... CASCADE → CREATE EXTENSION sequence.
• The epoch sentinel mechanism writes a UUID row into df._worker_epoch via write_epoch_sentinel in src/worker.rs and validates it with check_epoch_sentinel.
• If the row is missing, the schema is gone, or the UUID mismatches, the worker tears down the Duroxide runtime and returns to its waiting state.
• This guarantees safety, correctness, and robustness even when extension recreation occurs faster than the polling interval.

Frequently Asked Questions

What table does the epoch sentinel mechanism use?

The mechanism stores its heartbeat row in df._worker_epoch, a table created by pg_durable migration scripts such as sql/pg_durable--0.2.0--0.2.1.sql. This table holds the epoch_id UUID and a last_seen_at timestamp that the background worker updates during each poll cycle.

How does the background worker know if the extension was recreated?

The worker calls check_epoch_sentinel in src/worker.rs to update last_seen_at for its stored UUID. If the function returns false, or if any query error occurs because the df schema or table is missing, the worker concludes the extension was dropped or recreated and shuts down the Duroxide runtime.

Why is catalog polling alone insufficient for the pg_durable worker?

Catalog polling checks pg_extension on a fixed interval. A DROP EXTENSION ... CASCADE followed immediately by CREATE EXTENSION can happen between two ticks, so the worker would never see the transient dropped state and would continue running with a stale runtime and cached plans.

What happens if the worker fails to write the epoch sentinel?

If write_epoch_sentinel returns an error, the worker stores None for the epoch ID and falls back to plain extension-exists polling. This fallback is implemented in run_until_extension_dropped_or_shutdown inside src/worker.rs, ensuring the worker degrades gracefully rather than crashing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}