How Scheduled Background Jobs Work with Leader Election in Instatic

Instatic ensures only one instance processes scheduled background jobs at any given moment by using database advisory locks, with PostgreSQL providing distributed coordination and SQLite using sentinel tokens for single-process deployments.

In the CoreBunch/Instatic repository, every recurring background task—from the publish scheduler to the plugin scheduler—runs on every host instance, but only the elected leader performs the actual work. This leader election mechanism relies on a shared advisory lock implementation that transparently supports both high-availability PostgreSQL clusters and single-process SQLite environments.

The Advisory Lock Foundation

The core of Instatic's leader election resides in server/db/advisoryLock.ts, which provides a unified interface over PostgreSQL's native advisory locks and SQLite's single-process mode.

The implementation differs by database backend:

• PostgreSQL – Uses the native pg_try_advisory_lock and pg_advisory_unlock functions. Each lock is identified by a 31-bit integer (the advisory lock key). Only the instance that successfully acquires the lock becomes the leader for that tick; others receive null and skip the work.
• SQLite – Runs in single-process mode, so no cross-process coordination is required. The lock helper catches the "function not found" error and returns a sentinel token ('sqlite-leader'), allowing the caller to behave as if it were the leader without any database coordination.

The public API exported from advisoryLock.ts consists of three key functions:

Function	Purpose
tryAcquireLeader(db, lockKey)	Attempts to claim the lock; returns 'pg-advisory', 'sqlite-leader', or null.
releaseLeader(db, token, lockKey, logPrefix)	Releases a PostgreSQL advisory lock (no-op for SQLite).
withSchedulerLeaderLock(db, lockKey, logPrefix, fn)	Convenience wrapper that runs fn only if the instance wins the election, automatically releasing the lock afterwards.

Publish Scheduler Implementation

The publish scheduler in server/publish/publishScheduler.ts handles the automatic publishing of content rows where status = 'scheduled' and scheduled_publish_at is in the past.

The scheduler uses a setInterval timer (defaulting to 10 seconds) to drive a polling loop. Before each tick, it calls withSchedulerLeaderLock with the advisory lock key 982410937:

import { withSchedulerLeaderLock } from '@/server/db/advisoryLock';
import { listDuePublishSchedules, cancelScheduledPublish } from '@/server/repositories/data/rows';
import { publishDataRow, emitContentEntryUpdated } from '@/server/publish/publishRow';

const ADVISORY_LOCK_KEY = 982410937;

async function tickPublishScheduler(db: DbClient) {
  await withSchedulerLeaderLock(db, ADVISORY_LOCK_KEY, '[publish-scheduler]', async () => {
    const due = await listDuePublishSchedules(db, new Date().toISOString(), 25);
    for (const entry of due) {
      try {
        await publishDataRow(db, entry.rowId, null);
        await emitContentEntryUpdated(db, entry.rowId, ['status'], { kind: 'system' });
      } catch (err) {
        console.error(`[publish-scheduler] failed for ${entry.rowId}:`, err);
        await cancelScheduledPublish(db, entry.rowId, null);
      }
    }
  });
}

If another host instance holds the lock, the entire tick is skipped. This guarantees that only one instance ever publishes the same row, even when multiple servers are running behind a load balancer.

Plugin Scheduler Implementation

The plugin scheduler in server/plugins/scheduler.ts drives the api.cms.schedule.* API used by installed plugins. It follows the same leader election pattern but uses a distinct advisory lock key to avoid contention.

The flow mirrors the publish scheduler but adds an additional safety layer:

1. A setInterval timer (also 10 seconds) triggers tickPluginScheduler.
2. The tick wraps its work inside withSchedulerLeaderLock using lock key 712830541.
3. It fetches due schedules via selectDueSchedules and attempts an atomic row-level claim via tryClaimSchedule, preventing two ticks from firing the same schedule even if the advisory lock were somehow shared.
4. Each claimed schedule executes in a worker (runScheduleInWorker), results are recorded, and the advisory lock is released.

Because the advisory lock is scoped by a unique key, the publish scheduler and plugin scheduler never contend with each other, even when both run in the same process.

Failure Handling and High Availability

Instatic's leader election handles failure scenarios differently depending on the database backend:

• PostgreSQL – If the leader crashes before releasing the lock, PostgreSQL automatically frees the advisory lock when the connection is lost. This allows the next tick on another host to acquire the lock and become the new leader.
• SQLite – No crash hand-off is required because there is only one process; the sentinel token is always returned.
• Task-level failures – Both schedulers treat individual task failures (e.g., a publish error) as ordinary errors, logging them and either reverting state via cancelScheduledPublish or pausing the schedule after a configurable failure cap (in the plugin scheduler).

Code Examples

Running a One-Off Scheduled Publish

To implement a custom scheduler tick that respects leader election:

import { withSchedulerLeaderLock } from '@/server/db/advisoryLock';
import { listDuePublishSchedules, cancelScheduledPublish } from '@/server/repositories/data/rows';
import { publishDataRow, emitContentEntryUpdated } from '@/server/publish/publishRow';

const ADVISORY_LOCK_KEY = 982410937;

async function tickPublishScheduler(db: DbClient) {
  await withSchedulerLeaderLock(db, ADVISORY_LOCK_KEY, '[publish-scheduler]', async () => {
    const due = await listDuePublishSchedules(db, new Date().toISOString(), 25);
    for (const entry of due) {
      try {
        await publishDataRow(db, entry.rowId, null);
        await emitContentEntryUpdated(db, entry.rowId, ['status'], { kind: 'system' });
      } catch (err) {
        console.error(`[publish-scheduler] failed for ${entry.rowId}:`, err);
        await cancelScheduledPublish(db, entry.rowId, null);
      }
    }
  });
}

Using the Advisory Lock Helper Directly

For custom background jobs that need leader election:

import { tryAcquireLeader, releaseLeader } from '@/server/db/advisoryLock';

const LOCK_KEY = 123456789;
const LOG_PREFIX = '[my-job]';

async function runIfLeader(db: DbClient) {
  const token = await tryAcquireLeader(db, LOCK_KEY);
  if (!token) {
    // Another instance is the leader – skip work.
    return;
  }
  try {
    // …perform exclusive work here…
  } finally {
    await releaseLeader(db, token, LOCK_KEY, LOG_PREFIX);
  }
}

Starting the Plugin Scheduler at Boot

Initialize the plugin scheduler idempotently at application startup:

import { startScheduler } from '@/server/plugins/scheduler';
import { db } from '@/server/db/client';

startScheduler(db); // Safe to call on every boot

Summary

• server/db/advisoryLock.ts provides the central leader election mechanism using PostgreSQL advisory locks or SQLite sentinel tokens.
• Publish scheduler (server/publish/publishScheduler.ts) uses lock key 982410937 to ensure only one instance publishes scheduled content rows.
• Plugin scheduler (server/plugins/scheduler.ts) uses lock key 712830541 with an additional atomic row-level claim to prevent duplicate executions.
• PostgreSQL automatically releases locks on connection loss, enabling automatic failover to healthy instances.
• SQLite deployments skip coordination entirely by returning 'sqlite-leader' for all lock attempts.

Frequently Asked Questions

What happens if the leader instance crashes?

PostgreSQL automatically frees the advisory lock when the database connection is lost, allowing another instance to acquire the lock on the next tick. SQLite deployments do not require crash handling because they operate as a single process.

Can the publish and plugin schedulers run concurrently?

Yes. Each scheduler uses a distinct advisory lock key (982410937 for publish, 712830541 for plugins), ensuring they never block each other while maintaining single-leader semantics within their respective domains.

How does leader election work with SQLite?

SQLite runs in single-process mode, so no distributed coordination is needed. The advisory lock helper catches the missing function error and returns a sentinel token ('sqlite-leader'), causing the instance to always behave as the leader without database overhead.

What is the default polling interval for scheduled jobs?

Both the publish scheduler and plugin scheduler poll every 10 seconds by default, using setInterval to trigger their respective tick functions. This interval determines how quickly scheduled tasks execute after their target time is reached.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}