How ai-memory Ensures At-Least-Once Delivery of Downstream Effects

ai-memory guarantees at-least-once delivery by combining a single-writer SQLite actor, monotonic delivery cursors, explicit acknowledgments, managed-run leases, and idempotent router gating to ensure every downstream effect is processed even during crashes or network failures.

The akitaonrails/ai-memory repository implements a robust pipeline for AI agent observations where downstream effects—such as wiki updates, agent handoffs, and LLM-driven consolidations—must never be lost. By embedding at-least-once delivery semantics directly into its Rust-based storage layer and network transport, the system ensures that critical events persist through process failures, network partitions, and message replays.

Single-Writer SQLite Actor for Atomic Persistence

At the foundation of ai-memory’s reliability model sits a dedicated writer thread that serializes all state mutations. In crates/ai-memory-store/src/writer.rs, the WriterInner struct processes every write command within a single SQLite transaction, ensuring that state changes are persisted to disk before the function returns success.

This architecture prevents lost writes during crashes. If the process terminates mid-transaction, SQLite’s atomicity guarantees either a full commit or a complete rollback, leaving the database in a consistent state. All downstream effects flow through this bottleneck, ensuring that no observation reaches the delivery cursor without first surviving a disk write.

Monotonic Delivery Cursor for Idempotent Tracking

Each workstream maintains a delivery_cursor in the workstream_native_sessions table, implemented in crates/ai-memory-store/src/workstream.rs. This cursor tracks the highest event sequence number successfully processed by the consumer.

When a delivery succeeds, the cursor advances using MAX(delivery_cursor, ?1), making the update inherently idempotent. If a crash triggers a replay, duplicate events are silently ignored because the stored cursor already points past the event sequence. This mechanism ensures that every event is processed at least once while preventing the visible side effects of double-processing.

Explicit Delivery Acknowledgments

The system distinguishes between delivery and acknowledgment. Functions such as accept_managed_run_context and accept_startup_context in crates/ai-memory-store/src/writer.rs record that a specific key (e.g., SessionStart) was successfully consumed by the downstream system.

The store only clears the pending delivery flag after receiving this explicit ACK. If the consuming process crashes before acknowledging, the next managed run reads the unchanged cursor and re-delivers the event. This acknowledgment-based completion closes the loop between storage and effect application.

Managed-Run Leases and Replay Protection

Crash recovery relies on ManagedRunId leases with expiration timestamps, managed in crates/ai-memory-store/src/workstream.rs. When a managed run starts, it acquires a lease that grants exclusive rights to advance the delivery cursor for that specific run context.

If the lease expires or the host process crashes, a subsequent run can acquire the lease and safely replay any unacknowledged events. The lease acts as a fencing token, ensuring that only one active run processes a given workstream segment at a time while guaranteeing that no event remains orphaned.

Transport Layer with At-Least-Once Semantics

The MCP server implementation in crates/ai-memory-mcp/src/server.rs explicitly selects transport mechanisms—such as NATS JetStream—that provide at-least-once delivery guarantees at the network layer. Even if the network drops packets or the receiver temporarily disconnects, the broker retains messages and retransmits them until the receiver issues a transport-level acknowledgment.

This layer shields the application from transient network failures, ensuring that hook payloads arrive at the router even under adverse conditions.

Router Gating and Idempotent Ingestion

Incoming payloads pass through crates/ai-memory-hooks/src/router.rs, which tags each observation with a unique key. When the router encounters a duplicate key, it de-duplicates the observation but still processes the downstream side effects because the delivery cursor logic ensures idempotency.

This design means the router can safely retry deliveries without risk of corrupting the wiki state or triggering redundant LLM consolidations.

Implementation Flow: Preparing and Acknowledging a Run

The following Rust code demonstrates the typical three-phase flow for emitting a workstream event with guaranteed delivery:

use ai_memory_store::Store;

// 1️⃣ Prepare a managed run to establish a lease
let prep = PrepareWorkstreamRun {
    workspace_id,
    project_id,
    repo_fingerprint: "abc".into(),
    worktree_fingerprint: "def".into(),
    cwd: "/repo".into(),
    agent: AgentKind::Hermes,
    automatic_harness: true,
    available_agents: vec![AgentKind::Hermes],
    selection: WorkstreamSelection::Current,
    lease_owner: "my-host".into(),
};
let prepared = store.prepare_workstream_run(prep).await?;

// 2️⃣ Emit the event with a delivery key for tracking
let event = NewWorkstreamEvent {
    // ... observation fields ...
    delivery_key: Some("session_start".into()),
    // ...
};
store.finish_workstream_run(FinishWorkstreamRun {
    run_id: prepared.run_id,
    native_session_id: Some("session-1".into()),
    source_cursor: None,
    events: vec![event],
    complete: true,
    segment_path: None,
    exit_code: None,
}).await?;

// 3️⃣ Acknowledge successful delivery to advance the cursor
store.accept_managed_run_context(prepared.run_id).await?;

If step three fails due to a crash, the next call to prepare_workstream_run will detect the unacknowledged event through the delivery_cursor and replay the delivery. Because the cursor update uses MAX, the replay is idempotent and safe.

Summary

• Atomic persistence: The WriterInner single-threaded SQLite actor ensures all mutations survive crashes via transactional commits in writer.rs.
• Idempotent tracking: The delivery_cursor in workstream.rs advances monotonically using MAX, preventing double-processing during replays.
• Explicit acknowledgments: Functions like accept_managed_run_context separate delivery from completion, enabling crash recovery.
• Lease-based fencing: ManagedRunId leases prevent split-brain scenarios while allowing safe replay of orphaned events.
• Reliable transport: The MCP server selects brokers like NATS JetStream that guarantee at-least-once network delivery.
• Safe ingestion: The router.rs deduplication layer ensures duplicate network deliveries do not corrupt downstream state.

Frequently Asked Questions

What happens if a process crashes before acknowledging a delivery?

The delivery_cursor remains unchanged because accept_managed_run_context was never called. When a new managed run acquires the lease for that workstream, it reads the old cursor value and re-delivers all events from that point forward. The idempotent MAX update ensures that if the effect was already applied, the second delivery becomes a no-op.

How does ai-memory prevent duplicate processing during replays?

The system relies on monotonic cursor advancement rather than payload hashing. Because the cursor only moves forward using MAX(delivery_cursor, ?1), a replayed event that was already processed finds the cursor already past its sequence number, causing the delivery logic to skip the downstream effect.

What role does the SQLite single-writer pattern play in reliability?

By funneling all writes through the WriterInner thread in writer.rs, ai-memory eliminates race conditions and ensures that the delivery_cursor update happens atomically with the event persistence. This prevents the "lost update" problem where a cursor might advance without the corresponding data being durably stored.

How does the transport layer contribute to at-least-once guarantees?

According to crates/ai-memory-mcp/src/server.rs, the MCP server configures the underlying message broker (e.g., NATS JetStream) to require explicit acknowledgments and maintain persistent message logs. If the network drops a packet or the receiver disconnects, the broker retains the message and redelivers it until the application layer successfully processes it and advances the delivery_cursor.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}