How NDI (Nondeterministic Idempotence) Ensures Workflow Completion in Gas Town

NDI guarantees that any Gas Town agent can inspect persistent beads to drive workflows to completion regardless of individual operation failures, making the system self-healing and convergent.

Gas Town implements Nondeterministic Idempotence (NDI) as its core reliability model to solve the coordination challenges inherent in distributed workflow management. In the gastownhall/gastown repository, NDI ensures that agents—including Dogs, Witnesses, Deacons, and Mayors—can operate independently while the overall system still converges deterministically on a completed state. This article examines how NDI functions architecturally and practically within the Gas Town ecosystem.

What Is NDI in Gas Town?

According to the project glossary in docs/glossary.md, NDI is defined as “the overarching goal ensuring useful outcomes through orchestration of potentially unreliable processes.” Rather than guaranteeing that individual operations execute identically every time, NDI guarantees that persistent beads and oversight agents will eventually drive the workflow to completion even when operations fail or produce varying results.

The fundamental principle is that state—represented as immutable beads, their relationships, and labels—serves as the single source of truth. Agents never maintain mutable in-memory coordination; they always re-discover the current state from the durable bead database on every execution.

The NDI Execution Model

In Gas Town’s NDI model, execution is stateless at the agent level but persistent at the data level. When a Dog, Witness, or Deacon performs work, it does not track progress in local memory or session state. Instead, each agent:

1. Queries the current bead graph using commands like bd list --label mountain
2. Computes the necessary actions based on the discovered state
3. Mutates beads atomically (adding labels, updating status, or closing issues)
4. Exits without retaining coordination context

This approach ensures that any agent can resume work from any point without prior knowledge of previous attempts, eliminating “lost thread” problems common in coordinator-only designs.

How NDI Drives Completion Through Agent Coordination

The architecture implements NDI through five distinct agent responsibilities that collectively ensure workflow convergence:

Step 1: Dog State Discovery

Any Deacon Dog periodically audits a mountain convoy by executing bd list --label mountain. The Dog reads the current count of closed issues and compares it to previous audit data stored in bead labels. Because the Dog starts with a fresh context each run, it never depends on session data—it simply discovers the current state from the database.

Step 2: Witness Failure Handling

The Witness agent monitors polecat (worker) failures. When an issue fails three or more times, the Witness adds a mountain:skipped label, converting the issue into a blocked node. This logic is deterministic on bead state: any Dog that re-queries the bead can execute the same skip-after-N-fails logic, ensuring consistent enforcement regardless of which physical agent performs the check.

Step 3: Deacon Stall Investigation

If the audit reveals no progress, the Deacon Dog spawns a mountain-dog formula that investigates the stall. While the Deacon’s heartbeat runs deterministically every 5–10 minutes, the investigation logic is intentionally nondeterministic—it may succeed on the first retry or after several attempts. NDI ensures that repeated attempts converge on the same final state because each attempt reads the current bead status before acting.

Step 4: Mayor Optional Oversight

The Mayor receives mail from Dogs summarizing stalls or completions. While the Mayor can manually intervene (for example, clearing a mountain:skipped label using bd update $ISSUE --status=open --remove-label mountain:skipped), the system continues progressing without intervention. The underlying beads remain immutable once closed, making the Mayor’s actions optional rather than required for completion.

Step 5: Continuous Ready State Processing

All agents continuously run bd ready --epic=… to feed the next ready issue into the pipeline. If a prior step failed, the same issue will be re-slung until it succeeds or is marked skipped. The idempotent nature of bd ready means that feeding the same issue multiple times does not corrupt state; the beads remain the canonical record, and duplicate processing is safely handled as a no-op.

Architectural Implementation Examples

Mountain-Eater Design Pattern

The docs/design/convoy/mountain-eater.md file explicitly implements NDI in its design principle stating “No Agent Holds the Thread.” In this architecture, the epic (the bead itself) serves as the coordination thread, not any particular agent. The layered approach—Convoy Manager → Witness → Deacon Dog → Mayor—provides redundant monitoring where each layer independently discovers state from beads.

If the Witness layer misses a stall, the Deacon Dog catches it on the next heartbeat. If the Dog crashes, the next Dog instance resumes from the same bead state. This redundancy ensures eventual convergence without requiring a single coordinator.

Towers of Hanoi Demo

The docs/examples/hanoi-demo.md provides a concrete demonstration of NDI resilience. You can stop the demo mid-run, resume hours later, and the system will reach the same final state (all issues closed) regardless of how many sessions were used. The bead state persists across sessions, allowing the workflow to converge deterministically despite nondeterministic execution timing.

Core Guarantees Provided by NDI

Gas Town’s NDI implementation provides four critical guarantees that ensure workflow completion:

1. Stateless Coordination – No agent holds long-lived coordination state; all decisions derive from persistent beads queried at runtime.

2. Retry-Safety – Re-executing the same Dog audit, Witness check, or Deacon investigation any number of times does not diverge the workflow from its intended completion path.

3. Fault Tolerance – Individual polecat failures, session crashes, or compaction events (handled by plugins/compactor-dog/plugin.md) do not prevent eventual completion because the next agent will re-discover the required work from the bead graph.

4. Deterministic Outcome – Despite nondeterministic execution ordering, agent turnover, and variable retry counts, the final set of closed issues (the completed mountain) is deterministic.

Practical Code Examples

These commands demonstrate NDI principles in a live Gas Town rig:

# Initialize a new mountain workflow (creates the epic bead with "mountain" label)

gt mountain my-epic-auth-rebuild

# Check status via Dog audit (safe to run repeatedly; discovers current state fresh each time)

gt mountain status my-epic-auth-rebuild

# Simulate a manual Witness check (idempotent—running multiple times is safe)

bd list --label mountain --status=open | while read issue; do
  echo "Checking failure count for issue $issue"
  # Logic to increment failure labels based on bead state

done

# Resume a paused mountain after session crash (NDI ensures same state is used)

gt mountain resume my-epic-auth-rebuild

# Clear a skipped label to retry a failed issue (mayoral intervention pattern)

bd update ISSUE-123 --status=open --remove-label mountain:skipped

Summary

• NDI in Gas Town ensures that beads, not agents, hold the workflow state, enabling any agent to resume work from any point.
• The Dog → Witness → Deacon → Mayor pipeline provides redundant, stateless monitoring that converges on completion through persistent bead labels and statuses.
• Commands like bd list --label mountain, bd ready --epic=…, and gt mountain status implement NDI by re-discovering state rather than caching it.
• Key source files defining NDI include docs/glossary.md (definition), docs/design/convoy/mountain-eater.md (architecture), and docs/examples/hanoi-demo.md (practical demonstration).

Frequently Asked Questions

What distinguishes NDI from traditional idempotency?

Traditional idempotency requires that executing the same operation multiple times produces identical results and side effects. NDI relaxes this to allow nondeterministic execution ordering and varying intermediate states while still guaranteeing that the final system state converges deterministically. In Gas Town, this means polecat failures and retry storms do not prevent the workflow from eventually reaching the completed bead state.

How does Gas Town prevent data loss during agent crashes?

Because agents store no session state in memory, a crashing Dog or Witness loses only the current in-flight operation, not the workflow context. The next agent to wake queries the same bead database using bd list or bd ready, discovering the exact state left by the previous agent. The plugins/compactor-dog/plugin.md further ensures that background compaction operations are idempotent and can safely resume after crashes.

Can NDI handle external stateful dependencies?

Gas Town’s NDI applies strictly to the bead graph managed within the repository. When interacting with external systems (such as external APIs or databases), agents must ensure those interactions are externally idempotent or wrapped in bead-tracked transactions. The Witness pattern of labeling issues as mountain:skipped after failures provides a mechanism to block external calls until the system can guarantee safe retry.

Where is NDI formally specified in the Gas Town codebase?

The formal definition resides in docs/glossary.md under the “Nondeterministic Idempotence” entry. The architectural application is documented in docs/design/convoy/mountain-eater.md, specifically in the section titled “Design Principle: No Agent Holds the Thread.” The docs/examples/hanoi-demo.md file provides a runnable specification demonstrating how NDI behaves across session cycles.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}