How the no-mistakes Pipeline Parallelizes Step Execution with Round-Based Batching

The no-mistakes pipeline executes independent CI steps concurrently by grouping them into discrete rounds, using Goroutines and sync.WaitGroup to parallelize execution within each round while maintaining deterministic ordering across rounds.

The kunchenguid/no-mistakes repository implements a deterministic CI-style pipeline engine that maximizes CPU utilization through intelligent grouping of steps. This article explains how the pipeline employs round-based batching to parallelize step execution while preserving execution order dependencies, referencing the actual Go source code implementation.

Understanding the Round-Based Architecture

What Is a Round?

A round represents a discrete phase of pipeline execution where a group of steps runs simultaneously. When a step completes, it produces a round record—a StepRound structure containing what the step did, what findings were reported, and which findings the user or auto-fix selected to keep. The executor stores this metadata in SQLite via db.InsertStepRound, allowing subsequent steps to reference the execution history.

Step Dependency Management

Steps are categorized by their dependency requirements. Independent steps such as lint, test, and static analysis execute within the same round, while dependent steps (for example, push requiring successful rebase completion) wait for subsequent rounds. The executor in internal/pipeline/executor.go evaluates each step's readiness against the currentRound counter, ensuring steps only enter the active batch when their prerequisites are satisfied.

Parallel Execution Mechanics in executor.go

The core concurrency logic resides in internal/pipeline/executor.go. The Execute method implements a round counter that drives the batching loop. For each round, the executor collects runnable steps and dispatches them using Goroutines coordinated by a sync.WaitGroup.

Around lines 220-235, the parallel dispatch implementation appears as follows:

var wg sync.WaitGroup
for _, step := range batch {
    wg.Add(1)
    go func(s types.StepName) {
        defer wg.Done()
        e.runStep(ctx, s)            // core step execution
    }(step)
}
wg.Wait() // block until the whole batch finishes

This pattern ensures that all steps in a round complete before the pipeline advances. The runStep method handles the individual step logic, while wg.Wait() blocks the main thread until the entire batch reports completion via defer wg.Done().

Pipeline Orchestration Flow

Building the Step Graph

In internal/pipeline/pipeline.go, the pipeline constructs an ordered slice of step descriptors (types.StepName). The executor receives this step graph and creates descriptors for the current run, determining which steps have no pending dependencies and are therefore eligible for the current round.

Round Iteration and Advancement

The executor increments currentRound only after the sync.WaitGroup signals batch completion. If an auto-fix modifies a step's output, the system inserts a new round for that specific step via internal/pipeline/steps/round_history.go, which formats the history for the agent and maintains the deterministic sequence.

Persisting Round Metadata

Each step writes outcomes to the SQLite database defined in internal/db/round.go. The schema includes fields such as Round, Trigger, FindingsJSON, and SelectionSource. When the batch finishes, the executor updates the run's round metadata, ensuring that later steps can query previous results through the round history interface.

Implementing the Pipeline in Practice

The following example from the CLI entry point demonstrates how to initialize and run the pipeline with automatic round-based batching:

func runPipeline(repoPath string) error {
    // 1️⃣ Load config and initialise DB.
    cfg, err := config.LoadGlobal()
    if err != nil { return err }

    // 2️⃣ Build the step list for the current run.
    pl, err := pipeline.New(cfg, repoPath)
    if err != nil { return err }

    // 3️⃣ Execute – the executor does the round‑based batching internally.
    exec := pipeline.NewExecutor(pl)
    if err := exec.Execute(context.Background()); err != nil {
        return err
    }
    return nil
}

The Execute call automatically determines which steps are ready for round 1, runs them in parallel via Goroutines, waits for batch completion, records round metadata for each step, and continues with round 2 until the pipeline exhausts all steps.

Summary

• The no-mistakes pipeline uses round-based batching to group independent steps into parallel execution units, maximizing CPU utilization.
• Goroutines and sync.WaitGroup in internal/pipeline/executor.go manage concurrent step execution within each round while ensuring barrier synchronization.
• SQLite persistence in internal/db/round.go and internal/pipeline/steps/round_history.go maintains round metadata for deterministic ordering and historical reference.
• Dependent steps automatically sequence into later rounds while independent steps (lint, test, analysis) exploit multi-core parallelism.
• The design balances parallel performance with deterministic execution flow, ensuring reproducible CI results.

Frequently Asked Questions

How does no-mistakes handle step dependencies?

Steps with dependencies wait for specific rounds to complete before executing. The executor evaluates each step's readiness against the currentRound counter and historical round data stored in SQLite, ensuring dependent steps like push only run after prerequisite rounds (e.g., rebase) finish successfully.

What happens if a step fails during a parallel batch?

When e.runStep() returns an error or failure status for any step in the batch, the executor still waits for all Goroutines in the sync.WaitGroup to complete via wg.Wait() before advancing to the next round. The round metadata records individual step statuses, allowing subsequent rounds to handle partial failures appropriately without leaving orphan Goroutines.

Can the round-based batching utilize all CPU cores?

Yes. Because each step in a round runs in its own Goroutine and step implementations in internal/pipeline/steps/*.go (such as lint.go and test.go) are stateless, the Go runtime schedules these Goroutines across available CPU cores. This design allows lint, test, and analysis steps to run simultaneously without blocking each other, fully exploiting multi-core architectures.

Where is the round history stored and how is it accessed?

Round history persists in a SQLite database via internal/db/round.go, which defines the schema including Round, Trigger, and FindingsJSON fields. The internal/pipeline/steps/round_history.go file provides methods to insert round records and format history for the agent, enabling steps to reference previous round outcomes when determining execution eligibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}