How Worktree Isolation Enables Non-Blocking Pipeline Execution in no-mistakes

Worktree isolation enables non-blocking pipeline execution by carving each pipeline run into its own Git worktree linked to a shared bare gate repository, eliminating file conflicts and lock contention between concurrent processes.

The no-mistakes CI system achieves parallel pipeline execution without resource contention through an innovative use of Git worktrees. By isolating each run inside a dedicated worktree carved from a bare gate repository, the system ensures that multiple pipelines can build, test, and push code simultaneously without blocking each other or corrupting shared state.

The Architecture of Worktree Isolation

The Bare Gate Repository Pattern

At the core of this design lies a bare gate repository (*.git) that stores only remote-tracking refs and object data. According to the execution context documentation in internal/pipeline/steps/execution_context.go (lines 5-21), the architecture specifically uses a "worktree carved from a bare gate repository, not in the original repo" to maintain strict separation between the mutable execution environment and the shared upstream data.

Pointer-Based Worktree Creation

When a pipeline initiates, the system creates a worktree directory containing a .git pointer file that references the bare gate's common Git directory. This pointer allows Git commands to execute against the shared bare repository while maintaining an isolated checkout. As implemented in internal/git/git.go, the WorktreeAdd function establishes this link, enabling operations like git add, git commit, and git push to affect only the isolated checkout without touching the gate's internal state.

How Concurrent Runs Avoid Blocking

Eliminating Lock Contention

Traditional Git checkouts lock the working directory, preventing concurrent operations on the same branch. The bare gate repository eliminates this bottleneck because no process holds a checkout lock on the shared data. Each worktree operates as an independent filesystem view, allowing two pipelines to modify files, run tests, or push commits simultaneously. This design prevents the classic "worktree busy" error that typically blocks parallel execution in standard Git workflows.

Parallel Execution Without File Conflicts

Because every run works within its own worktree folder, filesystem modifications from one pipeline never interfere with another. The isolation ensures that build artifacts, temporary files, and working directory changes remain sandboxed within the individual worktree, while the underlying Git operations synchronize through the bare repository's object database.

Repository Root Resolution in Worktrees

Helper functions such as FindMainRepoRoot correctly resolve the repository root even when invoked from within a worktree context. The test suite in internal/git/git_branch_worktree_test.go verifies this behavior, with lines 66-94 confirming that a worktree still yields the main repository's root. This guarantees that subsequent Git commands—such as reading remotes or resolving branches—operate on the correct repository context regardless of the current working directory.

Lifecycle Management and Cleanup

Deterministic teardown prevents stale state accumulation. When a pipeline finishes, the system invokes WorktreeRemove to delete the temporary worktree directory and unlink it from the gate repository. As tested in internal/git/git_branch_worktree_test.go (lines 40-48), this cleanup frees the path for future runs and ensures no transient files persist to cause interference between executions.

Implementation Example

The following pattern demonstrates how no-mistakes orchestrates isolated pipeline runs using the worktree API:

// Create an isolated worktree for a new pipeline run.
ctx := context.Background()
repo := "/path/to/bare-gate.git"
wtDir := filepath.Join(os.TempDir(), "run-worktree")
sha := "a1b2c3d4" // commit to checkout

if err := git.WorktreeAdd(ctx, repo, wtDir, sha); err != nil {
    log.Fatalf("cannot add worktree: %v", err)
}

// … run lint, test, review, etc. inside wtDir …

// Clean up when the run finishes.
if err := git.WorktreeRemove(ctx, repo, wtDir); err != nil {
    log.Fatalf("cannot remove worktree: %v", err)
}

The WorktreeAdd and WorktreeRemove functions are implemented in internal/git/git.go and exercised by internal/git/git_branch_worktree_test.go. Shared Git operations such as Run and Push that operate on the worktree's pointer to the bare gate are provided in internal/pipeline/steps/common_git.go.

Summary

• Bare gate architecture: The shared bare repository stores refs and objects without a working copy, eliminating checkout locks.
• Pointer-based isolation: Each worktree uses a .git file to link to the bare gate while maintaining an independent filesystem view.
• Non-blocking concurrency: Multiple pipelines execute simultaneously without file conflicts or "worktree busy" errors.
• Correct root resolution: Helper functions accurately resolve the main repository root from within any worktree context.
• Deterministic cleanup: The WorktreeRemove function ensures temporary state is purged after each run, preventing resource leaks.

Frequently Asked Questions

What is a Git worktree and how does it differ from a clone?

A Git worktree creates an additional working directory linked to the same repository, allowing multiple branches to be checked out simultaneously without cloning the entire object database again. Unlike a full clone, a worktree shares the underlying Git objects with the main repository via a .git pointer file, reducing disk usage and synchronization overhead while maintaining independent working directories.

Why use a bare repository as the gate instead of a regular checkout?

A bare repository contains no working directory, which prevents any process from holding a lock on checked-out files. According to the no-mistakes source code in internal/pipeline/steps/execution_context.go, this design ensures that the shared upstream data remains immutable and accessible to multiple worktrees concurrently, whereas a regular checkout would create lock contention and block parallel pipeline runs.

How does no-mistakes prevent "worktree busy" errors during concurrent runs?

The system prevents these errors by ensuring no single process monopolizes the gate repository's working directory. Because the gate is bare and each pipeline runs in its own worktree with independent filesystem state, Git operations execute against the shared object database without contending for working directory locks. This architecture allows simultaneous git push and commit operations from different worktrees without blocking.

What happens if a pipeline run crashes before WorktreeRemove executes?

While the provided source analysis focuses on successful cleanup via WorktreeRemove in internal/git/git.go, crashed runs would leave orphaned worktree directories on the filesystem. These stale directories contain only the temporary checkout state and pointer files, not the actual Git objects stored in the bare gate. Operators would need to implement additional garbage collection or monitoring to detect and remove abandoned worktrees, though the bare gate repository itself remains uncorrupted by the failure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}