Evidence System for Run Tracking in no-mistakes: Architecture and Usage

The no-mistakes evidence system tracks run results by persisting artifact files (screenshots, logs, etc.) that prove user intent was satisfied, storing them in either temporary OS directories or in-repository folders that get staged and pushed automatically.

The no-mistakes framework implements a comprehensive evidence system to verify that automated runs satisfy their intended purpose. By capturing and preserving artifacts throughout the pipeline execution, the system provides concrete proof of what occurred during test steps. This article examines how the evidence system operates based on the actual implementation in the kunchenguid/no-mistakes repository.

Core Components of the Evidence System

The evidence architecture rests on three fundamental concepts implemented across specific source files.

Evidence Location Resolution

The system determines where to write evidence files through the resolveTestEvidenceLocation function in internal/pipeline/steps/test.go. By default, the framework creates a temporary directory under the OS-wide temp path at /tmp/no-mistakes-evidence/<run-id>. When users enable store_in_repo: true in the configuration, the system instead uses an in-repository directory such as evidence/.

The location resolution includes safety checks against .gitignore files. If the configured evidence directory matches an ignored pattern, the system falls back to the temporary location to prevent accidental commits of excluded files.

Evidence Metadata Structures

The internal/types/findings.go file defines the core data structures that describe produced artifacts. The Findings struct contains an Artifact collection that specifies what files were created and where they reside. Each artifact includes fields for kind (e.g., screenshot, log), label, and path.

The test step expects agent output matching the testFindingsSchema defined in internal/pipeline/steps/test.go, which validates JSON responses containing an "artifacts" array with specific metadata properties.

Evidence Handling and Lifecycle

After test completion, the push step in internal/pipeline/steps/push.go manages evidence persistence. For in-repo storage, it stages the evidence directory using git add -f <dir> and pushes the changes. The PR summary renderer in internal/pipeline/steps/prsummary.go then processes the metadata, embedding temporary files as collapsible details blocks or linking directly to committed repository files.

Configuration and Setup

Users control evidence behavior through the YAML configuration parsed by internal/config/config.go. The Evidence struct defines the available options:

type Evidence struct {
    StoreInRepo bool
    Dir         string
}

A typical configuration enabling repository storage appears as:

test:
  evidence:
    store_in_repo: true
    dir: evidence

When store_in_repo is false or omitted, the framework defaults to temporary filesystem storage outside version control.

Step-by-Step Workflow

The evidence system operates through a coordinated sequence across multiple pipeline stages.

1. Test Step Initialization

When TestStep.Execute runs, it resolves the evidence location and creates the directory using os.MkdirAll. The step injects the evidence path into the agent prompt, instructing the AI to write all proof files to that specific directory while leaving them untouched after creation.

2. Artifact Generation

The agent writes files to the designated evidence directory and returns structured JSON metadata:

{
  "artifacts": [
    {
      "kind": "screenshot",
      "label": "Checkout screenshot",
      "path": "artifacts/checkout.png"
    },
    {
      "kind": "log",
      "label": "Server log",
      "path": "artifacts/server.log"
    }
  ]
}

3. Repository Staging

In internal/pipeline/steps/push.go, the system checks if evidence should be committed:

if evidenceLocation.StoreInRepo {
    if err := git.Run(ctx, workDir, "add", "-f", evidenceLocation.Dir); err != nil {
        return fmt.Errorf("stage test evidence: %w", err)
    }
}

The -f flag forces staging even if the directory is normally ignored, though the system first checks gitIgnoresPath to avoid committing files that should remain excluded.

4. PR Summary Rendering

The PR summary step reads the Findings data and renders appropriate markdown. For temporary evidence, it generates local file path references within collapsible blocks. For in-repo evidence, it creates direct links viewable in the pull request interface.

Implementation Examples

Creating the Evidence Directory

The test step implements location resolution with fallback logic:

evidenceLocation := resolveTestEvidenceLocation(sctx.WorkDir, sctx.Run.Branch, sctx.Run.ID, sctx.Config.Test.Evidence)
evidenceDir := evidenceLocation.Dir
if evidenceLocation.StoreInRepo && gitIgnoresPath(ctx, sctx.WorkDir, evidenceDir) {
    evidenceLocation = testEvidenceLocation{Dir: testEvidenceDir(sctx.Run.ID)}
    evidenceDir = evidenceLocation.Dir
}
if err := os.MkdirAll(evidenceDir, 0o755); err != nil {
    return nil, fmt.Errorf("create test evidence dir: %w", err)
}

Writing Artifacts

While typically handled by the agent, direct file creation follows standard filesystem operations:

evidencePath := filepath.Join(evidenceDir, "checkout.png")
if err := os.WriteFile(evidencePath, pngData, 0o644); err != nil {
    return err
}

Summary

• The evidence system in no-mistakes persists run artifacts to prove user intent satisfaction through screenshots, logs, and other files.
• Storage defaults to /tmp/no-mistakes-evidence/<run-id> but can be configured to use in-repository directories via store_in_repo: true.
• The Evidence struct in internal/config/config.go and Findings struct in internal/types/findings.go define the configuration and metadata schemas.
• The test step creates evidence directories and guides agents to write files there, while the push step handles staging and the PR summary renders the results.
• Files persist through the run lifecycle without automatic deletion, ensuring evidence remains available for review.

Frequently Asked Questions

What types of files can serve as evidence in no-mistakes?

Any file type can function as evidence, including screenshots (PNG, JPG), log files (TXT, LOG), JSON data exports, or HTML reports. The system treats all files in the evidence directory equally, rendering them based on their declared kind in the artifact metadata returned by the test agent.

How do I configure the evidence system to store files in my repository?

Set store_in_repo: true in the test.evidence section of your no-mistakes configuration file. Optionally specify a custom subdirectory using the dir field. When enabled, the push step automatically stages and commits evidence files to the branch, making them visible in pull requests alongside code changes.

What happens if my evidence directory is listed in .gitignore?

The system detects gitignored paths through the gitIgnoresPath function and automatically falls back to a temporary directory location. This prevents the framework from accidentally committing files that should remain excluded from version control, while still preserving the evidence locally for that specific run.

How does the evidence system handle file cleanup after runs?

The framework does not automatically remove evidence files after test completion. Files remain in their designated directories (temporary or in-repo) until external cleanup processes remove them. The test step explicitly instructs agents not to delete evidence files, ensuring proof artifacts remain available for PR review and debugging purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}