Building an Agent Workbench with Verification Gates: A Complete Implementation Guide

The Agent Workbench from the rohitg00/ai-engineering-from-scratch repository provides a versioned, portable pack that bundles documentation, JSON schemas, and Python scripts to enforce deterministic verification gates—acceptance criteria, scope boundaries, and custom rules—ensuring reliable agent execution across any repository.

The Phase 14 capstone (Lesson 42) demonstrates how to construct a production-ready agent workbench that encapsulates the seven work-bench surfaces of the curriculum into a single, deployable unit. By implementing verification gates at critical execution points, this system prevents partial failures and unauthorized modifications while maintaining deterministic feedback logs for auditability.

Pack Architecture and Components

The workbench ships as a pack—a self-contained directory that can be dropped into any repository. The pack organizes artefacts into four distinct categories that separate human-readable contracts from machine-enforced constraints.

Documentation Layer (Docs)

The docs/ directory contains human-readable contracts that govern agent behavior:

• AGENTS.md – Central specification defining agent capabilities and constraints
• agent-rules.md – Operational rules the agent must obey during execution
• reliability-policy.md – uptime and retry expectations
• hand-off-protocol.md – Standards for transferring state between agents
• reviewer-rubric.md – Evaluation criteria for code review agents

Schema Contracts (Schemas)

JSON Schema files in schemas/ define the shape of runtime data:

• agent_state.schema.json – Validates the persisted agent state structure
• task_board.schema.json – Defines the task board format used by the verification system
• scope_contract.schema.json – Specifies allowed write operations and forbidden paths

Runtime Scripts (Scripts)

Python utilities in scripts/ handle execution and validation:

• init_agent.py – Probes runtime environment, dependencies, and state freshness (lines 53-65)
• run_with_feedback.py – Executes shell commands while capturing deterministic stdout/stderr feedback
• verify_agent.py – Enforces the three verification gates (lines 32-86)
• generate_handoff.py – Produces artefacts for downstream agent consumption

Installation and Versioning

The bin/install.sh script provides idempotent installation, copying the pack into a target repository and creating a .workbench-version marker. The VERSION file follows semantic versioning: major for schema/script changes, minor for behavioral changes, and patch for documentation updates.

The Three Verification Gates

The verify_agent.py script implements deterministic checks that block execution when violations occur. Each gate runs sequentially, and any block severity finding causes immediate exit with non-zero status (lines 83-86).

Acceptance-Criteria Gate

This gate ensures every command listed in a task's acceptance_criteria executed successfully. The script reads the task definition and cross-references it against the feedback_record.jsonl log produced by run_with_feedback.py. If a required command never ran or exited with a non-zero code, the gate returns a block-level finding (lines 32-46).

Scope-Boundary Gate

Before allowing writes to persist, this gate analyzes the scope report for any forbidden or off-scope modifications. The check references scope_contract.schema.json to validate that file system operations stayed within authorized boundaries (lines 49-57).

Custom-Rule Gate

The final gate validates custom rule-reports generated during execution. These rules might enforce linting standards, security constraints, or project-specific policies. Any failed rule report triggers a block severity finding (lines 60-63).

Building and Installing the Workbench

Assemble the pack using the lesson's build script, then deploy it to target repositories.

Step 1: Assemble the Pack

Run the lesson build script to generate the distributable pack:

cd phases/14-agent-engineering/42-agent-workbench-capstone
python3 code/main.py

This idempotent script copies schemas, scripts, and docs into outputs/agent-workbench-pack/ and prints a tree view of the generated structure. Re-running produces no duplicate files.

Step 2: Install into Target Repository

Execute the installer in your project directory:

cd /path/to/target-repo
/path/to/agent-workbench-pack/bin/install.sh

The installer checks for an existing .workbench-version marker and refuses to overwrite unless you provide the --force flag. It copies the directory tree, writes the version marker, and optionally adds CI hooks if a .github/workflows/ folder exists.

Step 3: Initialize the Agent

Run the initialization probe to validate the environment:

python3 scripts/init_agent.py

This script examines runtime availability, dependency versions, required environment variables, and the freshness of agent_state.json. It emits init_report.json containing a structured summary of the environment state.

Runtime Execution and Verification

Execute tasks through the feedback wrapper to ensure deterministic logging, then verify compliance before completing the task.

Record Task Execution

Use run_with_feedback.py to capture command output with truncation for determinism:

python3 scripts/run_with_feedback.py --note "install deps" pip install -r requirements.txt
python3 scripts/run_with_feedback.py --note "run tests" pytest

The script appends structured records to feedback_record.jsonl, including truncated stdout/stderr (using deterministic_tail), exit codes, and timing metadata.

Verify Task Completion

After execution, validate against the three gates:

python3 scripts/verify_agent.py demo-task-id

The command produces outputs/verification/demo-task-id.json:

{
  "task_id": "demo-task-id",
  "passed": false,
  "findings": [
    {"code":"acceptance.missing","severity":"block","detail":"never ran: pytest"},
    {"code":"rule.failed","severity":"block","detail":"rule failed: lint"}
  ]
}

If all gates pass, the verification JSON contains "passed": true and zero block-level findings.

Generate Hand-Off Artefacts

Finally, create consumable outputs for downstream agents:

python3 scripts/generate_handoff.py

This updates the task board and produces migration artefacts according to the hand-off protocol defined in hand-off-protocol.md.

Key Architectural Decisions

Deterministic Logging – Both run_with_feedback.py and verify_agent.py truncate long output streams to fixed head/tail sizes, ensuring reproducible logs across different terminal widths and execution environments.

Separation of Concerns – Schemas describe what the state must contain; scripts enforce how it is built and validated. This allows independent updates to validation logic without changing state structures.

Version-Driven Migration – The VERSION file serves as the single source of truth for compatibility. Scripts read .workbench-version to guarantee consistent behavior across upgrades, preventing schema mismatches between the workbench and running agents.

Summary

• The Agent Workbench bundles docs, schemas, and scripts into a versioned pack that installs idempotently into any repository using bin/install.sh.
• Three verification gates—acceptance criteria, scope boundaries, and custom rules—enforce deterministic safety checks via verify_agent.py lines 32-86.
• Deterministic feedback captured by run_with_feedback.py enables reproducible debugging and audit trails through feedback_record.jsonl.
• Semantic versioning in the VERSION file governs migration policy and compatibility guarantees.

Frequently Asked Questions

How do I override an existing workbench installation?

Run the installer with the --force flag: ./install.sh --force. This overwrites the existing pack and updates the .workbench-version marker. Without this flag, the installer refuses to overwrite to prevent accidental loss of custom state or local modifications.

What happens when a verification gate fails?

When any gate returns a finding with "severity": "block", verify_agent.py exits with a non-zero status code (lines 83-86) and writes a JSON report detailing the specific violations. The task is considered incomplete, and hand-off artefacts are not generated until you resolve the blocking findings and re-run verification.

Can I add custom verification rules to the workbench?

Yes. The custom-rule gate (lines 60-63 in verify_agent.py) reads rule reports from the execution environment. Implement custom validators as separate scripts that output to the reports directory in the format expected by scope_contract.schema.json. The verification script aggregates these reports and treats any failure as a block-level finding.

How does the workbench ensure deterministic feedback across different machines?

The run_with_feedback.py script implements output truncation using deterministic_tail, which captures fixed head and tail portions of stdout/stderr. This prevents variance caused by different terminal widths, buffering behaviors, or verbose logging levels, ensuring that feedback_record.jsonl remains identical across runs when given identical inputs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}