# Building an Agent Workbench with Verification Gates: A Complete Implementation Guide

> Implement an agent workbench with verification gates using the ai-engineering-from-scratch repo. Ensure reliable agent execution with deterministic criteria, scope boundaries, and custom rules.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-15

---

**The Agent Workbench from the rohitg00/ai-engineering-from-scratch repository provides a versioned, portable pack that bundles documentation, JSON schemas, and Python scripts to enforce deterministic verification gates—acceptance criteria, scope boundaries, and custom rules—ensuring reliable agent execution across any repository.**

The Phase 14 capstone (Lesson 42) demonstrates how to construct a production-ready agent workbench that encapsulates the seven work-bench surfaces of the curriculum into a single, deployable unit. By implementing verification gates at critical execution points, this system prevents partial failures and unauthorized modifications while maintaining deterministic feedback logs for auditability.

## Pack Architecture and Components

The workbench ships as a **pack**—a self-contained directory that can be dropped into any repository. The pack organizes artefacts into four distinct categories that separate human-readable contracts from machine-enforced constraints.

### Documentation Layer (Docs)

The `docs/` directory contains human-readable contracts that govern agent behavior:

- [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) – Central specification defining agent capabilities and constraints
- [`agent-rules.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/agent-rules.md) – Operational rules the agent must obey during execution
- [`reliability-policy.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/reliability-policy.md) – uptime and retry expectations
- [`hand-off-protocol.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/hand-off-protocol.md) – Standards for transferring state between agents
- [`reviewer-rubric.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/reviewer-rubric.md) – Evaluation criteria for code review agents

### Schema Contracts (Schemas)

JSON Schema files in `schemas/` define the shape of runtime data:

- [`agent_state.schema.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/agent_state.schema.json) – Validates the persisted agent state structure
- [`task_board.schema.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/task_board.schema.json) – Defines the task board format used by the verification system
- [`scope_contract.schema.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scope_contract.schema.json) – Specifies allowed write operations and forbidden paths

### Runtime Scripts (Scripts)

Python utilities in `scripts/` handle execution and validation:

- [`init_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/init_agent.py) – Probes runtime environment, dependencies, and state freshness (lines 53-65)
- [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py) – Executes shell commands while capturing deterministic stdout/stderr feedback
- [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py) – Enforces the three verification gates (lines 32-86)
- [`generate_handoff.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/generate_handoff.py) – Produces artefacts for downstream agent consumption

### Installation and Versioning

The [`bin/install.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/bin/install.sh) script provides idempotent installation, copying the pack into a target repository and creating a `.workbench-version` marker. The `VERSION` file follows semantic versioning: **major** for schema/script changes, **minor** for behavioral changes, and **patch** for documentation updates.

## The Three Verification Gates

The [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py) script implements deterministic checks that block execution when violations occur. Each gate runs sequentially, and any *block* severity finding causes immediate exit with non-zero status (lines 83-86).

### Acceptance-Criteria Gate

This gate ensures every command listed in a task's `acceptance_criteria` executed successfully. The script reads the task definition and cross-references it against the `feedback_record.jsonl` log produced by [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py). If a required command never ran or exited with a non-zero code, the gate returns a block-level finding (lines 32-46).

### Scope-Boundary Gate

Before allowing writes to persist, this gate analyzes the scope report for any *forbidden* or *off-scope* modifications. The check references [`scope_contract.schema.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scope_contract.schema.json) to validate that file system operations stayed within authorized boundaries (lines 49-57).

### Custom-Rule Gate

The final gate validates custom rule-reports generated during execution. These rules might enforce linting standards, security constraints, or project-specific policies. Any failed rule report triggers a block severity finding (lines 60-63).

## Building and Installing the Workbench

Assemble the pack using the lesson's build script, then deploy it to target repositories.

### Step 1: Assemble the Pack

Run the lesson build script to generate the distributable pack:

```bash
cd phases/14-agent-engineering/42-agent-workbench-capstone
python3 code/main.py

```

This idempotent script copies schemas, scripts, and docs into `outputs/agent-workbench-pack/` and prints a tree view of the generated structure. Re-running produces no duplicate files.

### Step 2: Install into Target Repository

Execute the installer in your project directory:

```bash
cd /path/to/target-repo
/path/to/agent-workbench-pack/bin/install.sh

```

The installer checks for an existing `.workbench-version` marker and refuses to overwrite unless you provide the `--force` flag. It copies the directory tree, writes the version marker, and optionally adds CI hooks if a `.github/workflows/` folder exists.

### Step 3: Initialize the Agent

Run the initialization probe to validate the environment:

```bash
python3 scripts/init_agent.py

```

This script examines runtime availability, dependency versions, required environment variables, and the freshness of [`agent_state.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/agent_state.json). It emits [`init_report.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/init_report.json) containing a structured summary of the environment state.

## Runtime Execution and Verification

Execute tasks through the feedback wrapper to ensure deterministic logging, then verify compliance before completing the task.

### Record Task Execution

Use [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py) to capture command output with truncation for determinism:

```bash
python3 scripts/run_with_feedback.py --note "install deps" pip install -r requirements.txt
python3 scripts/run_with_feedback.py --note "run tests" pytest

```

The script appends structured records to `feedback_record.jsonl`, including truncated stdout/stderr (using `deterministic_tail`), exit codes, and timing metadata.

### Verify Task Completion

After execution, validate against the three gates:

```bash
python3 scripts/verify_agent.py demo-task-id

```

The command produces [`outputs/verification/demo-task-id.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/outputs/verification/demo-task-id.json):

```json
{
  "task_id": "demo-task-id",
  "passed": false,
  "findings": [
    {"code":"acceptance.missing","severity":"block","detail":"never ran: pytest"},
    {"code":"rule.failed","severity":"block","detail":"rule failed: lint"}
  ]
}

```

If all gates pass, the verification JSON contains `"passed": true` and zero block-level findings.

### Generate Hand-Off Artefacts

Finally, create consumable outputs for downstream agents:

```bash
python3 scripts/generate_handoff.py

```

This updates the task board and produces migration artefacts according to the hand-off protocol defined in [`hand-off-protocol.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/hand-off-protocol.md).

## Key Architectural Decisions

**Deterministic Logging** – Both [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py) and [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py) truncate long output streams to fixed head/tail sizes, ensuring reproducible logs across different terminal widths and execution environments.

**Separation of Concerns** – Schemas describe *what* the state must contain; scripts enforce *how* it is built and validated. This allows independent updates to validation logic without changing state structures.

**Version-Driven Migration** – The `VERSION` file serves as the single source of truth for compatibility. Scripts read `.workbench-version` to guarantee consistent behavior across upgrades, preventing schema mismatches between the workbench and running agents.

## Summary

- The **Agent Workbench** bundles docs, schemas, and scripts into a versioned pack that installs idempotently into any repository using [`bin/install.sh`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/bin/install.sh).
- **Three verification gates**—acceptance criteria, scope boundaries, and custom rules—enforce deterministic safety checks via [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py) lines 32-86.
- **Deterministic feedback** captured by [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py) enables reproducible debugging and audit trails through `feedback_record.jsonl`.
- **Semantic versioning** in the `VERSION` file governs migration policy and compatibility guarantees.

## Frequently Asked Questions

### How do I override an existing workbench installation?

Run the installer with the `--force` flag: `./install.sh --force`. This overwrites the existing pack and updates the `.workbench-version` marker. Without this flag, the installer refuses to overwrite to prevent accidental loss of custom state or local modifications.

### What happens when a verification gate fails?

When any gate returns a finding with `"severity": "block"`, [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py) exits with a non-zero status code (lines 83-86) and writes a JSON report detailing the specific violations. The task is considered incomplete, and hand-off artefacts are not generated until you resolve the blocking findings and re-run verification.

### Can I add custom verification rules to the workbench?

Yes. The custom-rule gate (lines 60-63 in [`verify_agent.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/verify_agent.py)) reads rule reports from the execution environment. Implement custom validators as separate scripts that output to the reports directory in the format expected by [`scope_contract.schema.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scope_contract.schema.json). The verification script aggregates these reports and treats any failure as a block-level finding.

### How does the workbench ensure deterministic feedback across different machines?

The [`run_with_feedback.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/run_with_feedback.py) script implements output truncation using `deterministic_tail`, which captures fixed head and tail portions of stdout/stderr. This prevents variance caused by different terminal widths, buffering behaviors, or verbose logging levels, ensuring that `feedback_record.jsonl` remains identical across runs when given identical inputs.