# How the Testing-Reality-Checker Determines Production Readiness in Agency-Agents

> Discover how the testing-reality-checker determines production readiness. Learn about its strict three-phase validation process for agency-agents to ensure your projects are fully prepared.

- Repository: [Michael Sitarzewski/agency-agents](https://github.com/msitarzewski/agency-agents)
- Tags: deep-dive
- Published: 2026-03-09

---

**The testing-reality-checker defaults to a "NEEDS WORK" state and only flips to "READY" after passing a strict three-phase validation that requires overwhelming, evidence-based proof across reality-check commands, QA cross-validation, and end-to-end system testing.**

The `testing-reality-checker` in the `msitarzewski/agency-agents` repository acts as the final hard gate before deployment. Unlike typical checklists that assume success until failure is found, this agent starts from a position of skepticism and demands concrete proof that the system meets production standards. Its workflow is defined in [`testing/testing-reality-checker.md`](https://github.com/msitarzewski/agency-agents/blob/main/testing/testing-reality-checker.md) and orchestrated through [`specialized/agents-orchestrator.md`](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md).

## The Three-Phase Validation Workflow

The testing-reality-checker executes a sequential validation pipeline that moves from raw evidence gathering to holistic system verification.

### Phase 1: Reality-Check Commands and Evidence Gathering

The agent begins by validating what actually exists in the codebase rather than trusting documentation. It executes specific shell commands to gather raw artefacts:

- **List built files**: `ls -la resources/views/ || ls -la *.html` confirms the concrete UI stack (Laravel Blade, plain HTML, etc.)
- **Search for premium UI claims**: `grep -r "luxury\|premium\|glass\|morphism"` detects marketing assertions that must be substantiated
- **Capture screenshots**: `./qa-playwright-capture.sh http://localhost:8000 public/qa-screenshots` generates device-specific PNGs and a [`test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/test-results.json) payload containing load times, errors, and interaction status
- **Review artefacts**: `ls -la public/qa-screenshots/ && cat public/qa-screenshots/test-results.json` provides the agent with concrete UI and performance metrics

These commands ensure the agent works from observable reality rather than developer claims, as defined in lines 39-55 of [`testing/testing-reality-checker.md`](https://github.com/msitarzewski/agency-agents/blob/main/testing/testing-reality-checker.md).

### Phase 2: QA Cross-Validation

The agent cross-checks the gathered evidence against the earlier QA agent's report. It reads the QA findings and compares the screenshots and [`test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/test-results.json) data with the QA-reported issues. Any mismatch—such as a bug still visible in the screenshots—triggers an automatic **"FAIL"** flag for that item.

This step ensures that previous automated QA cannot hide defects once the system is fully assembled, as implemented in lines 56-61.

### Phase 3: End-to-End System Validation

Finally, the checker evaluates complete user journeys across devices. It examines desktop, tablet, and mobile screenshots for layout consistency. It compares before-and-after screenshots of interaction sequences (navigation clicks, form fills, accordion toggles). It validates performance numbers from [`test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/test-results.json), requiring load times under 3 seconds and error-free operation.

If every journey passes, the agent records a **PASS**; otherwise it logs a **FAIL** with explicit evidence links, as detailed in lines 62-67.

## Decision Logic and Production Readiness Criteria

After completing the three phases, the testing-reality-checker builds a **Production-Readiness Report** using the markdown template beginning at line 40. The report contains an **Overall Quality Rating** (C+, B-, etc.) derived from passed versus failed checks.

The **Production Readiness** field defaults to **"NEEDS WORK"** and only changes to **"READY"** when all of the following conditions are met:

1. **No automatic fail triggers** are present (no fantasy claims, missing screenshots, or performance exceeding 3 seconds), as defined in lines 22-27
2. Every **spec-vs-implementation** comparison shows **PASS** with concrete screenshot evidence
3. All **QA-identified issues** are resolved, verified by the cross-validation step
4. The **overall rating** meets or exceeds the predefined threshold (typically **B-** or better)

If any condition fails, the final status remains **"NEEDS WORK"** or **"FAILED"** for critical blockers. The report's "Deployment Readiness Assessment" line makes this explicit, as shown in lines 77-80.

## Integration in the CI/CD Pipeline

The [`specialized/agents-orchestrator.md`](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md) automatically spawns the testing-reality-checker as the **final step** of the autonomous development workflow at line 103. Only when the reality-checker returns **"READY"** does the orchestrator proceed to release or deployment. This integration makes the check a **hard quality gate** for the entire system, ensuring that no code reaches production without passing the evidence-based validation.

## Code Examples

Below are concise snippets showing how to trigger the check and interpret its results.

```bash

# Spawn the reality-checker via the orchestrator (recommended)

./run-orchestrator.sh project-specs/my-app-setup.md

# Directly invoke the checker for ad-hoc validation

cd testing
./testing-reality-checker.sh   # wraps the three steps described above

```

```python

# Example: programmatically parse the final report for the readiness flag

import toml, pathlib, re

report_path = pathlib.Path("public/qa-screenshots/report.md")
text = report_path.read_text()

# Extract the "Production Readiness" line

ready = re.search(r"Production Readiness:\s+(\w+)", text)
status = ready.group(1) if ready else "UNKNOWN"
print(f"⚙️  Production readiness: {status}")

```

```markdown
<!-- Sample excerpt from the generated report -->

## 📈 Success Metrics for Next Iteration

**What Needs Improvement**: …

## 🔄 Deployment Readiness Assessment

**Status**: **NEEDS WORK** (default unless overwhelming evidence supports ready)

```

## Summary

- The **testing-reality-checker** defaults to **"NEEDS WORK"** and requires overwhelming evidence to declare **"READY"**.
- It executes a **three-phase validation**: reality-check commands, QA cross-validation, and end-to-end system testing.
- **Automatic fail triggers** (fantasy claims, missing screenshots, slow performance) immediately block production readiness.
- The checker serves as a **hard quality gate** in the CI/CD pipeline, with the orchestrator only proceeding to deployment upon receiving a **"READY"** status.
- All evidence is captured in [`public/qa-screenshots/test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/public/qa-screenshots/test-results.json) and cross-referenced against the QA agent's findings.

## Frequently Asked Questions

### What triggers an automatic "FAIL" in the testing-reality-checker?

The testing-reality-checker flags an automatic **FAIL** when it detects **fantasy claims** (unsubstantiated marketing language like "luxury" or "premium" without evidence), **missing screenshots** in the `public/qa-screenshots/` directory, or **performance metrics** exceeding the 3-second load-time threshold defined in [`test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/test-results.json). These triggers are defined in lines 22-27 of [`testing/testing-reality-checker.md`](https://github.com/msitarzewski/agency-agents/blob/main/testing/testing-reality-checker.md).

### How does the reality-checker verify that QA issues are actually resolved?

During **Phase 2 (QA Cross-Validation)**, the agent reads the previous QA agent's report and compares it against the current screenshots and [`test-results.json`](https://github.com/msitarzewski/agency-agents/blob/main/test-results.json) data. If the QA report identified a bug that still appears in the fresh screenshots, or if the performance metrics still show errors, the checker flags a **FAIL** for that specific item. This ensures that earlier QA approvals cannot mask persistent defects.

### Can the testing-reality-checker be run independently of the orchestrator?

Yes, while the [`specialized/agents-orchestrator.md`](https://github.com/msitarzewski/agency-agents/blob/main/specialized/agents-orchestrator.md) automatically spawns the reality-checker as the final pipeline step at line 103, you can invoke it directly for ad-hoc validation. Navigate to the `testing` directory and execute [`./testing-reality-checker.sh`](https://github.com/msitarzewski/agency-agents/blob/main/./testing-reality-checker.sh), which wraps the three validation phases (reality-check commands, QA cross-validation, and end-to-end system testing) described in the workflow documentation.

### What quality rating must a project achieve to be considered "READY"?

The project must achieve an **Overall Quality Rating** of **B-** or better, though this threshold can be configured. More importantly, the status only flips from the default **"NEEDS WORK"** to **"READY"** when all automatic fail triggers are absent, every spec-vs-implementation comparison shows **PASS** with screenshot evidence, and all QA-identified issues are resolved. The rating is derived from the count of passed versus failed checks in the final report.