# How test_agent_readiness.py Validates That the Learning Skills Mirror Stays Complete

> Discover how test_agent_readiness.py validates the AI Engineering from Scratch learning skills mirror using contract tests against vercel.json, OpenAPI, SEO manifests, and HTML templates for complete, linked, and enriched lessons.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-09-04

---

**[`test_agent_readiness.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/test_agent_readiness.py) ensures the AI Engineering from Scratch learning skills mirror remains complete by running contract tests against [`vercel.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/vercel.json), OpenAPI specifications, SEO manifests, and HTML templates to verify every lesson and certification is present, correctly linked, and semantically enriched.**

The `rohitg00/ai-engineering-from-scratch` repository maintains a static site that serves as the canonical *learning skills mirror*—a complete representation of the curriculum including lessons, certifications, and supporting infrastructure. To prevent content drift or broken discoverability as the codebase evolves, the repository includes [`scripts/test_agent_readiness.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/test_agent_readiness.py), a comprehensive contract test suite that validates the generated site structure through explicit assertions against configuration files, API contracts, and semantic markup.

## Vercel Configuration and Routing Contracts

Before validating content, the script verifies that the deployment infrastructure correctly handles content negotiation and legacy URLs.

### Rewrite and Header Validation

The test reads [`vercel.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/vercel.json) (lines 66‑74) to assert that markdown negotiation rewrites exist and that the `accept` header is properly configured for content negotiation (lines 79‑89). This ensures that AI agents and browsers can request specific content formats and receive appropriate responses according to the deployment configuration.

### Legacy Redirect Handling

The script calls `assert_legacy_redirect` for [`/lesson.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main//lesson.html) and [`/certification.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main//certification.html) (lines 121‑124), verifying that deprecated routes return a **308 Permanent Redirect** status with a valid `Location` header. This preserves link equity while guiding traffic to modern URL patterns, ensuring the mirror maintains valid inbound links from external sources.

## Static Content Completeness Checks

The learning skills mirror must contain substantive content for every curriculum component without orphaned pages or missing assets.

### Core HTML Pages

The script verifies that essential pages—including [`developer.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/developer.html), [`contact.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/contact.html), [`privacy.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/privacy.html), [`404.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/404.html), and [`openapi.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/openapi.json)—exist in the `site/` directory (lines 97‑99). It further asserts that trust pages contain the phrase *“AI Engineering from Scratch”* and exceed 500 words of content (lines 100‑104), ensuring substantive information presence rather than placeholder content.

### SEO Manifests and Sitemap Consistency

To guarantee discoverability, the test loads [`lesson-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson-seo.json) and [`certification-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/certification-seo.json) (lines 124‑146), asserting that at least **500 lessons** and **4 certification tracks** are documented. Each entry must have a canonical URL matching its key path (lines 128‑134), preventing duplicate content issues.

The script then parses [`site/sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/sitemap.xml) (lines 148‑158) to confirm that the URL count matches the manifest lengths, specifically validating `/lesson?path=` and `/certification?id=` patterns while ensuring no legacy `.html?` URLs remain in the index that would confuse search crawlers.

## API Contract Validation

The mirror exposes both human-readable HTML and machine-readable API contracts that must remain consistent for agent consumption.

### OpenAPI Specification Compliance

Loading [`site/openapi.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/openapi.json) (lines 105‑108), the test verifies the spec version starts with `3.` and that required paths—including `/lesson`, `/certification`, and related endpoints—are present (lines 110‑116). It validates that each HTTP method (`GET`, `HEAD`) returns expected status codes and handles redirects correctly (lines 38‑46, 52‑62), ensuring API consumers receive predictable responses.

### Public HTML Routes and Parameter Handling

Using `assert_public_html_route` (lines 117‑120), the script inspects the `/lesson` and `/certification` endpoints to confirm they expose HTML for both success (200) and error (404) states. This validation includes checking query string parameters and ensuring the response content types support AI agent parsing.

## Semantic Markup and Structured Data

Beyond structural completeness, the mirror must provide rich semantic context for AI agents and search engines to understand the curriculum topology.

### HTML Template Markers

The script scans [`site/lesson.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/lesson.html) and [`site/certification.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/certification.html) for `AIFS:…:START` markers (lines 160‑170), asserting exactly one occurrence of each placeholder. These markers enable automated injection of SEO metadata during the build process, and their absence would indicate template corruption that breaks metadata generation.

### JSON-LD Schema Validation

Extracting JSON-LD from [`index.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/index.html) and [`about.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/about.html) (lines 176‑189), the test validates the presence of correct `@type` declarations including `WebSite`, `Course`, and `AboutPage` (lines 178‑190). It verifies entity IDs and relationships to ensure structured data compliance with Schema.org standards.

### Identity Sanity Checks

To prevent schema pollution that could confuse AI agents, the script serializes both JSON-LD schemas and asserts against a whitelist of disallowed fields (lines 191‑203), ensuring extraneous entities like `Person` or `Organization` do not appear in the combined structured data graph unless explicitly permitted.

## Infrastructure and Error Handling

The validation extends to serverless functions and error pages that support the mirror's reliability and cache behavior.

### API Endpoint Verification

The script confirms the existence of critical server-side scripts: [`api/lesson.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/api/lesson.js), [`api/certification.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/api/certification.js), and the auxiliary test [`scripts/test_seo_routes.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/test_seo_routes.js) (lines 172‑174). It specifically verifies that [`api/markdown.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/api/markdown.js) includes a `Vary` header for proper cache handling (lines 207‑208), ensuring content negotiation works correctly at the edge.

### 404 Page Integrity

Checking [`site/404.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/404.html) (lines 205‑206), the test ensures error pages contain references to [`llms.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/llms.txt) and [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml), providing fallback navigation for agents and users encountering missing content rather than serving dead ends.

## Running the Contract Test Locally

Execute the readiness validation from the repository root to verify your local build before deployment:

```bash
python3 scripts/test_agent_readiness.py

```

For integration with custom tooling or build scripts, import the test suite programmatically:

```python
from scripts.test_agent_readiness import main

if __name__ == "__main__":
    # Raises AssertionError if any contract is broken

    main()

```

## Integrating with CI/CD Pipelines

Automate validation in GitHub Actions to block deployments that would break the learning skills mirror:

```yaml

# .github/workflows/agent-readiness.yml

jobs:
  readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Python deps
        run: python -m pip install --upgrade pip
      - name: Run readiness contract
        run: python3 scripts/test_agent_readiness.py

```

If any assertion fails, the script raises an `AssertionError`, causing the CI job to fail and preventing deployment of an incomplete or misconfigured mirror.

## Summary

- **[`test_agent_readiness.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/test_agent_readiness.py)** serves as a comprehensive contract test for the `ai-engineering-from-scratch` learning skills mirror, validating both content completeness and technical correctness through 13 distinct validation stages.
- The script asserts against **[`vercel.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/vercel.json)** (lines 66‑89) to ensure proper routing, redirects, and content negotiation headers are configured for edge deployment.
- It verifies the presence of **500+ lessons** and **4+ certification tracks** through [`lesson-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson-seo.json) and [`certification-seo.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/certification-seo.json) (lines 124‑146), cross-referencing [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml) (lines 148‑158) for consistency and absence of legacy URLs.
- **OpenAPI compliance** is enforced by checking [`site/openapi.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/openapi.json) for correct spec versions, required paths, and response codes (lines 105‑116), ensuring machine-readable contracts remain valid.
- **Semantic richness** is validated through HTML template markers (lines 160‑170), JSON-LD structured data extraction (lines 176‑190), and identity sanity checks (lines 191‑203) that prevent schema pollution.
- Any validation failure raises an **`AssertionError`**, ensuring CI/CD pipelines block deployments that would result in an incomplete learning skills mirror.

## Frequently Asked Questions

### What triggers a failure in test_agent_readiness.py?

The script raises an `AssertionError` whenever a contract is violated, including missing SEO manifests that drop below the 500-lesson threshold, incorrect OpenAPI spec versions, absent HTML template markers, or legacy URLs remaining in [`sitemap.xml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/sitemap.xml). Each check includes specific line references (e.g., lines 128‑134 for canonical URL patterns) to facilitate rapid debugging of the specific contract breach.

### How does the script validate the OpenAPI specification?

The test loads [`site/openapi.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/openapi.json) and asserts that the version string starts with `3.` (lines 105‑108). It then verifies required paths such as `/lesson` and `/certification` exist (lines 110‑116) and validates that each HTTP method returns expected status codes and handles redirects according to the specification (lines 38‑46, 52‑62), ensuring the API contract remains stable for consuming agents.

### Why does the test check for specific HTML template markers?

The script scans [`site/lesson.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/lesson.html) and [`site/certification.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/certification.html) for `AIFS:…:START` markers (lines 160‑170) to ensure build-time SEO injection points remain intact. Asserting exactly one occurrence per marker prevents template corruption that would break metadata generation, ensuring the learning skills mirror maintains machine-readable descriptions for each curriculum component.

### Can I run test_agent_readiness.py outside of CI environments?

Yes. The script functions as a standalone Python executable with no external dependencies beyond the standard library. Run `python3 scripts/test_agent_readiness.py` from the repository root for local validation, or import the `main` function into custom tooling for programmatic contract testing against the generated static site.