How test_agent_readiness.py Validates That the Learning Skills Mirror Stays Complete

test_agent_readiness.py ensures the AI Engineering from Scratch learning skills mirror remains complete by running contract tests against vercel.json, OpenAPI specifications, SEO manifests, and HTML templates to verify every lesson and certification is present, correctly linked, and semantically enriched.

The rohitg00/ai-engineering-from-scratch repository maintains a static site that serves as the canonical learning skills mirror—a complete representation of the curriculum including lessons, certifications, and supporting infrastructure. To prevent content drift or broken discoverability as the codebase evolves, the repository includes scripts/test_agent_readiness.py, a comprehensive contract test suite that validates the generated site structure through explicit assertions against configuration files, API contracts, and semantic markup.

Vercel Configuration and Routing Contracts

Before validating content, the script verifies that the deployment infrastructure correctly handles content negotiation and legacy URLs.

Rewrite and Header Validation

The test reads vercel.json (lines 66‑74) to assert that markdown negotiation rewrites exist and that the accept header is properly configured for content negotiation (lines 79‑89). This ensures that AI agents and browsers can request specific content formats and receive appropriate responses according to the deployment configuration.

Legacy Redirect Handling

The script calls assert_legacy_redirect for /lesson.html and /certification.html (lines 121‑124), verifying that deprecated routes return a 308 Permanent Redirect status with a valid Location header. This preserves link equity while guiding traffic to modern URL patterns, ensuring the mirror maintains valid inbound links from external sources.

Static Content Completeness Checks

The learning skills mirror must contain substantive content for every curriculum component without orphaned pages or missing assets.

Core HTML Pages

The script verifies that essential pages—including developer.html, contact.html, privacy.html, 404.html, and openapi.json—exist in the site/ directory (lines 97‑99). It further asserts that trust pages contain the phrase “AI Engineering from Scratch” and exceed 500 words of content (lines 100‑104), ensuring substantive information presence rather than placeholder content.

SEO Manifests and Sitemap Consistency

To guarantee discoverability, the test loads lesson-seo.json and certification-seo.json (lines 124‑146), asserting that at least 500 lessons and 4 certification tracks are documented. Each entry must have a canonical URL matching its key path (lines 128‑134), preventing duplicate content issues.

The script then parses site/sitemap.xml (lines 148‑158) to confirm that the URL count matches the manifest lengths, specifically validating /lesson?path= and /certification?id= patterns while ensuring no legacy .html? URLs remain in the index that would confuse search crawlers.

API Contract Validation

The mirror exposes both human-readable HTML and machine-readable API contracts that must remain consistent for agent consumption.

OpenAPI Specification Compliance

Loading site/openapi.json (lines 105‑108), the test verifies the spec version starts with 3. and that required paths—including /lesson, /certification, and related endpoints—are present (lines 110‑116). It validates that each HTTP method (GET, HEAD) returns expected status codes and handles redirects correctly (lines 38‑46, 52‑62), ensuring API consumers receive predictable responses.

Public HTML Routes and Parameter Handling

Using assert_public_html_route (lines 117‑120), the script inspects the /lesson and /certification endpoints to confirm they expose HTML for both success (200) and error (404) states. This validation includes checking query string parameters and ensuring the response content types support AI agent parsing.

Semantic Markup and Structured Data

Beyond structural completeness, the mirror must provide rich semantic context for AI agents and search engines to understand the curriculum topology.

HTML Template Markers

The script scans site/lesson.html and site/certification.html for AIFS:…:START markers (lines 160‑170), asserting exactly one occurrence of each placeholder. These markers enable automated injection of SEO metadata during the build process, and their absence would indicate template corruption that breaks metadata generation.

JSON-LD Schema Validation

Extracting JSON-LD from index.html and about.html (lines 176‑189), the test validates the presence of correct @type declarations including WebSite, Course, and AboutPage (lines 178‑190). It verifies entity IDs and relationships to ensure structured data compliance with Schema.org standards.

Identity Sanity Checks

To prevent schema pollution that could confuse AI agents, the script serializes both JSON-LD schemas and asserts against a whitelist of disallowed fields (lines 191‑203), ensuring extraneous entities like Person or Organization do not appear in the combined structured data graph unless explicitly permitted.

Infrastructure and Error Handling

The validation extends to serverless functions and error pages that support the mirror's reliability and cache behavior.

API Endpoint Verification

The script confirms the existence of critical server-side scripts: api/lesson.js, api/certification.js, and the auxiliary test scripts/test_seo_routes.js (lines 172‑174). It specifically verifies that api/markdown.js includes a Vary header for proper cache handling (lines 207‑208), ensuring content negotiation works correctly at the edge.

404 Page Integrity

Checking site/404.html (lines 205‑206), the test ensures error pages contain references to llms.txt and sitemap.xml, providing fallback navigation for agents and users encountering missing content rather than serving dead ends.

Running the Contract Test Locally

Execute the readiness validation from the repository root to verify your local build before deployment:

python3 scripts/test_agent_readiness.py

For integration with custom tooling or build scripts, import the test suite programmatically:

from scripts.test_agent_readiness import main

if __name__ == "__main__":
    # Raises AssertionError if any contract is broken

    main()

Integrating with CI/CD Pipelines

Automate validation in GitHub Actions to block deployments that would break the learning skills mirror:

# .github/workflows/agent-readiness.yml

jobs:
  readiness:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Python deps
        run: python -m pip install --upgrade pip
      - name: Run readiness contract
        run: python3 scripts/test_agent_readiness.py

If any assertion fails, the script raises an AssertionError, causing the CI job to fail and preventing deployment of an incomplete or misconfigured mirror.

Summary

• test_agent_readiness.py serves as a comprehensive contract test for the ai-engineering-from-scratch learning skills mirror, validating both content completeness and technical correctness through 13 distinct validation stages.
• The script asserts against vercel.json (lines 66‑89) to ensure proper routing, redirects, and content negotiation headers are configured for edge deployment.
• It verifies the presence of 500+ lessons and 4+ certification tracks through lesson-seo.json and certification-seo.json (lines 124‑146), cross-referencing sitemap.xml (lines 148‑158) for consistency and absence of legacy URLs.
• OpenAPI compliance is enforced by checking site/openapi.json for correct spec versions, required paths, and response codes (lines 105‑116), ensuring machine-readable contracts remain valid.
• Semantic richness is validated through HTML template markers (lines 160‑170), JSON-LD structured data extraction (lines 176‑190), and identity sanity checks (lines 191‑203) that prevent schema pollution.
• Any validation failure raises an AssertionError, ensuring CI/CD pipelines block deployments that would result in an incomplete learning skills mirror.

Frequently Asked Questions

What triggers a failure in test_agent_readiness.py?

The script raises an AssertionError whenever a contract is violated, including missing SEO manifests that drop below the 500-lesson threshold, incorrect OpenAPI spec versions, absent HTML template markers, or legacy URLs remaining in sitemap.xml. Each check includes specific line references (e.g., lines 128‑134 for canonical URL patterns) to facilitate rapid debugging of the specific contract breach.

How does the script validate the OpenAPI specification?

The test loads site/openapi.json and asserts that the version string starts with 3. (lines 105‑108). It then verifies required paths such as /lesson and /certification exist (lines 110‑116) and validates that each HTTP method returns expected status codes and handles redirects according to the specification (lines 38‑46, 52‑62), ensuring the API contract remains stable for consuming agents.

Why does the test check for specific HTML template markers?

The script scans site/lesson.html and site/certification.html for AIFS:…:START markers (lines 160‑170) to ensure build-time SEO injection points remain intact. Asserting exactly one occurrence per marker prevents template corruption that would break metadata generation, ensuring the learning skills mirror maintains machine-readable descriptions for each curriculum component.

Can I run test_agent_readiness.py outside of CI environments?

Yes. The script functions as a standalone Python executable with no external dependencies beyond the standard library. Run python3 scripts/test_agent_readiness.py from the repository root for local validation, or import the main function into custom tooling for programmatic contract testing against the generated static site.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}