How the AI Engineering Curriculum's Lesson Audit Validation System Works

The lesson audit validation system in rohitg00/ai-engineering-from-scratch is a Python-based static analyzer that automatically verifies every lesson directory for structural compliance, required documentation, and valid internal links, exiting with a non-zero status code when violations are detected to block merges in CI.

The rohitg00/ai-engineering-from-scratch repository maintains strict quality standards for its self-paced AI engineering curriculum through an automated lesson audit validation system. This validation layer ensures that every lesson in the phases/ directory contains mandatory documentation, functional code examples, and properly structured metadata before changes can be merged into the main branch.

Core Architecture and Entry Points

The validation logic resides primarily in scripts/audit_lessons.py, which serves as the main entry point for both local development checks and continuous integration enforcement. The script accepts optional flags to filter by phase, output machine-readable JSON, or enable strict mode.

python scripts/audit_lessons.py [--phase N] [--json] [--strict]

This command structure is defined at line 5 of the main script.

On execution, the validator performs a glob search across phases/**/ to discover all lesson directories containing docs/en.md files. Each parent folder of a discovered markdown file becomes a lesson candidate for validation.

The Per-Lesson Validation Pipeline

For every discovered lesson, the system invokes audit_lesson(audit, lesson_path) at line 214, passing a central Audit object that aggregates violations. The function executes a series of discrete checks, each registering an Issue when rules fail.

Directory Structure and Naming Compliance

Rule L001 enforces the <phase>/<lesson> naming convention through the check_lesson_dir_pattern function (line 85). The validator ensures each lesson directory follows the established organizational pattern within the phases hierarchy.

Documentation Requirements

Rule L002 and Rule L004 validate the presence and format of docs/en.md. The check_docs_en_md function (lines 100–115) verifies that the file exists, uses UTF-8 encoding, and begins with a top-level H1 header (# Title) at line 115.

Code Artifact Validation

Rule L005 ensures the code/ directory contains non-empty source or configuration files. The check_code_main function at line 126 confirms that every lesson includes executable reference material.

Quiz Schema Enforcement

Rule L006 validates quiz.json through the check_quiz function (lines 137–188). This check verifies that the file exists, contains valid JSON, and conforms to a strict schema requiring object entries with correct fields and exactly six questions per lesson.

Internal Link Resolution

Rule L010 validates Markdown hyperlinks using check_internal_links (lines 196–214). The system resolves both anchor links ([text](#anchor)) and cross-lesson references ([text](../other-lesson)) against the repository file system, utilizing the helper scripts/link_check.py referenced in the companion comment at line 25.

Issue Collection and Reporting

Each failed check creates an Issue object containing a rule code (e.g., L002), the lesson path, the offending file, and a human-readable message. The Audit object tracks the total count of lessons_checked and accumulates all discovered issues.

The render_report(audit) function (lines 225–235) generates a human-readable text summary of the validation run. When the --json flag is provided, the script emits a structured JSON payload (lines 257–263) containing the lesson count and a list of issue dictionaries suitable for programmatic consumption.

The script exits with status 1 when any issues are present, otherwise 0 (line 272). This exit code behavior enables CI systems to treat audit failures as blocking checks.

CI/CD Integration and Merge Enforcement

The GitHub Actions workflow defined in .github/workflows/curriculum.yml executes the audit on every push and pull request:

audit:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v3
    - name: Run lesson audit
      run: python scripts/audit_lessons.py

If the audit detects violations, the job fails immediately, preventing the pull request from merging until contributors resolve the highlighted issues—such as adding a missing quiz.json or correcting broken internal links. Once the audit passes, subsequent workflow jobs for README synchronization and site rebuilding proceed.

Running the Audit Locally

Developers can execute targeted audits during curriculum development to validate specific phases or generate machine-readable reports:

# Validate only phase 04 (Computer Vision) and output JSON

python scripts/audit_lessons.py --phase 04 --json > audit-report.json

This local execution mirrors the CI environment, allowing contributors to identify compliance issues before submitting pull requests.

Extending the Validation Rules

The modular architecture allows curriculum maintainers to add custom compliance checks by implementing new functions and registering them in the audit_lesson workflow. For example, to enforce a custom header format:

def check_custom_header(audit: Audit, lesson: Path) -> None:
    header = lesson / "docs" / "en.md"
    first_line = header.read_text().splitlines()[0]
    if not first_line.startswith("# "):

        audit.add("L999", lesson, header, "Missing required H1 header")

Adding this call within audit_lesson causes the new L999 rule to appear in reports alongside built-in checks, maintaining consistency with the existing issue code format.

Summary

• The lesson audit validation system is implemented in scripts/audit_lessons.py with helper utilities in scripts/link_check.py.
• It validates six specific compliance rules (L001–L006, L010) covering directory naming, documentation presence, code content, quiz schema, and link resolution.
• The system uses a central Audit object to collect Issue instances, exiting with code 1 when violations exist to block CI merges.
• JSON and text reporting modes support both human review and automated bot integrations.
• The validator is integrated into .github/workflows/curriculum.yml to enforce quality gates on every pull request.

Frequently Asked Questions

What happens if a lesson is missing the required quiz.json file?

The audit will fail with Rule L006, generating an issue in the report indicating that quiz.json is missing or invalid. The CI workflow will block the merge until the file is added with the correct schema, including exactly six questions with properly formatted object entries.

How does the audit system handle internal links between lessons?

Rule L010 invokes check_internal_links (lines 196–214) to resolve Markdown hyperlinks against the repository filesystem. The validator checks both anchor references within files and relative paths to other lessons, utilizing the link_check.py helper to ensure all internal navigation targets exist.

Can I run the audit on a single lesson instead of the entire curriculum?

Yes. Use the --phase flag to limit validation to a specific phase directory, or filter the output manually. While there is no single-lesson flag, targeting a phase reduces the scope significantly. The --json flag outputs structured data that can be filtered by lesson path if needed.

Where does the audit store its results when run in CI?

The audit outputs results to stdout, which GitHub Actions captures in the workflow logs. When using the --json flag in CI, the JSON report is printed to stdout and can be redirected to a file or parsed by subsequent workflow steps for PR comments or artifact storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}