How to Run Tests for a Specific Lesson Using the Stdlib Test Runners in AI Engineering From Scratch

To run tests for a specific lesson in the ai-engineering-from-scratch curriculum, navigate to the lesson directory and execute python3 -m unittest discover -s code/tests -v from the lesson root, or use the scripts/lesson_run.py helper for automated syntax checking and execution.

The ai-engineering-from-scratch repository enforces a strict "stdlib-only" testing policy that prohibits external frameworks like pytest or Jest. Each lesson contains unit tests designed to run with the language's standard library runner, ensuring zero-dependency validation across the curriculum.

Locating Lesson Tests in the Repository

Each lesson lives under the path phases/<phase-number>-<phase-slug>/<lesson-number>-<lesson-slug>/. The executable code resides in the code/ subdirectory, while unit tests are stored in code/tests/ (or adjacent to it). For example, a reinforcement learning lesson on Markov Decision Processes would be located at phases/09-reinforcement-learning/01-mdps-states-actions-rewards/.

To verify the test structure exists:

cd phases/09-reinforcement-learning/01-mdps-states-actions-rewards
ls code/tests

Expected output includes test_*.py files such as test_main.py or test_utils.py.

Running Tests with the Stdlib Runner

According to the official guidance in AGENTS.md (lines 111-112), all test suites must run via the language's stdlib runner. This design choice eliminates external dependencies and ensures lessons remain portable across environments.

Python unittest Discovery

For Python lessons, use the built-in unittest module with discovery mode. From the lesson root (the directory containing code/), execute:

python3 -m unittest discover -s code/tests -v

The -s code/tests flag specifies the discovery start directory, while -v enables verbose output for debugging. This command automatically finds all test_*.py files and executes them using the standard library harness.

TypeScript and Compiled Languages

TypeScript lessons use npx tsx --test, while Rust and Julia execute tests inline during compilation. These follow the same stdlib-only philosophy but require different commands based on their respective language specifications.

Automating Tests with lesson_run.py

The repository provides scripts/lesson_run.py, a helper script that automates syntax checking and optional execution across the curriculum. As documented in the script's docstring (lines 16-21) and implemented in its main function (lines 24-28), it supports targeting specific phases or lessons with strict validation.

Syntax-Only Validation

To validate all lessons in a phase without executing them:

python3 scripts/lesson_run.py --phase 09 --strict

The script walks the phases/ tree, compiles all .py files for syntax verification, and reports a concise summary of passed, failed, and skipped lessons. Failures include the specific file path and error type, such as SyntaxError: invalid syntax.

Executing Specific Lessons

To run both syntax checks and execution for a single lesson:

cd phases/09-reinforcement-learning/01-mdps-states-actions-rewards
python3 ../../../../scripts/lesson_run.py --execute

The script checks for # requires: headers in entry files and skips execution if external dependencies are declared, preventing runtime errors while reporting the missing requirements.

Interpreting Test Results

The stdlib runners and helper script use standardized exit codes and output formats:

• PASS: All tests succeeded with exit code 0.
• FAIL: Exceptions raised during testing display the offending test name and full traceback.
• SKIP: Lessons without Python files or with external dependency declarations are ignored during syntax-only runs.

Summary

• Navigate to phases/<phase>/<lesson>/ and run python3 -m unittest discover -s code/tests -v to execute Python tests directly using the standard library.
• Use python3 scripts/lesson_run.py --phase <number> --strict for automated syntax validation across multiple lessons.
• The AGENTS.md policy mandates stdlib-only testing with no external frameworks required.
• Lessons with external dependencies declare them via # requires: headers and are automatically skipped by the helper script.

Frequently Asked Questions

Where are the test files located in the ai-engineering-from-scratch repository?

Test files reside in the code/tests/ subdirectory within each lesson directory, following the path pattern phases/<phase-number>-<phase-slug>/<lesson-number>-<lesson-slug>/code/tests/. These contain standard library unit tests named with the test_*.py convention.

What testing framework does the curriculum use?

The repository enforces a strict stdlib-only policy as documented in AGENTS.md. Python lessons use the built-in unittest module, TypeScript uses tsx with Node.js test runners, and Rust/Julia use inline compilation tests. No external testing frameworks like pytest or Jest are permitted.

How do I skip lessons with external dependencies?

Lessons declaring external dependencies include a # requires: comment in their entry main.py file. The lesson_run.py script automatically detects these headers and skips execution for those lessons, reporting them as skipped in the summary rather than failing with an import error.

Can I run tests for an entire phase at once?

Yes. Use python3 scripts/lesson_run.py --phase <number> --strict to validate all lessons within a specific phase. Add the --execute flag to run entry files after syntax checks, though lessons with external dependencies will still be skipped automatically to prevent runtime failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}