Testing Approach in ai-engineering-from-scratch: Pure Python unittest Validation

The repository employs a pure Python unittest framework with fixture-based data loading and stub adapters to validate AI engineering lessons without external testing dependencies.

The ai-engineering-from-scratch curriculum maintains rigorous quality standards through a consistent testing approach across all lesson directories. Each module contains a code/tests/ folder that utilizes Python's standard library unittest module to verify implementations. This stdlib-first strategy ensures students can execute tests immediately without installing third-party frameworks like pytest or mock.

unittest Testing Strategy Overview

The testing approach centers on Python's built-in unittest module as the sole testing framework. Every lesson follows a uniform structure where the code/tests/ directory contains modules defining classes that inherit from unittest.TestCase. These test files import public symbols from sibling main.* modules—such as from main import tokenize, score—to exercise the lesson's functionality directly. This design enforces a "stdlib-first" policy that keeps the curriculum accessible and avoids dependency bloat.

Test Discovery and Execution

Tests execute via standard Python discovery commands that require no external runners. The continuous integration pipeline runs commands like:

python3 -m unittest discover -v phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests

This guarantees that every lesson remains isolated and runnable independently. The discovery mechanism automatically finds all TestCase subclasses, ensuring that adding new tests requires no configuration changes to the runner.

Design Patterns in the Testing Approach

The repository implements several robust patterns to ensure comprehensive validation.

Fixture-Based Setup

Helper functions like fixture_tasks() in phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py load canned data from lesson assets. This pattern provides deterministic inputs for evaluation pipelines, allowing tests to verify behavior against known expected outputs without external API calls or model dependencies.

Stub Adapters and Mocking

Rather than relying on external mocking libraries, the testing approach uses minimal stub implementations. The StubAdapter class inherits from ModelAdapter to provide deterministic behavior for testing the evaluation pipeline. These lightweight stubs verify that the run_eval routine handles model responses correctly without executing real inference.

Parametric and Edge-Case Coverage

Tests cover both happy-path scenarios and failure modes. The testing approach includes specific validations such as test_rule_based_always_correct_on_targets for baseline behavior, alongside edge cases like test_no_tasks_returns_empty and test_unknown_metric_raises to ensure graceful error handling when encountering invalid inputs.

Parallel vs. Sequential Validation

The run_eval function undergoes rigorous concurrency testing. The test_parallel_sequential_match method in test_runner.py executes the same evaluation with parallel=False and parallel=True, max_workers=4, then asserts identical scoring results. This confirms that threading implementations do not introduce race conditions or scoring discrepancies.

Coverage Areas in code/tests/

The testing approach validates four critical domains across the curriculum:

• Adapter Contracts: Verifying that abstract methods in base classes raise NotImplementedError when not implemented by subclasses.
• Scoring Utilities: Exact match calculations, F-score computation, BLEU-4, and ROUGE-L metrics, including helper functions like _correct_from_score, _ngram_counts, and _brevity_penalty.
• Report Generation: Structure validation for JSON reports via render_report and Markdown formatting via render_markdown_block.
• Ranking Logic: Leaderboard ordering verification, such as test_rule_based_beats_noisy, ensuring that evaluation metrics correctly rank model performance.

Representative Code Examples

The following snippets demonstrate the unittest patterns found in the repository.

Testing tokenization utilities:

# phases/19-capstone-projects/71-classical-metrics/code/tests/test_metrics.py

class TestTokenize(unittest.TestCase):
    def test_basic_tokenize(self) -> None:
        self.assertEqual(tokenize("The Cat, sat!"), ["the", "cat", "sat"])

Fixture-driven evaluation with parallel validation:

# phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py

class TestRunEval(unittest.TestCase):
    def test_parallel_sequential_match(self) -> None:
        tasks = fixture_tasks()
        adapter_a = RuleBasedAdapter()
        adapter_b = RuleBasedAdapter()
        seq_res, _ = run_eval([adapter_a], tasks, parallel=False)
        par_res, _ = run_eval([adapter_b], tasks, parallel=True, max_workers=4)
        self.assertEqual(
            {r.task_id: r.score for r in seq_res},
            {r.task_id: r.score for r in par_res},
        )

Error handling for invalid metrics:

# phases/19-capstone-projects/71-classical-metrics/code/tests/test_metrics.py

class TestDispatcher(unittest.TestCase):
    def test_unknown_metric_raises(self) -> None:
        with self.assertRaises(ValueError):
            score("perplexity", "x", ["y"])

Summary

• The testing approach relies exclusively on Python's standard library unittest module, avoiding external dependencies.
• Each lesson's code/tests/ directory contains unittest.TestCase subclasses that import from sibling main.* modules.
• Fixture functions like fixture_tasks() provide deterministic data, while StubAdapter classes enable isolated testing of evaluation pipelines.
• The validation strategy covers adapter contracts, scoring algorithms (BLEU, ROUGE, F-score), report generation, and parallel execution consistency.
• Tests execute via python3 -m unittest discover, ensuring every lesson remains independently runnable and auditable.

Frequently Asked Questions

Does the repository use pytest for testing?

No, the repository strictly uses Python's built-in unittest module. The testing approach intentionally avoids pytest and other third-party frameworks to maintain a "stdlib-first" policy, allowing students to run tests immediately without additional installations.

How are tests organized within each lesson?

Each lesson contains a code/ directory with a main.* entry point and a sibling code/tests/ folder. The test modules define classes inheriting from unittest.TestCase that import and exercise the public API from the main module, ensuring tight integration between implementation and validation.

What mocking strategy does the testing approach use?

Instead of external mocking libraries, the codebase implements stub classes like StubAdapter that inherit from abstract base classes such as ModelAdapter. These minimal implementations provide deterministic responses for testing evaluation pipelines without requiring real model inference or external API calls.

How does the test suite verify parallel execution correctness?

The test_parallel_sequential_match test in phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py executes the run_eval function with both parallel=False and parallel=True, max_workers=4, then asserts that the resulting scores match exactly. This validates that multi-threaded execution produces identical results to sequential processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}