# Testing Approach in ai-engineering-from-scratch: Pure Python unittest Validation

> Explore the pure Python unittest testing approach in ai-engineering-from-scratch. Learn fixture-based data loading and stub adapters for dependency free AI validation.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: testing-approach
- Published: 2026-06-13

---

**The repository employs a pure Python `unittest` framework with fixture-based data loading and stub adapters to validate AI engineering lessons without external testing dependencies.**

The `ai-engineering-from-scratch` curriculum maintains rigorous quality standards through a consistent `testing approach` across all lesson directories. Each module contains a `code/tests/` folder that utilizes Python's standard library `unittest` module to verify implementations. This stdlib-first strategy ensures students can execute tests immediately without installing third-party frameworks like `pytest` or `mock`.

## unittest Testing Strategy Overview

The `testing approach` centers on Python's built-in `unittest` module as the sole testing framework. Every lesson follows a uniform structure where the `code/tests/` directory contains modules defining classes that inherit from `unittest.TestCase`. These test files import public symbols from sibling `main.*` modules—such as `from main import tokenize, score`—to exercise the lesson's functionality directly. This design enforces a "stdlib-first" policy that keeps the curriculum accessible and avoids dependency bloat.

## Test Discovery and Execution

Tests execute via standard Python discovery commands that require no external runners. The continuous integration pipeline runs commands like:

```bash
python3 -m unittest discover -v phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests

```

This guarantees that every lesson remains isolated and runnable independently. The discovery mechanism automatically finds all `TestCase` subclasses, ensuring that adding new tests requires no configuration changes to the runner.

## Design Patterns in the Testing Approach

The repository implements several robust patterns to ensure comprehensive validation.

### Fixture-Based Setup

Helper functions like `fixture_tasks()` in [`phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py) load canned data from lesson assets. This pattern provides deterministic inputs for evaluation pipelines, allowing tests to verify behavior against known expected outputs without external API calls or model dependencies.

### Stub Adapters and Mocking

Rather than relying on external mocking libraries, the `testing approach` uses minimal stub implementations. The `StubAdapter` class inherits from `ModelAdapter` to provide deterministic behavior for testing the evaluation pipeline. These lightweight stubs verify that the `run_eval` routine handles model responses correctly without executing real inference.

### Parametric and Edge-Case Coverage

Tests cover both happy-path scenarios and failure modes. The `testing approach` includes specific validations such as `test_rule_based_always_correct_on_targets` for baseline behavior, alongside edge cases like `test_no_tasks_returns_empty` and `test_unknown_metric_raises` to ensure graceful error handling when encountering invalid inputs.

### Parallel vs. Sequential Validation

The `run_eval` function undergoes rigorous concurrency testing. The `test_parallel_sequential_match` method in [`test_runner.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/test_runner.py) executes the same evaluation with `parallel=False` and `parallel=True, max_workers=4`, then asserts identical scoring results. This confirms that threading implementations do not introduce race conditions or scoring discrepancies.

## Coverage Areas in code/tests/

The `testing approach` validates four critical domains across the curriculum:

- **Adapter Contracts**: Verifying that abstract methods in base classes raise `NotImplementedError` when not implemented by subclasses.
- **Scoring Utilities**: Exact match calculations, F-score computation, BLEU-4, and ROUGE-L metrics, including helper functions like `_correct_from_score`, `_ngram_counts`, and `_brevity_penalty`.
- **Report Generation**: Structure validation for JSON reports via `render_report` and Markdown formatting via `render_markdown_block`.
- **Ranking Logic**: Leaderboard ordering verification, such as `test_rule_based_beats_noisy`, ensuring that evaluation metrics correctly rank model performance.

## Representative Code Examples

The following snippets demonstrate the `unittest` patterns found in the repository.

Testing tokenization utilities:

```python

# phases/19-capstone-projects/71-classical-metrics/code/tests/test_metrics.py

class TestTokenize(unittest.TestCase):
    def test_basic_tokenize(self) -> None:
        self.assertEqual(tokenize("The Cat, sat!"), ["the", "cat", "sat"])

```

Fixture-driven evaluation with parallel validation:

```python

# phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py

class TestRunEval(unittest.TestCase):
    def test_parallel_sequential_match(self) -> None:
        tasks = fixture_tasks()
        adapter_a = RuleBasedAdapter()
        adapter_b = RuleBasedAdapter()
        seq_res, _ = run_eval([adapter_a], tasks, parallel=False)
        par_res, _ = run_eval([adapter_b], tasks, parallel=True, max_workers=4)
        self.assertEqual(
            {r.task_id: r.score for r in seq_res},
            {r.task_id: r.score for r in par_res},
        )

```

Error handling for invalid metrics:

```python

# phases/19-capstone-projects/71-classical-metrics/code/tests/test_metrics.py

class TestDispatcher(unittest.TestCase):
    def test_unknown_metric_raises(self) -> None:
        with self.assertRaises(ValueError):
            score("perplexity", "x", ["y"])

```

## Summary

- The `testing approach` relies exclusively on Python's standard library `unittest` module, avoiding external dependencies.
- Each lesson's `code/tests/` directory contains `unittest.TestCase` subclasses that import from sibling `main.*` modules.
- Fixture functions like `fixture_tasks()` provide deterministic data, while `StubAdapter` classes enable isolated testing of evaluation pipelines.
- The validation strategy covers adapter contracts, scoring algorithms (BLEU, ROUGE, F-score), report generation, and parallel execution consistency.
- Tests execute via `python3 -m unittest discover`, ensuring every lesson remains independently runnable and auditable.

## Frequently Asked Questions

### Does the repository use pytest for testing?

No, the repository strictly uses Python's built-in `unittest` module. The `testing approach` intentionally avoids `pytest` and other third-party frameworks to maintain a "stdlib-first" policy, allowing students to run tests immediately without additional installations.

### How are tests organized within each lesson?

Each lesson contains a `code/` directory with a `main.*` entry point and a sibling `code/tests/` folder. The test modules define classes inheriting from `unittest.TestCase` that import and exercise the public API from the main module, ensuring tight integration between implementation and validation.

### What mocking strategy does the testing approach use?

Instead of external mocking libraries, the codebase implements stub classes like `StubAdapter` that inherit from abstract base classes such as `ModelAdapter`. These minimal implementations provide deterministic responses for testing evaluation pipelines without requiring real model inference or external API calls.

### How does the test suite verify parallel execution correctness?

The `test_parallel_sequential_match` test in [`phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/75-end-to-end-eval-runner/code/tests/test_runner.py) executes the `run_eval` function with both `parallel=False` and `parallel=True, max_workers=4`, then asserts that the resulting scores match exactly. This validates that multi-threaded execution produces identical results to sequential processing.