# How to Test LoopX: A Complete Guide to Unit Tests, Smokes, and Canary Gates

> Learn how to test LoopX effectively with unit tests, smokes, and canary gates. This guide covers all you need to validate your AI agent control-plane changes.

- Repository: [huangruiteng/loopx](https://github.com/huangruiteng/loopx)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Testing LoopX requires a layered approach combining fast deterministic unit tests under `tests/`, durable public smokes in `examples/`, and risk-based canaries via the `loopx canary` CLI to validate changes to the stateful control-plane for long-running AI agents.**

LoopX is a stateful control-plane for long-running AI agents where small code changes can alter todo selection, user gate behavior, or host scheduling. Because these changes impact production reliability, the repository implements a comprehensive testing strategy spanning deterministic unit tests, CLI-output budgets, and live model qualification. This guide walks through the complete testing workflow using actual commands and source file paths from the `huangruiteng/loopx` repository.

## Install Test Dependencies

Before running any tests, install the package with test extras. The dependencies are declared in [`pyproject.toml`](https://github.com/huangruiteng/loopx/blob/main/pyproject.toml) at the repository root.

```bash
python -m pip install -e ".[test]"

```

This installs **pytest**, **ruff**, **mypy**, and other quality tools alongside the core LoopX package.

## Run the Fast PR Gate

The fast PR gate provides immediate feedback on code quality and unit test correctness. This layer validates pure rule correctness and schema validation without invoking live models or external services.

Execute the full fast gate with these commands:

```bash
python -m ruff check tests loopx/canary loopx/control_plane loopx/domain_packs loopx/presentation
python -m mypy
python -m pytest -q
git diff --check

```

The **ruff** and **mypy** commands enforce style and type safety across the core modules, while **pytest** runs the unit test suite located in `tests/`. According to the LoopX source code, these tests cover components like [`test_turn_envelope.py`](https://github.com/huangruiteng/loopx/blob/main/test_turn_envelope.py) and [`test_skillsbench_turn_runtime.py`](https://github.com/huangruiteng/loopx/blob/main/test_skillsbench_turn_runtime.py) to ensure cross-module interactions remain stable.

## Execute Focused Smoke Tests

After unit tests pass, validate specific public boundaries using durable smoke tests. These exercises test the shipped CLI and public-private boundaries using public-safe fixtures.

### CLI-Output Budget Regression

Run the CLI-output budget smoke to detect accidental growth of agent-facing output:

```bash
python examples/control_plane/cli-output-budget-regression-smoke.py

```

This script, located at [`examples/control_plane/cli-output-budget-regression-smoke.py`](https://github.com/huangruiteng/loopx/blob/main/examples/control_plane/cli-output-budget-regression-smoke.py), ensures contract stability by verifying that CLI output remains within defined budgets.

### Area-Specific Smokes

Select a focused smoke matching your development area. For example, to test the skillsbench turn runtime:

```bash
python examples/skillsbench/turn_runtime_smoke.py

```

All smoke examples live under `examples/` and are enumerated in [`tests/test_smoke_suite.py`](https://github.com/huangruiteng/loopx/blob/main/tests/test_smoke_suite.py). These scripts exercise public-safe fixtures without requiring production credentials.

## Trigger Risk-Based Canaries

The **canary** system automatically selects the smallest risk-based test slice that touches every changed public surface. This is implemented in `loopx/canary/` and invoked via the CLI.

Run the premerge canary against your current git diff:

```bash
loopx canary premerge --from-git-diff

```

This command analyzes changes in `loopx/canary/` and executes only the tests relevant to modified code paths, providing faster feedback than the full suite while maintaining coverage of affected surfaces.

## Run the Full Public Smoke Suite

For nightly CI or comprehensive validation before major releases, execute the full-public smoke fleet. This runs all durable smokes in parallel to verify cross-module interactions under realistic conditions.

```bash
loopx canary smoke-suite --suite full-public --jobs 4 --timeout-seconds 120

```

The suite definition lives in [`.github/workflows/full-public-smokes.yml`](https://github.com/huangruiteng/loopx/blob/main/.github/workflows/full-public-smokes.yml), which configures the nightly workflow. This layer complements unit tests by validating the actual CLI entry points and runtime behavior that deterministic tests cannot fully replicate.

## Verify CLI-Output Budgets

After running smokes, verify that output budgets remain healthy using the health collector:

```bash
loopx canary smoke-health --receipt smoke-results

```

The health check logic is implemented in [`loopx/canary/smoke_health.py`](https://github.com/huangruiteng/loopx/blob/main/loopx/canary/smoke_health.py). This step aggregates results from previous test runs to confirm that agent-facing contracts remain stable and that no regression introduced excessive output growth.

## Qualify Model Behavior

The **model-behavior qualification** layer validates that the live Doubao model respects the same contracts enforced by deterministic tests. This low-frequency gate requires the `ARK_API_KEY` environment variable.

Execute the qualification script:

```bash
python3 scripts/qualify-doubao-model-behavior-live.py \
  --qualification-id <public-safe-run-id>

```

This script, located at [`scripts/qualify-doubao-model-behavior-live.py`](https://github.com/huangruiteng/loopx/blob/main/scripts/qualify-doubao-model-behavior-live.py), runs the default packet against the live model to certify production-ready behavior. According to the testing documentation, a deterministic failure always takes precedence over a model pass, ensuring that code correctness remains the primary gate.

## Execute Release-Commit Gate

The final **release-qualification** gate aggregates receipts from all previous testing layers to prove the release builds from a clean source tree.

Run the release gate:

```bash
loopx canary release-qualification \
  --manifest-json release-qualification.json \
  --repo-root .

```

The manifest handling is implemented in [`loopx/canary/release_qualification.py`](https://github.com/huangruiteng/loopx/blob/main/loopx/canary/release_qualification.py). This command validates that unit tests, smokes, canaries, and model qualifications have all passed before allowing the commit to enter production.

## Summary

- **Install dependencies** with `pip install -e ".[test]"` as defined in [`pyproject.toml`](https://github.com/huangruiteng/loopx/blob/main/pyproject.toml) to access the testing toolchain.
- **Fast PR gates** combine `ruff`, `mypy`, `pytest`, and git checks for immediate deterministic feedback.
- **Focused smokes** like [`cli-output-budget-regression-smoke.py`](https://github.com/huangruiteng/loopx/blob/main/cli-output-budget-regression-smoke.py) validate specific CLI contracts without production credentials.
- **Risk-based canaries** via `loopx canary premerge` automatically select tests relevant to your git diff, implemented in `loopx/canary/`.
- **Full-public suite** runs nightly via [`.github/workflows/full-public-smokes.yml`](https://github.com/huangruiteng/loopx/blob/main/.github/workflows/full-public-smokes.yml) to exercise complete public boundaries.
- **Model qualification** requires `ARK_API_KEY` and runs via [`scripts/qualify-doubao-model-behavior-live.py`](https://github.com/huangruiteng/loopx/blob/main/scripts/qualify-doubao-model-behavior-live.py) to certify live model behavior.
- **Release gate** aggregates all receipts through `loopx canary release-qualification` before deployment.

## Frequently Asked Questions

### What is the difference between unit tests and smoke tests in LoopX?

Unit tests under `tests/` validate pure rule correctness and schema validation using deterministic inputs, while smoke tests in `examples/` exercise the actual CLI and cross-module interactions with public-safe fixtures. The unit tests run in seconds via `pytest -q`, whereas smokes validate runtime behavior that unit tests cannot fully capture.

### How do I run only the tests relevant to my code changes?

Use the risk-based canary command: `loopx canary premerge --from-git-diff`. This analyzes your current git diff and automatically selects the minimal test slice that covers all changed public surfaces, implemented in [`loopx/canary/__init__.py`](https://github.com/huangruiteng/loopx/blob/main/loopx/canary/__init__.py). This is faster than running the full public suite while ensuring coverage of modified code paths.

### Why does LoopX require a live model qualification test?

Because LoopX controls long-running AI agents, deterministic tests cannot fully validate model behavior. The [`qualify-doubao-model-behavior-live.py`](https://github.com/huangruiteng/loopx/blob/main/qualify-doubao-model-behavior-live.py) script verifies that the live Doubao model respects the same contracts enforced by unit tests. This low-frequency gate ensures that model updates or prompt changes do not break production agent behavior, though deterministic failures always take precedence over model passes.

### Where are the test configurations and CI definitions stored?

Test dependencies and extras are declared in [`pyproject.toml`](https://github.com/huangruiteng/loopx/blob/main/pyproject.toml). CI workflows for the fast layer reside in [`.github/workflows/python-tests.yml`](https://github.com/huangruiteng/loopx/blob/main/.github/workflows/python-tests.yml), while the nightly full-public smokes are defined in [`.github/workflows/full-public-smokes.yml`](https://github.com/huangruiteng/loopx/blob/main/.github/workflows/full-public-smokes.yml). The authoritative testing documentation lives at [`docs/development/testing-and-quality.md`](https://github.com/huangruiteng/loopx/blob/main/docs/development/testing-and-quality.md).