How to Test LoopX: A Complete Guide to Unit Tests, Smokes, and Canary Gates

Testing LoopX requires a layered approach combining fast deterministic unit tests under tests/, durable public smokes in examples/, and risk-based canaries via the loopx canary CLI to validate changes to the stateful control-plane for long-running AI agents.

LoopX is a stateful control-plane for long-running AI agents where small code changes can alter todo selection, user gate behavior, or host scheduling. Because these changes impact production reliability, the repository implements a comprehensive testing strategy spanning deterministic unit tests, CLI-output budgets, and live model qualification. This guide walks through the complete testing workflow using actual commands and source file paths from the huangruiteng/loopx repository.

Install Test Dependencies

Before running any tests, install the package with test extras. The dependencies are declared in pyproject.toml at the repository root.

python -m pip install -e ".[test]"

This installs pytest, ruff, mypy, and other quality tools alongside the core LoopX package.

Run the Fast PR Gate

The fast PR gate provides immediate feedback on code quality and unit test correctness. This layer validates pure rule correctness and schema validation without invoking live models or external services.

Execute the full fast gate with these commands:

python -m ruff check tests loopx/canary loopx/control_plane loopx/domain_packs loopx/presentation
python -m mypy
python -m pytest -q
git diff --check

The ruff and mypy commands enforce style and type safety across the core modules, while pytest runs the unit test suite located in tests/. According to the LoopX source code, these tests cover components like test_turn_envelope.py and test_skillsbench_turn_runtime.py to ensure cross-module interactions remain stable.

Execute Focused Smoke Tests

After unit tests pass, validate specific public boundaries using durable smoke tests. These exercises test the shipped CLI and public-private boundaries using public-safe fixtures.

CLI-Output Budget Regression

Run the CLI-output budget smoke to detect accidental growth of agent-facing output:

python examples/control_plane/cli-output-budget-regression-smoke.py

This script, located at examples/control_plane/cli-output-budget-regression-smoke.py, ensures contract stability by verifying that CLI output remains within defined budgets.

Area-Specific Smokes

Select a focused smoke matching your development area. For example, to test the skillsbench turn runtime:

python examples/skillsbench/turn_runtime_smoke.py

All smoke examples live under examples/ and are enumerated in tests/test_smoke_suite.py. These scripts exercise public-safe fixtures without requiring production credentials.

Trigger Risk-Based Canaries

The canary system automatically selects the smallest risk-based test slice that touches every changed public surface. This is implemented in loopx/canary/ and invoked via the CLI.

Run the premerge canary against your current git diff:

loopx canary premerge --from-git-diff

This command analyzes changes in loopx/canary/ and executes only the tests relevant to modified code paths, providing faster feedback than the full suite while maintaining coverage of affected surfaces.

Run the Full Public Smoke Suite

For nightly CI or comprehensive validation before major releases, execute the full-public smoke fleet. This runs all durable smokes in parallel to verify cross-module interactions under realistic conditions.

loopx canary smoke-suite --suite full-public --jobs 4 --timeout-seconds 120

The suite definition lives in .github/workflows/full-public-smokes.yml, which configures the nightly workflow. This layer complements unit tests by validating the actual CLI entry points and runtime behavior that deterministic tests cannot fully replicate.

Verify CLI-Output Budgets

After running smokes, verify that output budgets remain healthy using the health collector:

loopx canary smoke-health --receipt smoke-results

The health check logic is implemented in loopx/canary/smoke_health.py. This step aggregates results from previous test runs to confirm that agent-facing contracts remain stable and that no regression introduced excessive output growth.

Qualify Model Behavior

The model-behavior qualification layer validates that the live Doubao model respects the same contracts enforced by deterministic tests. This low-frequency gate requires the ARK_API_KEY environment variable.

Execute the qualification script:

python3 scripts/qualify-doubao-model-behavior-live.py \
  --qualification-id <public-safe-run-id>

This script, located at scripts/qualify-doubao-model-behavior-live.py, runs the default packet against the live model to certify production-ready behavior. According to the testing documentation, a deterministic failure always takes precedence over a model pass, ensuring that code correctness remains the primary gate.

Execute Release-Commit Gate

The final release-qualification gate aggregates receipts from all previous testing layers to prove the release builds from a clean source tree.

Run the release gate:

loopx canary release-qualification \
  --manifest-json release-qualification.json \
  --repo-root .

The manifest handling is implemented in loopx/canary/release_qualification.py. This command validates that unit tests, smokes, canaries, and model qualifications have all passed before allowing the commit to enter production.

Summary

• Install dependencies with pip install -e ".[test]" as defined in pyproject.toml to access the testing toolchain.
• Fast PR gates combine ruff, mypy, pytest, and git checks for immediate deterministic feedback.
• Focused smokes like cli-output-budget-regression-smoke.py validate specific CLI contracts without production credentials.
• Risk-based canaries via loopx canary premerge automatically select tests relevant to your git diff, implemented in loopx/canary/.
• Full-public suite runs nightly via .github/workflows/full-public-smokes.yml to exercise complete public boundaries.
• Model qualification requires ARK_API_KEY and runs via scripts/qualify-doubao-model-behavior-live.py to certify live model behavior.
• Release gate aggregates all receipts through loopx canary release-qualification before deployment.

Frequently Asked Questions

What is the difference between unit tests and smoke tests in LoopX?

Unit tests under tests/ validate pure rule correctness and schema validation using deterministic inputs, while smoke tests in examples/ exercise the actual CLI and cross-module interactions with public-safe fixtures. The unit tests run in seconds via pytest -q, whereas smokes validate runtime behavior that unit tests cannot fully capture.

How do I run only the tests relevant to my code changes?

Use the risk-based canary command: loopx canary premerge --from-git-diff. This analyzes your current git diff and automatically selects the minimal test slice that covers all changed public surfaces, implemented in loopx/canary/__init__.py. This is faster than running the full public suite while ensuring coverage of modified code paths.

Why does LoopX require a live model qualification test?

Because LoopX controls long-running AI agents, deterministic tests cannot fully validate model behavior. The qualify-doubao-model-behavior-live.py script verifies that the live Doubao model respects the same contracts enforced by unit tests. This low-frequency gate ensures that model updates or prompt changes do not break production agent behavior, though deterministic failures always take precedence over model passes.

Where are the test configurations and CI definitions stored?

Test dependencies and extras are declared in pyproject.toml. CI workflows for the fast layer reside in .github/workflows/python-tests.yml, while the nightly full-public smokes are defined in .github/workflows/full-public-smokes.yml. The authoritative testing documentation lives at docs/development/testing-and-quality.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}