How to Debug Flaky Tests Using the flake‑hunt Harness in Freebuff

Use the flake-hunt test harness to isolate, reproduce, and diagnose flaky tests through automatic retries with clean environments, per‑run log capture, and timing instrumentation.

The flake‑hunt harness is a dedicated tool in the CodebuffAI/freebuff repository that solves the problem of nondeterministic test failures. When tests depend on asynchronous processes, external services like tmux sessions, or timing‑sensitive UI updates, they can fail intermittently without code changes. This harness wraps the Bun test runner and provides deterministic reproduction capabilities that standard test execution lacks.

What the flake‑hunt Harness Provides

The harness adds four key capabilities on top of a regular bun test invocation:

Capability	Description
Retries with isolation	Each test runs up to a configurable number of attempts (default 3), with a fresh temporary directory and tmux session per retry
Log capture	Every retry streams console output to .flake-hunt/<test-name>/run-<n>.log files
Timing instrumentation	High‑resolution timestamps around async calls are saved to timings.json for race‑condition analysis
Failure summarisation	A concise table shows which attempts passed/failed, with log diffs exported as JSON

Key Files in the Repository

Understanding the harness requires familiarity with these source locations:

• scripts/flake-hunt.ts — Main entry point that parses CLI flags, spawns fresh Bun processes, and aggregates results
• cli/src/__tests__/terminal-watchdog.test.ts — Example flaky test that historically required the harness due to missing writes on the watchdog
• cli/scripts/smoke-binary.ts — Helper script that launches the compiled freebuff binary in a fresh tmux session; reused by the harness for each retry

Running the Harness Locally

Debug a flaky test identified in CI by running the harness directly:

bun run scripts/flake-hunt.ts --test cli/src/__tests__/terminal-watchdog.test.ts

Available flags include:

• --retries <n> — Set custom retry count (default: 3)
• --output-dir <path> — Change output location from default .flake-hunt/
• --ci — Enable CI mode with stricter failure thresholds

Analyzing Harness Output

After execution, examine the generated directory structure:

.flake-hunt/terminal-watchdog.test.ts/
├── run-1.log
├── run-2.log
├── run-3.log
├── timings.json
└── summary.json

Compare logs across runs to identify:

• Timestamp variations exceeding expected latency windows
• Missing or extra writes to the terminal watchdog pipe (the historical flake source noted in test comments)
• Errors from the tmux broker (runner-side flake pattern)

Fixing Flakiness Patterns

Once diagnostics reveal the root cause, apply targeted fixes:

Pattern	Solution	Example Location
Race conditions on async writes	Add explicit await flushWrites() calls	Terminal watchdog implementations
Timeout sensitivity	Increase watchdog timeout or use polling utilities	waitFor(() => …) helpers
External process dependency	Mock tmux sessions in unit tests	cli/src/utils/__tests__/tmux-helpers.test.ts

Verify fixes by re‑running the harness until all retries pass consistently.

CI Integration

The harness integrates directly into the build pipeline. The ci.yml workflow invokes:

bun run scripts/flake-hunt.ts --ci

When flakes are detected, the job fails and uploads artifacts:

- name: Upload flake-hunt logs
  uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: flake-hunt-logs
    path: .flake-hunt/

Why Use flake‑hunt Over Standard Retries

Adhoc retry loops in test files pollute the codebase and cannot isolate environmental state. The harness approach in scripts/flake-hunt.ts provides:

• Process isolation — Each retry runs in a fresh environment, eliminating state leakage
• Deterministic artifacts — Logs and timing data are automatically captured for every failure
• Zero code changes — No modifications to test files required; configuration lives in the harness invocation

Summary

• The flake‑hunt harness lives at scripts/flake-hunt.ts and wraps Bun's test runner for deterministic flaky test reproduction
• Isolation through fresh tmux sessions and temp directories prevents cross‑run state pollution
• Log capture to .flake-hunt/<test>/ directories enables post‑mortem analysis without re‑running tests
• CI integration via --ci flag and artifact uploads makes flaky tests visible to the entire team
• Historical examples like cli/src/__tests__/terminal-watchdog.test.ts demonstrate how missing async writes cause flakes that the harness surfaces

Frequently Asked Questions

How do I know if a test is flaky versus genuinely broken?

Run the harness with high retry counts (10–20). A genuinely broken test fails on every attempt; a flaky test shows mixed results. The harness summary table in summary.json exposes this pattern immediately without manual re‑execution.

Can I use flake‑hunt for tests that don't involve tmux?

Yes. While the harness was designed around Freebuff's terminal‑heavy architecture, the isolation and logging mechanisms work for any Bun test. Simply omit tmux‑specific expectations from your test environment.

Where are the timing logs stored and how do I read them?

Timing data is written to timings.json in each test's output directory. The JSON structure contains start/end timestamps for key async boundaries, allowing you to calculate delta times and identify race conditions programmatically.

Does running more retries slow down CI significantly?

The harness executes retries sequentially to maintain isolation. For a default of 3 retries, expect 3× the single‑run duration. Configure CI jobs with appropriate timeouts, or run the harness only on suspect test files rather than the full suite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}