# How to Debug Flaky Tests Using the flake‑hunt Harness in Freebuff

> Debug flaky tests with flake-hunt harness. Isolate and reproduce issues using automatic retries, log capture, and timing instrumentation. Get reliable tests faster.

- Repository: [Codebuff/freebuff](https://github.com/CodebuffAI/freebuff)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Use the `flake-hunt` test harness to isolate, reproduce, and diagnose flaky tests through automatic retries with clean environments, per‑run log capture, and timing instrumentation.**

The **flake‑hunt** harness is a dedicated tool in the [CodebuffAI/freebuff](https://github.com/CodebuffAI/freebuff) repository that solves the problem of nondeterministic test failures. When tests depend on asynchronous processes, external services like tmux sessions, or timing‑sensitive UI updates, they can fail intermittently without code changes. This harness wraps the Bun test runner and provides deterministic reproduction capabilities that standard test execution lacks.

## What the flake‑hunt Harness Provides

The harness adds four key capabilities on top of a regular `bun test` invocation:

| Capability | Description |
|------------|-------------|
| **Retries with isolation** | Each test runs up to a configurable number of attempts (default 3), with a fresh temporary directory and tmux session per retry |
| **Log capture** | Every retry streams console output to `.flake-hunt/<test-name>/run-<n>.log` files |
| **Timing instrumentation** | High‑resolution timestamps around async calls are saved to [`timings.json`](https://github.com/CodebuffAI/freebuff/blob/main/timings.json) for race‑condition analysis |
| **Failure summarisation** | A concise table shows which attempts passed/failed, with log diffs exported as JSON |

## Key Files in the Repository

Understanding the harness requires familiarity with these source locations:

- **[`scripts/flake-hunt.ts`](https://github.com/CodebuffAI/freebuff/blob/main/scripts/flake-hunt.ts)** — Main entry point that parses CLI flags, spawns fresh Bun processes, and aggregates results
- **[`cli/src/__tests__/terminal-watchdog.test.ts`](https://github.com/CodebuffAI/freebuff/blob/main/cli/src/__tests__/terminal-watchdog.test.ts)** — Example flaky test that historically required the harness due to missing writes on the watchdog
- **[`cli/scripts/smoke-binary.ts`](https://github.com/CodebuffAI/freebuff/blob/main/cli/scripts/smoke-binary.ts)** — Helper script that launches the compiled `freebuff` binary in a fresh tmux session; reused by the harness for each retry

## Running the Harness Locally

Debug a flaky test identified in CI by running the harness directly:

```bash
bun run scripts/flake-hunt.ts --test cli/src/__tests__/terminal-watchdog.test.ts

```

Available flags include:

- `--retries <n>` — Set custom retry count (default: 3)
- `--output-dir <path>` — Change output location from default `.flake-hunt/`
- `--ci` — Enable CI mode with stricter failure thresholds

## Analyzing Harness Output

After execution, examine the generated directory structure:

```bash
.flake-hunt/terminal-watchdog.test.ts/
├── run-1.log
├── run-2.log
├── run-3.log
├── timings.json
└── summary.json

```

Compare logs across runs to identify:

- Timestamp variations exceeding expected latency windows
- Missing or extra writes to the terminal watchdog pipe (the historical flake source noted in test comments)
- Errors from the tmux broker (`runner-side flake` pattern)

## Fixing Flakiness Patterns

Once diagnostics reveal the root cause, apply targeted fixes:

| Pattern | Solution | Example Location |
|---------|----------|----------------|
| Race conditions on async writes | Add explicit `await flushWrites()` calls | Terminal watchdog implementations |
| Timeout sensitivity | Increase watchdog timeout or use polling utilities | `waitFor(() => …)` helpers |
| External process dependency | Mock tmux sessions in unit tests | [`cli/src/utils/__tests__/tmux-helpers.test.ts`](https://github.com/CodebuffAI/freebuff/blob/main/cli/src/utils/__tests__/tmux-helpers.test.ts) |

Verify fixes by re‑running the harness until all retries pass consistently.

## CI Integration

The harness integrates directly into the build pipeline. The [`ci.yml`](https://github.com/CodebuffAI/freebuff/blob/main/ci.yml) workflow invokes:

```bash
bun run scripts/flake-hunt.ts --ci

```

When flakes are detected, the job fails and uploads artifacts:

```yaml
- name: Upload flake-hunt logs
  uses: actions/upload-artifact@v4
  if: failure()
  with:
    name: flake-hunt-logs
    path: .flake-hunt/

```

## Why Use flake‑hunt Over Standard Retries

**Adhoc retry loops** in test files pollute the codebase and cannot isolate environmental state. The harness approach in [`scripts/flake-hunt.ts`](https://github.com/CodebuffAI/freebuff/blob/main/scripts/flake-hunt.ts) provides:

- **Process isolation** — Each retry runs in a fresh environment, eliminating state leakage
- **Deterministic artifacts** — Logs and timing data are automatically captured for every failure
- **Zero code changes** — No modifications to test files required; configuration lives in the harness invocation

## Summary

- The **flake‑hunt** harness lives at [`scripts/flake-hunt.ts`](https://github.com/CodebuffAI/freebuff/blob/main/scripts/flake-hunt.ts) and wraps Bun's test runner for deterministic flaky test reproduction
- **Isolation** through fresh tmux sessions and temp directories prevents cross‑run state pollution
- **Log capture** to `.flake-hunt/<test>/` directories enables post‑mortem analysis without re‑running tests
- **CI integration** via `--ci` flag and artifact uploads makes flaky tests visible to the entire team
- Historical examples like [`cli/src/__tests__/terminal-watchdog.test.ts`](https://github.com/CodebuffAI/freebuff/blob/main/cli/src/__tests__/terminal-watchdog.test.ts) demonstrate how missing async writes cause flakes that the harness surfaces

## Frequently Asked Questions

### How do I know if a test is flaky versus genuinely broken?

Run the harness with high retry counts (10–20). A genuinely broken test fails on every attempt; a flaky test shows mixed results. The harness summary table in [`summary.json`](https://github.com/CodebuffAI/freebuff/blob/main/summary.json) exposes this pattern immediately without manual re‑execution.

### Can I use flake‑hunt for tests that don't involve tmux?

Yes. While the harness was designed around Freebuff's terminal‑heavy architecture, the isolation and logging mechanisms work for any Bun test. Simply omit tmux‑specific expectations from your test environment.

### Where are the timing logs stored and how do I read them?

Timing data is written to [`timings.json`](https://github.com/CodebuffAI/freebuff/blob/main/timings.json) in each test's output directory. The JSON structure contains start/end timestamps for key async boundaries, allowing you to calculate delta times and identify race conditions programmatically.

### Does running more retries slow down CI significantly?

The harness executes retries sequentially to maintain isolation. For a default of 3 retries, expect 3× the single‑run duration. Configure CI jobs with appropriate timeouts, or run the harness only on suspect test files rather than the full suite.