# How the E2E Test Harness Manages Temporary Daemons in no-mistakes

> Discover how the no-mistakes E2E test harness manages temporary daemons with an inventory-based manager for isolated, sandboxed processes and automatic cleanup for robust testing.

- Repository: [Kun Chen/no-mistakes](https://github.com/kunchenguid/no-mistakes)
- Tags: internals
- Published: 2026-07-25

---

**The no-mistakes E2E test harness uses an inventory-based manager in `internal/e2edaemon` to spawn isolated daemon processes with sandboxed `$NM_HOME` directories, enforcing concurrency limits and guaranteeing cleanup through graceful shutdown hooks and a background reaper process.**

The `no-mistakes` project requires a real daemon instance to validate its end-to-end workflows, but testing against a developer's local daemon risks data corruption and resource conflicts. The E2E test harness solves this by orchestrating temporary, sandboxed daemons that are tightly coupled to test lifecycles through a dedicated inventory management system.

## Inventory-Based Ownership and Concurrency Control

The `internal/e2edaemon` package maintains a central **inventory** that tracks every temporary daemon spawned during test execution. This inventory serves as the source of truth for resource allocation and cleanup.

### The Inventory Directory Structure

The harness stores daemon metadata in `$NM_HOME/e2e/daemon-inventory`, where each spawned daemon receives a JSON file containing:

- A unique slot ID
- The running process PID
- The absolute path to the temporary `$NM_HOME` directory

This file-based inventory enables atomic reservation of daemon slots and prevents race conditions when multiple tests run concurrently.

### Concurrency Limits with NM_E2E_DAEMON_MAX

To prevent host overload, the harness enforces a concurrency cap via the `NM_E2E_DAEMON_MAX` environment variable (defaulting to **2**). When a test calls `e2edaemon.Acquire`, the function atomically checks the inventory against this limit and either reserves a slot or returns a clear error if capacity is reached.

According to the package documentation in [`internal/e2edaemon/doc.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/e2edaemon/doc.go), this design ensures the test suite never exhausts system resources, even during parallel test execution.

## Sandboxed Daemon Launch Process

Each test receives an isolated daemon instance through a combination of temporary directories and controlled process spawning.

### Isolated NM_HOME Directories

The harness creates a fresh sandbox for every test using Go's `t.TempDir()`:

```go
func (h *Harness) setup() {
    h.NMHome = h.T.TempDir()
    // NM_HOME is set to this temporary path
}

```

This temporary directory becomes the daemon's workspace, housing its database, log files, and Unix socket. By isolating `$NM_HOME`, the test daemon cannot access the user's real `~/.no-mistakes` configuration or data.

### Process Spawning and PID Tracking

The harness launches the daemon via `e2edaemon.Acquire(h.NMHome, h.NMBin, timeout)`, which:

1. Writes the inventory entry to reserve a slot
2. Executes the binary using `exec.CommandContext` with the test's environment variables
3. Records the child PID through `Ownership.SyncPID`

As implemented in [`internal/e2e/harness.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/e2e/harness.go) (lines 166-173), the daemon inherits the test process's environment while maintaining isolation from the host's default configuration.

## Automatic Cleanup and Lifecycle Management

The harness guarantees daemon termination through multiple complementary mechanisms that handle both normal test completion and unexpected crashes.

### Graceful Shutdown on Test Exit

The harness registers a cleanup function via `t.Cleanup(h.Close)` that executes three critical steps:

1. **Terminate the daemon**: Sends a graceful shutdown command (`nm daemon stop`) and waits for the PID to exit
2. **Release the inventory slot**: Calls `e2edaemon.Release` to delete the JSON entry and free the concurrency slot
3. **Remove temporary files**: The `t.TempDir()` cleanup automatically deletes the sandboxed `$NM_HOME`

This sequence ensures no resources leak between test runs, even when tests fail or panic.

### The Reaper Process for Crash Recovery

When the test process crashes or receives a SIGKILL, the graceful cleanup may not execute. To handle this, the [`scripts/e2e.sh`](https://github.com/kunchenguid/no-mistakes/blob/main/scripts/e2e.sh) script invokes [`reapmain.go`](https://github.com/kunchenguid/no-mistakes/blob/main/reapmain.go) at exit, which scans the inventory directory for entries with non-existent PIDs and safely removes stale records.

As noted in [`scripts/e2e.sh`](https://github.com/kunchenguid/no-mistakes/blob/main/scripts/e2e.sh) (lines 48-55), this reaper prevents abandoned daemon processes from accumulating across CI runs.

## Test Integration and Visibility

The harness exposes helpers that enable tests to verify correct daemon lifecycle behavior.

### Harness Helper Methods

Tests interact with daemon lifecycle state through methods defined in [`internal/e2e/harness.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/e2e/harness.go):

- `daemonStopDir()`: Verifies that `nm daemon stop` writes its PID file into the temporary `$NM_HOME` (asserted in [`internal/e2e/harness_test.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/e2e/harness_test.go) lines 105-107)
- `h.daemonOwn.SyncPID(pid)`: Synchronizes the live process ID into the inventory for consistency checks

These utilities are exercised across the test suite, including [`internal/e2e/daemon_run_test.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/e2e/daemon_run_test.go) and [`internal/cli/daemon_lifecycle_test.go`](https://github.com/kunchenguid/no-mistakes/blob/main/internal/cli/daemon_lifecycle_test.go), ensuring the temporary daemon behaves identically to production deployments.

## Summary

- The **inventory-based architecture** in `internal/e2edaemon` tracks all temporary daemons via JSON files in `$NM_HOME/e2e/daemon-inventory`
- **Concurrency control** via `NM_E2E_DAEMON_MAX` limits parallel daemons to 2 by default
- **Sandboxed environments** using `t.TempDir()` ensure complete isolation from user data
- **Automatic cleanup** combines `t.Cleanup` hooks for graceful shutdown with a **reaper process** in [`scripts/e2e.sh`](https://github.com/kunchenguid/no-mistakes/blob/main/scripts/e2e.sh) for crash recovery
- **PID synchronization** through `Ownership.SyncPID` maintains accurate process tracking throughout the test lifecycle

## Frequently Asked Questions

### How does the E2E test harness prevent test daemons from interfering with my local no-mistakes installation?

The harness generates a unique temporary directory via `t.TempDir()` for each test and sets this as `NM_HOME`. This sandboxed directory contains all daemon data, including the database and Unix socket, ensuring complete isolation from the default `~/.no-mistakes` location used by local development instances.

### What happens if a test crashes while a temporary daemon is running?

If a test process crashes or is killed, the standard `t.Cleanup` hooks may not execute. The harness mitigates this through a reaper mechanism in [`scripts/e2e.sh`](https://github.com/kunchenguid/no-mistakes/blob/main/scripts/e2e.sh) that invokes [`reapmain.go`](https://github.com/kunchenguid/no-mistakes/blob/main/reapmain.go) to scan the inventory directory and remove entries for processes that no longer exist, preventing daemon accumulation across CI runs.

### How many temporary daemons can run simultaneously during E2E testing?

The harness enforces a default concurrency limit of **2** simultaneous daemons, controlled by the `NM_E2E_DAEMON_MAX` environment variable. The `e2edaemon.Acquire` function checks this cap against the current inventory before spawning new processes, returning an error if the limit is reached.

### Can I configure the timeout for daemon startup in the E2E harness?

Yes, the `e2edaemon.Acquire` function accepts a timeout parameter (as seen in the signature `e2edaemon.Acquire(h.NMHome, h.NMBin, timeout)`) that controls how long the harness waits for the daemon to initialize before failing the test setup.