# gstack E2E Test Infrastructure Using claude -p: A Deep Dive into AI-Driven Integration Testing

> Explore the gstack E2E test infrastructure using claude -p. Learn how it validates AI-driven skills by spawning Claude-Code subprocesses and parsing NDJSON output for assertions.

- Repository: [Garry Tan/gstack](https://github.com/garrytan/gstack)
- Tags: deep-dive
- Published: 2026-05-15

---

**The gstack E2E test infrastructure using claude -p validates AI-driven skills by spawning real Claude-Code subprocesses, piping test prompts via stdin, and parsing NDJSON output to assert on tool invocations and error states.**

The [garrytan/gstack](https://github.com/garrytan/gstack) repository employs a sophisticated end-to-end testing tier that treats `claude -p` as a black-box runtime environment. This architecture ensures that every skill operates correctly within the actual Claude-Code CLI rather than against mocked interfaces, providing high-fidelity validation of the full execution pipeline.

## Core Architecture of the gstack E2E Test Infrastructure

The testing framework centers on a Node.js harness that isolates each test case within a fresh `claude -p` subprocess. This approach guarantees that tests run against the exact binary environment that end users experience, eliminating discrepancies between test mocks and production behavior.

### Session Runner and Process Spawning

The [[`test/helpers/session-runner.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/session-runner.ts)](https://github.com/garrytan/gstack/blob/main/test/helpers/session-runner.ts) module constructs shell commands that feed prompts into the Claude CLI. It uses `spawnSync` to execute a command pattern similar to:

```bash
sh -c 'cat "$PROMPT_FILE" | claude -p --output-format stream-json --verbose'

```

This execution model pipes the test prompt directly into the subprocess via stdin, requesting structured NDJSON output through the `--output-format stream-json` flag. The synchronous spawn ensures deterministic test flow while capturing the complete stdout stream for downstream parsing.

### Output Parsing and Provider Abstraction

Once the subprocess completes, [[`test/helpers/providers/claude.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/providers/claude.ts)](https://github.com/garrytan/gstack/blob/main/test/helpers/providers/claude.ts) handles the NDJSON deserialization. The provider normalizes streaming events into typed objects, exposing helper functions like `extractToolSummary` to inspect which tools were invoked. It specifically looks for error markers such as `is_error:true` within success responses, translating these into an `error_api` status for assertion logic.

### Interactive Testing with Pseudo-TTY

For scenarios requiring terminal interaction, [[`test/helpers/claude-pty-runner.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/claude-pty-runner.ts)](https://github.com/garrytan/gstack/blob/main/test/helpers/claude-pty-runner.ts) provides a pseudo-TTY wrapper around the Claude process. This component enables testing of interactive skills that depend on terminal capabilities, such as the sidebar-agent tests, while maintaining the same NDJSON output contract as the standard session runner.

## How the E2E Tests Execute

Each skill test follows a standardized four-phase execution pattern implemented across the `test/skill-e2e-*.test.ts` files. The following TypeScript snippet illustrates the canonical test flow found in the suite:

```typescript
// Phase 1: Construct CLI command with isolated environment
const cmd = `sh -c 'cat "$PROMPT_FILE" | claude -p \
  --output-format stream-json --verbose'`;

// Phase 2: Spawn process in temporary directory
const { stdout } = spawnSync('sh', ['-c', cmd], { 
  env: { ...process.env, ...extraEnv },
  cwd: os.tmpdir() 
});

// Phase 3: Parse NDJSON stream into event objects
const events = parseNDJSON(stdout.toString());

// Phase 4: Assert on tool usage and error states
expect(events).toContainEqual(
  expect.objectContaining({ subtype: 'tool', name: 'Read' })
);

```

The `parseNDJSON` function, defined in the provider layer, converts the raw stream into an array of event objects that tests can query for specific tool calls, error conditions, or response content.

## Test Isolation and Cost Management

The gstack E2E test infrastructure using claude -p implements several safeguards to ensure reliable, cost-effective execution:

- **Process Isolation**: Every test runs within a temporary working directory created via `os.tmpdir()`, preventing file system pollution and ensuring clean state between test cases.
- **Cost Controls**: Each full suite execution costs approximately **$3.85** in API and CLI overhead. The test suite remains intentionally constrained to 20-30 tests to manage CI expenses.
- **Runtime Expectations**: Complete tier execution averages **20 minutes** of wall-clock time, as documented in [[`ARCHITECTURE.md`](https://github.com/garrytan/gstack/blob/main/ARCHITECTURE.md)](https://github.com/garrytan/gstack/blob/main/ARCHITECTURE.md#e2e-via-claude-p).
- **Tool Restrictions**: Subprocesses invoke Claude with constrained toolsets such as `--allowedTools Read,Grep,Glob` (defined in `claude/SKILL.md.tmpl`), limiting the execution surface to deterministic operations.
- **Parallelization Strategy**: Tests execute sequentially by default, though the harness architecture supports parallelization for independent skill groups when CI resource budgets permit.

## Validating Skill Behavior

Concrete test expectations reside in files like [[`test/skill-e2e.test.ts`](https://github.com/garrytan/gstack/blob/main/test/skill-e2e.test.ts)](https://github.com/garrytan/gstack/blob/main/test/skill-e2e.test.ts) and specialized variants such as [`test/skill-e2e-plan.test.ts`](https://github.com/garrytan/gstack/blob/main/test/skill-e2e-plan.test.ts). These files verify three critical dimensions:

1. **Skill Routing**: Confirming that the correct skill activates for a given prompt pattern
2. **Tool Invocation**: Asserting that allowed tools execute with correct parameters (e.g., `subtype: 'tool'`, `name: 'Read'`)
3. **Error Detection**: Validating that API failures surface correctly through `is_error:true` markers in the NDJSON stream

The test framework parses the verbose output to distinguish between successful tool executions and error states that might otherwise appear as successful API responses with embedded error flags.

## Summary

- The **session runner** in [`test/helpers/session-runner.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/session-runner.ts) spawns isolated `claude -p` subprocesses using shell command construction and `spawnSync`.
- The **provider layer** in [`test/helpers/providers/claude.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/providers/claude.ts) parses NDJSON output and normalizes error states for programmatic assertions.
- **Pseudo-TTY support** via [`test/helpers/claude-pty-runner.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/claude-pty-runner.ts) enables interactive skill testing when terminal emulation is required.
- Tests execute in **temporary directories** to guarantee isolation, with each run costing approximately **$3.85** and the full suite completing in roughly **20 minutes**.
- The infrastructure validates **tool usage**, **error detection**, and **end-to-end workflow correctness** against the actual Claude-Code CLI binary.

## Frequently Asked Questions

### How does the gstack E2E test infrastructure handle Claude API errors?

The provider implementation in [`test/helpers/providers/claude.ts`](https://github.com/garrytan/gstack/blob/main/test/helpers/providers/claude.ts) inspects every NDJSON event for `is_error:true` fields. When detected, these markers translate to an `error_api` status that tests can assert against, ensuring the framework correctly identifies failures even when the subprocess exits successfully.

### Why does gstack use `claude -p` instead of mocking the Claude API?

Using the real `claude -p` subprocess validates the entire execution stack including the Claude-Code CLI itself, skill routing logic, and tool permission boundaries. This approach catches integration issues that unit tests with mocked APIs would miss, such as unexpected tool restrictions or CLI argument parsing errors.

### What is the typical cost and runtime for the full E2E test suite?

According to the project documentation in [`ARCHITECTURE.md`](https://github.com/garrytan/gstack/blob/main/ARCHITECTURE.md) and [`CONTRIBUTING.md`](https://github.com/garrytan/gstack/blob/main/CONTRIBUTING.md), the complete E2E tier costs approximately **$3.85 per run** and executes in roughly **20 minutes**. The suite intentionally limits coverage to 20-30 critical path tests to maintain these cost and time constraints.

### How are test prompts isolated from the repository during execution?

The session runner creates a temporary working directory using `os.tmpdir()` for each test invocation. This ensures that the `claude -p` subprocess operates in a clean environment without access to the repository's source files, preventing pollution of the AI's context with implementation details that could skew test results.