gstack E2E Test Infrastructure Using claude -p: A Deep Dive into AI-Driven Integration Testing

The gstack E2E test infrastructure using claude -p validates AI-driven skills by spawning real Claude-Code subprocesses, piping test prompts via stdin, and parsing NDJSON output to assert on tool invocations and error states.

The garrytan/gstack repository employs a sophisticated end-to-end testing tier that treats claude -p as a black-box runtime environment. This architecture ensures that every skill operates correctly within the actual Claude-Code CLI rather than against mocked interfaces, providing high-fidelity validation of the full execution pipeline.

Core Architecture of the gstack E2E Test Infrastructure

The testing framework centers on a Node.js harness that isolates each test case within a fresh claude -p subprocess. This approach guarantees that tests run against the exact binary environment that end users experience, eliminating discrepancies between test mocks and production behavior.

Session Runner and Process Spawning

The [test/helpers/session-runner.ts](https://github.com/garrytan/gstack/blob/main/test/helpers/session-runner.ts) module constructs shell commands that feed prompts into the Claude CLI. It uses spawnSync to execute a command pattern similar to:

sh -c 'cat "$PROMPT_FILE" | claude -p --output-format stream-json --verbose'

This execution model pipes the test prompt directly into the subprocess via stdin, requesting structured NDJSON output through the --output-format stream-json flag. The synchronous spawn ensures deterministic test flow while capturing the complete stdout stream for downstream parsing.

Output Parsing and Provider Abstraction

Once the subprocess completes, [test/helpers/providers/claude.ts](https://github.com/garrytan/gstack/blob/main/test/helpers/providers/claude.ts) handles the NDJSON deserialization. The provider normalizes streaming events into typed objects, exposing helper functions like extractToolSummary to inspect which tools were invoked. It specifically looks for error markers such as is_error:true within success responses, translating these into an error_api status for assertion logic.

Interactive Testing with Pseudo-TTY

For scenarios requiring terminal interaction, [test/helpers/claude-pty-runner.ts](https://github.com/garrytan/gstack/blob/main/test/helpers/claude-pty-runner.ts) provides a pseudo-TTY wrapper around the Claude process. This component enables testing of interactive skills that depend on terminal capabilities, such as the sidebar-agent tests, while maintaining the same NDJSON output contract as the standard session runner.

How the E2E Tests Execute

Each skill test follows a standardized four-phase execution pattern implemented across the test/skill-e2e-*.test.ts files. The following TypeScript snippet illustrates the canonical test flow found in the suite:

// Phase 1: Construct CLI command with isolated environment
const cmd = `sh -c 'cat "$PROMPT_FILE" | claude -p \
  --output-format stream-json --verbose'`;

// Phase 2: Spawn process in temporary directory
const { stdout } = spawnSync('sh', ['-c', cmd], { 
  env: { ...process.env, ...extraEnv },
  cwd: os.tmpdir() 
});

// Phase 3: Parse NDJSON stream into event objects
const events = parseNDJSON(stdout.toString());

// Phase 4: Assert on tool usage and error states
expect(events).toContainEqual(
  expect.objectContaining({ subtype: 'tool', name: 'Read' })
);

The parseNDJSON function, defined in the provider layer, converts the raw stream into an array of event objects that tests can query for specific tool calls, error conditions, or response content.

Test Isolation and Cost Management

The gstack E2E test infrastructure using claude -p implements several safeguards to ensure reliable, cost-effective execution:

• Process Isolation: Every test runs within a temporary working directory created via os.tmpdir(), preventing file system pollution and ensuring clean state between test cases.
• Cost Controls: Each full suite execution costs approximately $3.85 in API and CLI overhead. The test suite remains intentionally constrained to 20-30 tests to manage CI expenses.
• Runtime Expectations: Complete tier execution averages 20 minutes of wall-clock time, as documented in [ARCHITECTURE.md](https://github.com/garrytan/gstack/blob/main/ARCHITECTURE.md#e2e-via-claude-p).
• Tool Restrictions: Subprocesses invoke Claude with constrained toolsets such as --allowedTools Read,Grep,Glob (defined in claude/SKILL.md.tmpl), limiting the execution surface to deterministic operations.
• Parallelization Strategy: Tests execute sequentially by default, though the harness architecture supports parallelization for independent skill groups when CI resource budgets permit.

Validating Skill Behavior

Concrete test expectations reside in files like [test/skill-e2e.test.ts](https://github.com/garrytan/gstack/blob/main/test/skill-e2e.test.ts) and specialized variants such as test/skill-e2e-plan.test.ts. These files verify three critical dimensions:

1. Skill Routing: Confirming that the correct skill activates for a given prompt pattern
2. Tool Invocation: Asserting that allowed tools execute with correct parameters (e.g., subtype: 'tool', name: 'Read')
3. Error Detection: Validating that API failures surface correctly through is_error:true markers in the NDJSON stream

The test framework parses the verbose output to distinguish between successful tool executions and error states that might otherwise appear as successful API responses with embedded error flags.

Summary

• The session runner in test/helpers/session-runner.ts spawns isolated claude -p subprocesses using shell command construction and spawnSync.
• The provider layer in test/helpers/providers/claude.ts parses NDJSON output and normalizes error states for programmatic assertions.
• Pseudo-TTY support via test/helpers/claude-pty-runner.ts enables interactive skill testing when terminal emulation is required.
• Tests execute in temporary directories to guarantee isolation, with each run costing approximately $3.85 and the full suite completing in roughly 20 minutes.
• The infrastructure validates tool usage, error detection, and end-to-end workflow correctness against the actual Claude-Code CLI binary.

Frequently Asked Questions

How does the gstack E2E test infrastructure handle Claude API errors?

The provider implementation in test/helpers/providers/claude.ts inspects every NDJSON event for is_error:true fields. When detected, these markers translate to an error_api status that tests can assert against, ensuring the framework correctly identifies failures even when the subprocess exits successfully.

Why does gstack use claude -p instead of mocking the Claude API?

Using the real claude -p subprocess validates the entire execution stack including the Claude-Code CLI itself, skill routing logic, and tool permission boundaries. This approach catches integration issues that unit tests with mocked APIs would miss, such as unexpected tool restrictions or CLI argument parsing errors.

What is the typical cost and runtime for the full E2E test suite?

According to the project documentation in ARCHITECTURE.md and CONTRIBUTING.md, the complete E2E tier costs approximately $3.85 per run and executes in roughly 20 minutes. The suite intentionally limits coverage to 20-30 critical path tests to maintain these cost and time constraints.

How are test prompts isolated from the repository during execution?

The session runner creates a temporary working directory using os.tmpdir() for each test invocation. This ensures that the claude -p subprocess operates in a clean environment without access to the repository's source files, preventing pollution of the AI's context with implementation details that could skew test results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}