# How to Handle Experiments That Exceed Timeout or Hang Indefinitely in ChaosBlade

> Discover how ChaosBlade manages experiments exceeding timeout or hanging indefinitely using background execution and automatic destroy commands for reliable chaos engineering.

- Repository: [ChaosBlade/chaosblade](https://github.com/chaosblade-io/chaosblade)
- Tags: how-to-guide
- Published: 2026-02-27

---

**ChaosBlade handles experiments that exceed timeout or hang indefinitely by combining a user-defined timeout flag with a process-hang execution mode that starts commands in the background, then schedules an automatic destroy command via a detached shell process that runs after the timeout expires.**

ChaosBlade treats timeout and process hang as orthogonal concepts to ensure no experiment blocks the Blade CLI forever. Whether you are burning CPU on a host or injecting network latency into a Kubernetes pod, the `chaosblade-io/chaosblade` source code provides multiple safeguards to prevent runaway experiments.

## Understanding Timeout and Process Hang in ChaosBlade

ChaosBlade distinguishes between two failure modes:

- **Timeout**: A user-defined limit (in seconds or Go duration format) after which the experiment should be automatically destroyed.
- **Process Hang**: An execution mode where the experiment process runs indefinitely in the background (e.g., a CPU burn loop), requiring the CLI to exit immediately while the chaos continues.

These concepts work together: a hanging process can still respect a timeout via automatic cleanup, while a non-hanging process can exceed its deadline and be terminated.

## How the Timeout Flag Works Under the Hood

### Flag Definition and Injection

The timeout flag is defined centrally in [`build/spec/spec.go`](https://github.com/chaosblade-io/chaosblade/blob/main/build/spec/spec.go) and added to every action's flag list:

```go
{Name: "timeout", Desc: "set timeout for experiment"}

```

In [`cli/cmd/exp.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/exp.go), the `addTimeoutFlag` function ensures every action inherits this flag if not explicitly declared:

```go
if !contains {
    flags = append(flags, &spec.ExpFlag{Name: "timeout", Desc: "set timeout for experiment"})
}

```

### Validation and Parsing

When you run `blade create`, the `RunE` function in [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) validates the timeout value. It accepts either a plain integer (interpreted as seconds) or a Go duration string (`10s`, `2m`, `1h`):

```go
if tt != "" {
    if _, err := strconv.ParseUint(tt, 10, 64); err != nil {
        if _, err := time.ParseDuration(tt); err != nil {
            return err
        }
    }
}

```

### Storage in the Experiment Model

After validation, `createExpModel` copies all visited flags, including `timeout`, into `ExpModel.ActionFlags`. This ensures the timeout value persists with the experiment metadata for later cleanup stages.

## Handling Process Hang Experiments

### Detecting Hang-Capable Actions

Each action's specification can declare itself as a hanging process via the `ProcessHang()` method. This is common in OS, middleware, Docker, CRI, and cloud executors. When creating the experiment, [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) copies this flag to `ExpModel.ActionProcessHang`.

### Executor Implementation

In [`exec/os/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/os/executor.go) (and similar files for middleware, Docker, and cloud), the executor distinguishes between normal and hang modes:

- **Normal mode**: Calls `command.CombinedOutput()` and waits for completion, returning the decoded JSON response.
- **Hang mode**: If `ActionProcessHang && !isDestroy`, the executor calls `command.Start()` and immediately returns the PID via `spec.ReturnSuccess(command.Process.Pid)`. The real command continues running in the background.

```go
// Simplified logic from exec/os/executor.go
if expModel.ActionProcessHang && !isDestroy {
    err := command.Start()
    if err != nil {
        return spec.ReturnFail(spec.Code[spec.ExecError], err.Error())
    }
    return spec.ReturnSuccess(command.Process.Pid)
}
// Normal execution
output, err := command.CombinedOutput()

```

### PID Tracking and Status Verification

For hanging processes, the CLI checks PID existence using `gopsutil` after execution. The experiment is marked as **Success** only if the background process is still alive, confirming the hang is active.

## Kubernetes-Specific Timeout Handling

### Waiting Time and Context Timeouts

Kubernetes experiments use an additional `waitingTime` flag (default **20s**) defined in [`exec/kubernetes/spec.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/spec.go). In [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go), this creates a `context.WithTimeout` that limits how long the driver polls the Custom Resource status:

```go
// From exec/kubernetes/executor.go
ctx, cancel := context.WithTimeout(context.Background(), waitingTime)
defer cancel()
// Poll resource status until timeout or success

```

If the timeout is reached before the resource reports success, the executor stops waiting and returns the current status, preventing indefinite blocking on cluster operations.

## Automatic Cleanup After Timeout

### The Post-Run Destroy Mechanism

After an experiment finishes, `actionPostRunEFunc` in [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) schedules automatic cleanup if a timeout was specified and the experiment is not asynchronous. It constructs a detached shell command:

```bash
nohup /bin/sh -c 'sleep <timeout>; <blade> destroy <uid>' &

```

This ensures that even if the original `blade` process crashes, a separate shell process will destroy the experiment after the sleep duration expires.

### Container and Pod Scope Adjustments

For `container` and `pod` scopes, the post-run logic adds an extra **60 seconds** to the timeout. This safety margin accounts for the Kubernetes controller's reporting latency, ensuring the destroy command only runs after the underlying infrastructure has fully processed the chaos injection.

## Practical Examples

Create a CPU burn experiment that stops automatically after 30 seconds:

```bash
blade create host cpu burn --timeout 30

```

Run a Kubernetes network loss experiment with custom polling and total timeout:

```bash
blade create k8s mypod network loss --waitingTime 45s --timeout 120

```

Force a process-hang experiment to run asynchronously (CLI exits immediately while chaos continues):

```bash
blade create host cpu burn --async true

# Blade prints the UID and returns; the actual burn process continues in the background

```

Manually destroy an experiment before its scheduled timeout:

```bash
blade destroy <uid>

```

## Summary

- **Timeout flags** are automatically injected into every action via [`build/spec/spec.go`](https://github.com/chaosblade-io/chaosblade/blob/main/build/spec/spec.go) and [`cli/cmd/exp.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/exp.go), ensuring users can always specify a limit.
- **Validation** in [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) accepts integer seconds or Go duration strings (`10s`, `2m`).
- **Process hang mode** allows experiments like CPU burn to run indefinitely in the background, returning immediately with a PID while the chaos continues.
- **Kubernetes experiments** use a separate `waitingTime` flag with `context.WithTimeout` to prevent indefinite polling of cluster resources.
- **Automatic cleanup** is scheduled via a detached `nohup` shell command that sleeps for the timeout duration then destroys the experiment, with a 60-second buffer for container and pod scopes.

## Frequently Asked Questions

### What happens if I don't specify a timeout when creating a ChaosBlade experiment?

If you omit the timeout flag, the experiment runs without an automatic cleanup schedule. For non-hanging experiments, the CLI waits for completion indefinitely. For hanging experiments (like CPU burn), the process continues running in the background until you manually run `blade destroy <uid>` or the system reboots.

### How does ChaosBlade prevent the CLI from hanging when running Kubernetes experiments?

The Kubernetes executor in [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go) uses a `waitingTime` flag (default 20 seconds) combined with `context.WithTimeout`. This limits how long the CLI polls the Custom Resource status. If the timeout is reached before the experiment reports success, the CLI returns the current status rather than blocking indefinitely.

### Can I stop a hanging experiment before the timeout expires?

Yes. Hanging experiments return a UID immediately upon creation. You can terminate the experiment early by running `blade destroy <uid>`. This sends a destroy signal to the background process regardless of the scheduled timeout, which is handled by the `actionPostRunEFunc` cleanup logic in [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go).

### Why does ChaosBlade add 60 seconds to the timeout for container and pod experiments?

The post-run cleanup logic adds a 60-second safety margin specifically for `container` and `pod` scopes because the underlying Kubernetes controller requires additional time to report the final status of the chaos injection. This ensures the automatic destroy command only executes after the infrastructure has fully processed the experiment, preventing premature cleanup attempts.