How to Handle Experiments That Exceed Timeout or Hang Indefinitely in ChaosBlade

ChaosBlade handles experiments that exceed timeout or hang indefinitely by combining a user-defined timeout flag with a process-hang execution mode that starts commands in the background, then schedules an automatic destroy command via a detached shell process that runs after the timeout expires.

ChaosBlade treats timeout and process hang as orthogonal concepts to ensure no experiment blocks the Blade CLI forever. Whether you are burning CPU on a host or injecting network latency into a Kubernetes pod, the chaosblade-io/chaosblade source code provides multiple safeguards to prevent runaway experiments.

Understanding Timeout and Process Hang in ChaosBlade

ChaosBlade distinguishes between two failure modes:

• Timeout: A user-defined limit (in seconds or Go duration format) after which the experiment should be automatically destroyed.
• Process Hang: An execution mode where the experiment process runs indefinitely in the background (e.g., a CPU burn loop), requiring the CLI to exit immediately while the chaos continues.

These concepts work together: a hanging process can still respect a timeout via automatic cleanup, while a non-hanging process can exceed its deadline and be terminated.

How the Timeout Flag Works Under the Hood

Flag Definition and Injection

The timeout flag is defined centrally in build/spec/spec.go and added to every action's flag list:

{Name: "timeout", Desc: "set timeout for experiment"}

In cli/cmd/exp.go, the addTimeoutFlag function ensures every action inherits this flag if not explicitly declared:

if !contains {
    flags = append(flags, &spec.ExpFlag{Name: "timeout", Desc: "set timeout for experiment"})
}

Validation and Parsing

When you run blade create, the RunE function in cli/cmd/create.go validates the timeout value. It accepts either a plain integer (interpreted as seconds) or a Go duration string (10s, 2m, 1h):

if tt != "" {
    if _, err := strconv.ParseUint(tt, 10, 64); err != nil {
        if _, err := time.ParseDuration(tt); err != nil {
            return err
        }
    }
}

Storage in the Experiment Model

After validation, createExpModel copies all visited flags, including timeout, into ExpModel.ActionFlags. This ensures the timeout value persists with the experiment metadata for later cleanup stages.

Handling Process Hang Experiments

Detecting Hang-Capable Actions

Each action's specification can declare itself as a hanging process via the ProcessHang() method. This is common in OS, middleware, Docker, CRI, and cloud executors. When creating the experiment, cli/cmd/create.go copies this flag to ExpModel.ActionProcessHang.

Executor Implementation

In exec/os/executor.go (and similar files for middleware, Docker, and cloud), the executor distinguishes between normal and hang modes:

• Normal mode: Calls command.CombinedOutput() and waits for completion, returning the decoded JSON response.
• Hang mode: If ActionProcessHang && !isDestroy, the executor calls command.Start() and immediately returns the PID via spec.ReturnSuccess(command.Process.Pid). The real command continues running in the background.

// Simplified logic from exec/os/executor.go
if expModel.ActionProcessHang && !isDestroy {
    err := command.Start()
    if err != nil {
        return spec.ReturnFail(spec.Code[spec.ExecError], err.Error())
    }
    return spec.ReturnSuccess(command.Process.Pid)
}
// Normal execution
output, err := command.CombinedOutput()

PID Tracking and Status Verification

For hanging processes, the CLI checks PID existence using gopsutil after execution. The experiment is marked as Success only if the background process is still alive, confirming the hang is active.

Kubernetes-Specific Timeout Handling

Waiting Time and Context Timeouts

Kubernetes experiments use an additional waitingTime flag (default 20s) defined in exec/kubernetes/spec.go. In exec/kubernetes/executor.go, this creates a context.WithTimeout that limits how long the driver polls the Custom Resource status:

// From exec/kubernetes/executor.go
ctx, cancel := context.WithTimeout(context.Background(), waitingTime)
defer cancel()
// Poll resource status until timeout or success

If the timeout is reached before the resource reports success, the executor stops waiting and returns the current status, preventing indefinite blocking on cluster operations.

Automatic Cleanup After Timeout

The Post-Run Destroy Mechanism

After an experiment finishes, actionPostRunEFunc in cli/cmd/create.go schedules automatic cleanup if a timeout was specified and the experiment is not asynchronous. It constructs a detached shell command:

nohup /bin/sh -c 'sleep <timeout>; <blade> destroy <uid>' &

This ensures that even if the original blade process crashes, a separate shell process will destroy the experiment after the sleep duration expires.

Container and Pod Scope Adjustments

For container and pod scopes, the post-run logic adds an extra 60 seconds to the timeout. This safety margin accounts for the Kubernetes controller's reporting latency, ensuring the destroy command only runs after the underlying infrastructure has fully processed the chaos injection.

Practical Examples

Create a CPU burn experiment that stops automatically after 30 seconds:

blade create host cpu burn --timeout 30

Run a Kubernetes network loss experiment with custom polling and total timeout:

blade create k8s mypod network loss --waitingTime 45s --timeout 120

Force a process-hang experiment to run asynchronously (CLI exits immediately while chaos continues):

blade create host cpu burn --async true

# Blade prints the UID and returns; the actual burn process continues in the background

Manually destroy an experiment before its scheduled timeout:

blade destroy <uid>

Summary

• Timeout flags are automatically injected into every action via build/spec/spec.go and cli/cmd/exp.go, ensuring users can always specify a limit.
• Validation in cli/cmd/create.go accepts integer seconds or Go duration strings (10s, 2m).
• Process hang mode allows experiments like CPU burn to run indefinitely in the background, returning immediately with a PID while the chaos continues.
• Kubernetes experiments use a separate waitingTime flag with context.WithTimeout to prevent indefinite polling of cluster resources.
• Automatic cleanup is scheduled via a detached nohup shell command that sleeps for the timeout duration then destroys the experiment, with a 60-second buffer for container and pod scopes.

Frequently Asked Questions

What happens if I don't specify a timeout when creating a ChaosBlade experiment?

If you omit the timeout flag, the experiment runs without an automatic cleanup schedule. For non-hanging experiments, the CLI waits for completion indefinitely. For hanging experiments (like CPU burn), the process continues running in the background until you manually run blade destroy <uid> or the system reboots.

How does ChaosBlade prevent the CLI from hanging when running Kubernetes experiments?

The Kubernetes executor in exec/kubernetes/executor.go uses a waitingTime flag (default 20 seconds) combined with context.WithTimeout. This limits how long the CLI polls the Custom Resource status. If the timeout is reached before the experiment reports success, the CLI returns the current status rather than blocking indefinitely.

Can I stop a hanging experiment before the timeout expires?

Yes. Hanging experiments return a UID immediately upon creation. You can terminate the experiment early by running blade destroy <uid>. This sends a destroy signal to the background process regardless of the scheduled timeout, which is handled by the actionPostRunEFunc cleanup logic in cli/cmd/create.go.

Why does ChaosBlade add 60 seconds to the timeout for container and pod experiments?

The post-run cleanup logic adds a 60-second safety margin specifically for container and pod scopes because the underlying Kubernetes controller requires additional time to report the final status of the chaos injection. This ensures the automatic destroy command only executes after the infrastructure has fully processed the experiment, preventing premature cleanup attempts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}