# How to Debug Failed or Hanging Chaos Experiments in ChaosBlade

> Debug failed or hanging ChaosBlade experiments with the --debug flag. Uncover detailed logs to understand and resolve issues with your ChaosBlade experiments efficiently.

- Repository: [ChaosBlade/chaosblade](https://github.com/chaosblade-io/chaosblade)
- Tags: how-to-guide
- Published: 2026-02-27

---

**Enable the `--debug` flag when running `blade` commands to expose detailed execution logs that reveal why ChaosBlade experiments fail or hang.**

ChaosBlade executes chaos experiments through a hierarchy of Cobra commands that invoke specialized executors for OS, Docker, Kubernetes, JVM, and cloud resources. When you need to debug failed or hanging chaos experiments, the built-in debug logger provides visibility into every step of the execution pipeline, from command parsing to binary invocation.

## Enable Debug Mode in ChaosBlade

The root CLI command defines a global `--debug` (or `-d`) flag that toggles the `util.Debug` variable used by the `chaosblade-spec-go/log` package.

In [`cli/cmd/cli.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/cli.go), the flag is registered as follows:

```go
// cli/cmd/cli.go – registers the global debug flag
flags.BoolVarP(&util.Debug, "debug", "d", false, "Set client to DEBUG mode")

```

When `util.Debug` is set to `true`, every `log.Debugf` call throughout the codebase prints to **STDOUT**. If you are running ChaosBlade in server mode, these logs appear in the server process output.

To enable debug output for any experiment, prepend `--debug` to your command:

```bash

# Run a CPU load experiment with full debug output

blade --debug exec cpu fullload --timeout 30

```

## Interpreting Debug Output to Debug Failed or Hanging Chaos Experiments

Every executor logs its internal steps using `log.Debugf`, allowing you to trace the exact binary and arguments being executed. For example, the OS executor in [`exec/os/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/os/executor.go) logs both the command invocation and the raw result:

```go
// exec/os/executor.go – command execution logging
log.Debugf(ctx, "run command, %s %v", chaosOsBin, argsArray)
log.Debugf(ctx, "Command Result, output: %v, err: %v", outMsg, err)

```

Similar debug statements exist in [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go), [`exec/docker/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/docker/executor.go), and other executors. By following the debug stream, you can identify:

- **The exact binary and arguments** launched by the executor
- **Raw stdout and stderr** from the underlying chaos tools
- **Timeout handling and retry logic** (the `addTimeoutFlag` in [`cli/cmd/exp.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/exp.go) adds a default `--timeout` flag to prevent indefinite hangs)
- **Errors returned by the spec-go channel layer** (logged via `log.Errorf` in files like [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go))

## Common Causes of Failed or Hanging Chaos Experiments

| Symptom | Likely Cause | Debug Clues |
|---------|--------------|-------------|
| **No output, process stays alive** | Missing `--timeout` causes the executor to wait indefinitely | Debug log never reaches the "Command Result" line; check the executor's `log.Debugf` output before the command |
| **Permission denied / command not found** | The underlying tool (e.g., `chaos_os`) is not on `$PATH` or lacks privileges | Debug log shows the full command line; verify the binary exists on the host |
| **K8s / Docker executor returns "exec not found"** | Wrong container ID / pod name, or the container runtime is not reachable | Debug log from [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go) prints the target selector and API call |
| **Unexpected HTTP 5xx from the server** | Server-side error, often a database or RPC issue | [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) logs the full JSON response with `log.Debugf` |
| **Experiment completes but never disappears from `blade query`** | The server did not mark the experiment as finished (e.g., panic) | Server logs (when running `blade server start`) show stack traces; enable `--debug` on the server binary |

## Step-by-Step Debugging Procedure

Follow this systematic approach to debug failed or hanging chaos experiments:

1. **Re-run the experiment with `--debug`**
   ```bash
   blade --debug <exp> <target> <action> [flags]
   ```

2. **Watch the debug stream for the "run command" line** — this confirms the exact tool and arguments being invoked.

3. **If the command never returns**, verify the `timeout` flag is set. Add an explicit timeout to prevent indefinite hangs:
   ```bash
   blade --debug <exp> ... --timeout 60
   ```

4. **If you see an error**, the debug log includes the error message (e.g., `log.Errorf`). Use that message to locate the failing executor file path shown in the source link.

5. **When using server mode**, start the server with `--debug` as well:
   ```bash
   blade server start --debug
   ```

   This provides client-side debug output plus server-side logs for a complete picture.

6. **Inspect the experiment record** with `blade query` (or `blade query --uid <uid>`) to verify status fields that the server updates.

## Advanced Debugging Techniques

For complex scenarios, consider these additional approaches:

- **Custom log levels**: The `log` package respects `util.LogLevel`, though it is currently unused in the CLI. If you need finer-grained control, patch [`cli/cmd/cli.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/cli.go) to expose a `--log-level` flag.

- **Redirect server logs**: In server mode, pipe the server's stdout to a file for later analysis:
  ```bash
  blade server start --debug > server.log 2>&1 &
  ```

- **Inspect generated spec files**: Malformed `chaosblade-*-spec-*.yaml` files can cause missing flags and subsequent hangs. Verify these specifications are correct when troubleshooting executor behavior.

## Summary

- Enable **debug mode** with the global `--debug` flag to expose internal execution logs via `util.Debug` and `log.Debugf` calls throughout the codebase.
- Use debug output from executors like [`exec/os/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/os/executor.go) and [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go) to see exact command lines, raw output, and error details.
- Always set an explicit **`--timeout`** (added via [`cli/cmd/exp.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/exp.go)) to prevent experiments from hanging indefinitely.
- When running in **server mode**, enable `--debug` on both client and server to capture full request/response cycles logged in [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go).
- Query experiment status with `blade query` to verify server-side state updates.

## Frequently Asked Questions

### Why does my ChaosBlade experiment hang indefinitely?

Experiments hang when the `--timeout` flag is omitted because the executor waits indefinitely for the underlying command to complete. The `addTimeoutFlag` function in [`cli/cmd/exp.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/exp.go) adds this flag to every action, but if you do not specify a value, the default is often no timeout. Run your command with `--debug` to confirm the executor is stuck at the "run command" log line without reaching "Command Result."

### How do I see the exact command ChaosBlade is executing?

Enable debug mode with `blade --debug`. The executor logs the precise binary and arguments before invocation. For example, [`exec/os/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/os/executor.go) logs `log.Debugf(ctx, "run command, %s %v", chaosOsBin, argsArray)`, allowing you to copy the command and run it manually on the host to verify permissions or binary availability.

### What should I check when the Kubernetes executor fails?

When debugging Kubernetes experiments, look for the debug output in [`exec/kubernetes/executor.go`](https://github.com/chaosblade-io/chaosblade/blob/main/exec/kubernetes/executor.go) that prints the target selector and API call. Common failures include incorrect pod names, missing container IDs, or unreachable container runtimes. The debug log will show the exact selector used (e.g., pod name, namespace) and any error returned by the Kubernetes API, helping you verify the target resources exist and are accessible.

### Where are server-side errors logged when using `blade server`?

Server-side errors are logged to the server's stdout/stderr. Start the server with `blade server start --debug` to enable verbose logging. The [`cli/cmd/create.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/create.go) file logs the full JSON response from the server with `log.Debugf`, so running the client with `--debug` as well will show you the complete request/response cycle, including any HTTP 5xx errors or panic stack traces from the server.