How to Debug Failed or Hanging Chaos Experiments in ChaosBlade

Enable the --debug flag when running blade commands to expose detailed execution logs that reveal why ChaosBlade experiments fail or hang.

ChaosBlade executes chaos experiments through a hierarchy of Cobra commands that invoke specialized executors for OS, Docker, Kubernetes, JVM, and cloud resources. When you need to debug failed or hanging chaos experiments, the built-in debug logger provides visibility into every step of the execution pipeline, from command parsing to binary invocation.

Enable Debug Mode in ChaosBlade

The root CLI command defines a global --debug (or -d) flag that toggles the util.Debug variable used by the chaosblade-spec-go/log package.

In cli/cmd/cli.go, the flag is registered as follows:

// cli/cmd/cli.go – registers the global debug flag
flags.BoolVarP(&util.Debug, "debug", "d", false, "Set client to DEBUG mode")

When util.Debug is set to true, every log.Debugf call throughout the codebase prints to STDOUT. If you are running ChaosBlade in server mode, these logs appear in the server process output.

To enable debug output for any experiment, prepend --debug to your command:

# Run a CPU load experiment with full debug output

blade --debug exec cpu fullload --timeout 30

Interpreting Debug Output to Debug Failed or Hanging Chaos Experiments

Every executor logs its internal steps using log.Debugf, allowing you to trace the exact binary and arguments being executed. For example, the OS executor in exec/os/executor.go logs both the command invocation and the raw result:

// exec/os/executor.go – command execution logging
log.Debugf(ctx, "run command, %s %v", chaosOsBin, argsArray)
log.Debugf(ctx, "Command Result, output: %v, err: %v", outMsg, err)

Similar debug statements exist in exec/kubernetes/executor.go, exec/docker/executor.go, and other executors. By following the debug stream, you can identify:

• The exact binary and arguments launched by the executor
• Raw stdout and stderr from the underlying chaos tools
• Timeout handling and retry logic (the addTimeoutFlag in cli/cmd/exp.go adds a default --timeout flag to prevent indefinite hangs)
• Errors returned by the spec-go channel layer (logged via log.Errorf in files like cli/cmd/create.go)

Common Causes of Failed or Hanging Chaos Experiments

Symptom	Likely Cause	Debug Clues
No output, process stays alive	Missing --timeout causes the executor to wait indefinitely	Debug log never reaches the "Command Result" line; check the executor's log.Debugf output before the command
Permission denied / command not found	The underlying tool (e.g., chaos_os) is not on $PATH or lacks privileges	Debug log shows the full command line; verify the binary exists on the host
K8s / Docker executor returns "exec not found"	Wrong container ID / pod name, or the container runtime is not reachable	Debug log from exec/kubernetes/executor.go prints the target selector and API call
Unexpected HTTP 5xx from the server	Server-side error, often a database or RPC issue	cli/cmd/create.go logs the full JSON response with log.Debugf
Experiment completes but never disappears from blade query	The server did not mark the experiment as finished (e.g., panic)	Server logs (when running blade server start) show stack traces; enable --debug on the server binary

Step-by-Step Debugging Procedure

Follow this systematic approach to debug failed or hanging chaos experiments:

1. Re-run the experiment with --debug

   blade --debug <exp> <target> <action> [flags]

2. Watch the debug stream for the "run command" line — this confirms the exact tool and arguments being invoked.

3. If the command never returns, verify the timeout flag is set. Add an explicit timeout to prevent indefinite hangs:

   blade --debug <exp> ... --timeout 60

4. If you see an error, the debug log includes the error message (e.g., log.Errorf). Use that message to locate the failing executor file path shown in the source link.

5. When using server mode, start the server with --debug as well:

   blade server start --debug

   This provides client-side debug output plus server-side logs for a complete picture.

6. Inspect the experiment record with blade query (or blade query --uid <uid>) to verify status fields that the server updates.

Advanced Debugging Techniques

For complex scenarios, consider these additional approaches:

• Custom log levels: The log package respects util.LogLevel, though it is currently unused in the CLI. If you need finer-grained control, patch cli/cmd/cli.go to expose a --log-level flag.

• Redirect server logs: In server mode, pipe the server's stdout to a file for later analysis:

  blade server start --debug > server.log 2>&1 &

• Inspect generated spec files: Malformed chaosblade-*-spec-*.yaml files can cause missing flags and subsequent hangs. Verify these specifications are correct when troubleshooting executor behavior.

Summary

• Enable debug mode with the global --debug flag to expose internal execution logs via util.Debug and log.Debugf calls throughout the codebase.
• Use debug output from executors like exec/os/executor.go and exec/kubernetes/executor.go to see exact command lines, raw output, and error details.
• Always set an explicit --timeout (added via cli/cmd/exp.go) to prevent experiments from hanging indefinitely.
• When running in server mode, enable --debug on both client and server to capture full request/response cycles logged in cli/cmd/create.go.
• Query experiment status with blade query to verify server-side state updates.

Frequently Asked Questions

Why does my ChaosBlade experiment hang indefinitely?

Experiments hang when the --timeout flag is omitted because the executor waits indefinitely for the underlying command to complete. The addTimeoutFlag function in cli/cmd/exp.go adds this flag to every action, but if you do not specify a value, the default is often no timeout. Run your command with --debug to confirm the executor is stuck at the "run command" log line without reaching "Command Result."

How do I see the exact command ChaosBlade is executing?

Enable debug mode with blade --debug. The executor logs the precise binary and arguments before invocation. For example, exec/os/executor.go logs log.Debugf(ctx, "run command, %s %v", chaosOsBin, argsArray), allowing you to copy the command and run it manually on the host to verify permissions or binary availability.

What should I check when the Kubernetes executor fails?

When debugging Kubernetes experiments, look for the debug output in exec/kubernetes/executor.go that prints the target selector and API call. Common failures include incorrect pod names, missing container IDs, or unreachable container runtimes. The debug log will show the exact selector used (e.g., pod name, namespace) and any error returned by the Kubernetes API, helping you verify the target resources exist and are accessible.

Where are server-side errors logged when using blade server?

Server-side errors are logged to the server's stdout/stderr. Start the server with blade server start --debug to enable verbose logging. The cli/cmd/create.go file logs the full JSON response from the server with log.Debugf, so running the client with --debug as well will show you the complete request/response cycle, including any HTTP 5xx errors or panic stack traces from the server.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}