Common Failure Modes When Injecting Faults Into Containers With ChaosBlade
Most failures when injecting faults into containers with ChaosBlade occur during the executor resolution and validation phase—before the actual fault is applied—and typically present as "exec not found" errors, invalid container ID messages, or daemon connectivity issues.
Understanding how the chaosblade-io/chaosblade repository handles container experiments helps you diagnose why fault injection fails. The tool delegates container operations to external executor modules (chaosblade-exec-docker and chaosblade-exec-cri), which means many failures surface during command lookup and validation rather than during the actual resource manipulation.
How ChaosBlade Executes Container Faults
ChaosBlade supports container fault injection through two distinct execution paths defined in the core repository:
- Docker scope – Implemented in [
exec/docker/spec.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/docker/spec.go) and [exec/docker/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/docker/executor.go), targeting native Docker daemons. - CRI scope – Implemented in [
exec/cri/spec.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/cri/spec.go) and [exec/cri/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/cri/executor.go), supporting containerd, CRI-O, and other CRI-compliant runtimes.
Both paths follow the same resolution flow:
- Parse CLI request – The command model validates supported actions and flags.
- Build executor key –
exec.GetExecutorKey(model.Target, model.ActionName)constructs a key likedocker/remove. - Lookup executor map – The map is populated by external packages (
exec.GetAllExecutors()). - Execute or fail – If the key exists, the external module runs the fault; otherwise, it returns a not-found error.
Because the concrete fault-logic lives in external modules, most failure modes when injecting faults into containers surface during steps 1–3, not during the actual resource modification.
Common Failure Modes When Injecting Faults Into Containers
Executor Not Found (DockerExecNotFound / CriExecNotFound)
The most frequent error occurs when the requested action is not registered in the executor map. In [exec/docker/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/docker/executor.go) lines 46–48, the code returns spec.DockerExecNotFound when the lookup fails. The CRI equivalent in [exec/cri/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/cri/executor.go) lines 46–48 returns spec.CriExecNotFound.
This happens when:
- The external
chaosblade-exec-dockerorchaosblade-exec-crimodule is missing or outdated. - You request an unsupported action (e.g.,
docker/foo). - The action exists for Docker but not for CRI, or vice versa.
# This will fail because "foo" is not a registered action
blade create docker foo --container-id 65eead213dd3
Output:
Error: docker exec not found: docker/foo
Missing or Invalid Container Identifiers
Container ID errors propagate from the external executor when the runtime cannot locate the target. According to [exec/cri/spec.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/cri/spec.go) line 43, when using containerd, you must provide the full container ID, not the truncated display name.
Missing the --container-id flag triggers early validation errors via spec.ParameterInvalid in the model layer.
# Missing required flag
blade create docker remove
Output:
Error: required flag "container-id" not set
Permission and Daemon Connectivity Errors
When the Docker daemon is unreachable or the user lacks permissions, the external executor returns a low-level connection error that ChaosBlade wraps into a failed Response with a non-zero Code. These errors surface after the executor is found but before the container is modified.
# Docker daemon stopped or insufficient permissions
blade create docker remove --container-id abcdef123456
Typical output includes phrases like "permission denied" or "cannot connect to the Docker daemon."
CLI Validation Failures
Scope mismatches occur when you use a docker target while the experiment scope is set to host or k8s. The CLI registration logic in [cli/cmd/exp.go](https://github.com/chaosblade-io/chaosblade/blob/master/cli/cmd/exp.go) validates scope compatibility during command attachment, rejecting mismatched combinations before execution.
Missing required flags (e.g., omitting --container-id) abort immediately with spec.ParameterInvalid from the spec validation layer in chaosblade-spec-go.
Runtime Incompatibility and Concurrent Conflicts
Attempting a Docker-only action (like specific network emulations) on a containerd node fails because the CRI executor map lacks that action key, triggering the CriExecNotFound error.
Concurrent experiment conflicts surface when the Kubernetes operator layer returns a conflict status for duplicate experiments, or when the external executor rejects overlapping fault injection on the same container resource.
Examples of Container Fault Injection Failures
The following examples demonstrate actual failure outputs based on the ChaosBlade source code implementation:
1. Successful lookup with runtime failure (permission denied)
sudo systemctl stop docker
blade create docker remove --container-id 65eead213dd3
The executor resolves correctly, but the external module fails to instantiate the Docker client, returning a wrapped error through log.Errorf in [exec/docker/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/docker/executor.go).
2. CRI action not supported
blade create cri remove --container-id 65eead213dd3
If chaosblade-exec-cri does not implement the remove action, you receive:
Error: cri exec not found: cri/remove
3. Validation error with correct scope but missing flag
blade create docker remove --namespace default
This fails because --container-id is mandatory for the remove action, triggering the parameter validation defined in the command model.
Summary
When troubleshooting failure modes when injecting faults into containers in ChaosBlade:
- Executor not found errors indicate missing external modules (
chaosblade-exec-docker/cri) or unsupported actions—check that the action exists for your target runtime. - Validation errors happen before execution—verify required flags like
--container-idare present and that containerd targets use the full 64-character ID. - Permission and connectivity issues surface as runtime errors from the external executor—ensure the Docker daemon or CRI socket is accessible.
- Scope mismatches prevent command registration—align your
--targetflag (dockervscri) with your actual container runtime.
Frequently Asked Questions
Why does ChaosBlade return "docker exec not found" when I know Docker is installed?
This error originates in [exec/docker/executor.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/docker/executor.go) lines 46–48 and indicates the action-specific executor is missing from the registry, not that Docker itself is missing. Download the corresponding chaosblade-exec-docker binary or verify the action is supported for Docker (some actions are CRI-only).
What is the difference between Docker and CRI targets in ChaosBlade?
The Docker target uses the native Docker daemon API via exec/docker/spec.go, while the CRI target interfaces with containerd, CRI-O, and other CRI-compliant runtimes via exec/cri/spec.go. CRI requires full container IDs, whereas Docker accepts short IDs.
How do I resolve "required flag container-id not set" errors?
The container ID is mandatory for most container actions. Check the specific action's ExpFlags definition in the external executor module; if using containerd, ensure you provide the full 64-character container ID as noted in [exec/cri/spec.go](https://github.com/chaosblade-io/chaosblade/blob/master/exec/cri/spec.go).
Why do container experiments fail with "permission denied" even when run as root?
This usually indicates the Docker daemon socket is not accessible or the ChaosBlade process lacks CAP_SYS_ADMIN capabilities for CRI operations. Verify the Docker daemon is running (systemctl status docker) and that your user has access to /var/run/docker.sock or the CRI endpoint specified in your kubelet configuration.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" Maintain an open-source project? Get it listed too →