# How Experiment Cleanup and Resource Recovery Work in ChaosBlade

> Learn how ChaosBlade cleans up experiments by reverting faults and persisting states. Discover resource recovery and the force removal option in Kubernetes.

- Repository: [ChaosBlade/chaosblade](https://github.com/chaosblade-io/chaosblade)
- Tags: internals
- Published: 2026-02-27

---

**ChaosBlade destroys experiments by injecting a destroy flag into the execution context, invoking the original executor in rollback mode to revert faults, and persisting the final state in a SQLite database, with optional force-removal of Kubernetes resources via the `--force-remove` flag.**

ChaosBlade tracks every fault injection in a lightweight SQLite-style database and creates corresponding runtime resources, from local files to Kubernetes Custom Resources. When you execute **`blade destroy UID`**, the framework coordinates a three-phase cleanup pipeline that ensures faults are reverted and resources are recovered regardless of the target environment. This article examines the source code in the `chaosblade-io/chaosblade` repository to trace the exact code paths that handle experiment destruction, database updates, and forced resource removal.

## The Experiment Cleanup and Resource Recovery Pipeline

The entry point for all cleanup operations is `DestroyCommand.runDestroyWithUid` in [`cli/cmd/destroy.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/destroy.go). This method first queries the internal database for the experiment record associated with the provided UID.

If the record exists, the command calls `destroyAndRemoveExperimentByUidAndForceFlag`. If the record is missing—common when Kubernetes experiments are created directly via the API rather than the CLI—it falls back to `destroyAndRemoveK8sExperimentWithoutRecordByForceFlag`.

```go
// cli/cmd/destroy.go L76-92
// Query the DB for the experiment model.
// Branch based on existence and target type.

```

### Injecting the Destroy Flag

Before any rollback logic executes, ChaosBlade injects a **destroy flag** into the context. This flag signals to all downstream executors (Docker, JVM, Kubernetes) that they must enter cleanup mode rather than injection mode.

```go
// cli/cmd/destroy.go L209-211
ctx := spec.SetDestroyFlag(context.Background(), uid)

```

The constant `spec.DestroyFlag` is inspected by executor implementations to switch behavior from fault injection to recovery.

## Executing Rollback Logic in `destroyExperiment`

The core cleanup logic resides in `destroyExperiment`, which reconstructs the original `ExpModel` and invokes the same executor that created the fault.

```go
// cli/cmd/destroy.go L207-218
func (dc *DestroyCommand) destroyExperiment(uid string, executor spec.Executor, expModel *spec.ExpModel) error {
    ctx := spec.SetDestroyFlag(context.Background(), uid)   // set destroy flag
    ctx = context.WithValue(ctx, spec.Uid, uid)
    response := executor.Exec(uid, ctx, expModel)          // executor runs its destroy path
    if !response.Success {
        return response
    }
    checkError(GetDS().UpdateExperimentModelByUid(uid, Destroyed, ""))
    return nil
}

```

This design isolates cleanup logic within the same executor that performed the injection, ensuring symmetry between creation and destruction phases.

### Updating Experiment Status

After successful execution, the framework updates the experiment record in [`data/experiment.go`](https://github.com/chaosblade-io/chaosblade/blob/main/data/experiment.go). The `UpdateExperimentModelByUid` method writes the **`Destroyed`** status and current timestamp to the SQLite backend.

```go
// data/experiment.go L64-71
stmt, err := s.DB.Prepare(`UPDATE experiment
    SET status = ?, error = ?, update_time = ?
    WHERE uid = ?`)

```

The `experiment` table schema defines states including `Created`, `Success`, `Error`, and `Destroyed`, providing a complete audit trail of the experiment lifecycle.

## Force-Removing Orphaned Resources with `--force-remove`

When an executor fails or a Kubernetes finalizer prevents resource deletion, the `--force-remove` flag triggers two additional cleanup steps defined in [`cli/cmd/destroy.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/destroy.go):

| Step | Function | Description |
|------|----------|-------------|
| **Kubernetes CR deletion** | `checkAndForceRemoveForK8sExp` | Calls `kubernetes.RemoveFinalizer` to delete the `ChaosBlade` custom resource by removing the finalizer that keeps it alive after failed destruction. |
| **Database record deletion** | `checkAndForceRemoveForExpRecord` | Executes `DeleteExperimentModelByUid` to remove the experiment row from the database even if the destroy executor failed. |

```go
// cli/cmd/destroy.go L191-196
func (dc *DestroyCommand) checkAndForceRemoveForK8sExp(name, kubeconfig, proxyURL string) error {
    if dc.forceRemove {
        return kubernetes.RemoveFinalizer(name, kubeconfig, proxyURL, dc.token)
    }
    return nil
}

// cli/cmd/destroy.go L198-204
func (dc *DestroyCommand) checkAndForceRemoveForExpRecord(uid string) error {
    if dc.forceRemove {
        return GetDS().DeleteExperimentModelByUid(uid)
    }
    return nil
}

```

This guarantees that no orphan resources persist in the cluster or local database when manual intervention is required.

## Revoking Pre-Execution Preparations

Beyond experiment destruction, ChaosBlade manages **preparations**—sandbox processes like JVM attachments or C++ socket hooks—that require separate cleanup via `blade revoke UID`.

The `RevokeCommand` in [`cli/cmd/revoke.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/revoke.go) handles type-specific detachment:

```go
// cli/cmd/revoke.go L51-71
switch record.ProgramType {
case PrepareJvmType:
    response = jvm.Detach(ctx, record.Port)
case PrepareCPlusType:
    response = cplus.Revoke(ctx, record.Port)
case PrepareK8sType:
    response = channel.Run(ctx, "kubectl", "delete ns chaosblade")
}

```

Upon successful detachment, the preparation status is updated to **`Revoked`**:

```go
// cli/cmd/revoke.go L78-84
if response.Success {
    checkError(GetDS().UpdatePreparationRecordByUid(uid, Revoked, ""))
}

```

This bifurcated approach—`destroy` for experiments and `revoke` for preparations—ensures that both injected faults and their underlying execution environments are properly recovered.

## Summary

- ChaosBlade uses a **destroy flag** (`spec.SetDestroyFlag`) injected into the execution context to signal rollback mode to executors, maintaining clean separation between injection and cleanup logic.
- The `destroyExperiment` function in [`cli/cmd/destroy.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/destroy.go) orchestrates the rollback by invoking the original executor, then updates the SQLite database status to `Destroyed` via `UpdateExperimentModelByUid`.
- The `--force-remove` flag provides safety nets by calling `kubernetes.RemoveFinalizer` for stuck Kubernetes resources and `DeleteExperimentModelByUid` for database records when standard cleanup fails.
- **Preparations** (JVM, C++ sandboxes) are cleaned up via `blade revoke`, which updates records to `Revoked` status in a separate code path at [`cli/cmd/revoke.go`](https://github.com/chaosblade-io/chaosblade/blob/main/cli/cmd/revoke.go).
- All cleanup operations preserve an audit trail through state transitions in the lightweight SQLite database, ensuring observability of the entire experiment lifecycle.

## Frequently Asked Questions

### What happens if `blade destroy` is called on a UID that doesn't exist in the database?

If the experiment record is missing—such as when a Kubernetes experiment was created directly via the API rather than the CLI—`DestroyCommand.runDestroyWithUid` falls back to `destroyAndRemoveK8sExperimentWithoutRecordByForceFlag`. This path attempts to clean up the Kubernetes `ChaosBlade` custom resource directly without relying on local database state, though it requires the `--force-remove` flag to guarantee deletion.

### How does ChaosBlade ensure that the same executor handles both creation and destruction?

The framework retrieves the executor instance associated with the experiment's target type from the registry and passes it to `destroyExperiment`. By injecting the `spec.DestroyFlag` into the context before calling `executor.Exec`, the same code path that injected the fault recognizes the flag and executes rollback-specific logic, ensuring the exact reverse operations are performed for every fault type.

### What is the difference between `blade destroy` and `blade revoke`?

`blade destroy` terminates active experiments (network delays, CPU stress, etc.) and updates the experiment status to `Destroyed`, whereas `blade revoke` cleans up **preparations**—the sandbox environments like JVM attachments or C++ agents required for certain fault types. Destroy handles the fault itself; revoke handles the infrastructure that enabled the fault.

### When should I use the `--force-remove` flag?

Use `--force-remove` when an experiment's executor has crashed, a Kubernetes finalizer is preventing CR deletion, or the database record is corrupted and blocking cleanup. This flag skips error checks for the destroy executor and directly invokes `kubernetes.RemoveFinalizer` for cluster resources and `DeleteExperimentModelByUid` for the local database entry, ensuring complete resource recovery even in failure scenarios.