How Experiment Cleanup and Resource Recovery Work in ChaosBlade

ChaosBlade destroys experiments by injecting a destroy flag into the execution context, invoking the original executor in rollback mode to revert faults, and persisting the final state in a SQLite database, with optional force-removal of Kubernetes resources via the --force-remove flag.

ChaosBlade tracks every fault injection in a lightweight SQLite-style database and creates corresponding runtime resources, from local files to Kubernetes Custom Resources. When you execute blade destroy UID, the framework coordinates a three-phase cleanup pipeline that ensures faults are reverted and resources are recovered regardless of the target environment. This article examines the source code in the chaosblade-io/chaosblade repository to trace the exact code paths that handle experiment destruction, database updates, and forced resource removal.

The Experiment Cleanup and Resource Recovery Pipeline

The entry point for all cleanup operations is DestroyCommand.runDestroyWithUid in cli/cmd/destroy.go. This method first queries the internal database for the experiment record associated with the provided UID.

If the record exists, the command calls destroyAndRemoveExperimentByUidAndForceFlag. If the record is missing—common when Kubernetes experiments are created directly via the API rather than the CLI—it falls back to destroyAndRemoveK8sExperimentWithoutRecordByForceFlag.

// cli/cmd/destroy.go L76-92
// Query the DB for the experiment model.
// Branch based on existence and target type.

Injecting the Destroy Flag

Before any rollback logic executes, ChaosBlade injects a destroy flag into the context. This flag signals to all downstream executors (Docker, JVM, Kubernetes) that they must enter cleanup mode rather than injection mode.

// cli/cmd/destroy.go L209-211
ctx := spec.SetDestroyFlag(context.Background(), uid)

The constant spec.DestroyFlag is inspected by executor implementations to switch behavior from fault injection to recovery.

Executing Rollback Logic in destroyExperiment

The core cleanup logic resides in destroyExperiment, which reconstructs the original ExpModel and invokes the same executor that created the fault.

// cli/cmd/destroy.go L207-218
func (dc *DestroyCommand) destroyExperiment(uid string, executor spec.Executor, expModel *spec.ExpModel) error {
    ctx := spec.SetDestroyFlag(context.Background(), uid)   // set destroy flag
    ctx = context.WithValue(ctx, spec.Uid, uid)
    response := executor.Exec(uid, ctx, expModel)          // executor runs its destroy path
    if !response.Success {
        return response
    }
    checkError(GetDS().UpdateExperimentModelByUid(uid, Destroyed, ""))
    return nil
}

This design isolates cleanup logic within the same executor that performed the injection, ensuring symmetry between creation and destruction phases.

Updating Experiment Status

After successful execution, the framework updates the experiment record in data/experiment.go. The UpdateExperimentModelByUid method writes the Destroyed status and current timestamp to the SQLite backend.

// data/experiment.go L64-71
stmt, err := s.DB.Prepare(`UPDATE experiment
    SET status = ?, error = ?, update_time = ?
    WHERE uid = ?`)

The experiment table schema defines states including Created, Success, Error, and Destroyed, providing a complete audit trail of the experiment lifecycle.

Force-Removing Orphaned Resources with --force-remove

When an executor fails or a Kubernetes finalizer prevents resource deletion, the --force-remove flag triggers two additional cleanup steps defined in cli/cmd/destroy.go:

Step	Function	Description
Kubernetes CR deletion	checkAndForceRemoveForK8sExp	Calls kubernetes.RemoveFinalizer to delete the ChaosBlade custom resource by removing the finalizer that keeps it alive after failed destruction.
Database record deletion	checkAndForceRemoveForExpRecord	Executes DeleteExperimentModelByUid to remove the experiment row from the database even if the destroy executor failed.

// cli/cmd/destroy.go L191-196
func (dc *DestroyCommand) checkAndForceRemoveForK8sExp(name, kubeconfig, proxyURL string) error {
    if dc.forceRemove {
        return kubernetes.RemoveFinalizer(name, kubeconfig, proxyURL, dc.token)
    }
    return nil
}

// cli/cmd/destroy.go L198-204
func (dc *DestroyCommand) checkAndForceRemoveForExpRecord(uid string) error {
    if dc.forceRemove {
        return GetDS().DeleteExperimentModelByUid(uid)
    }
    return nil
}

This guarantees that no orphan resources persist in the cluster or local database when manual intervention is required.

Revoking Pre-Execution Preparations

Beyond experiment destruction, ChaosBlade manages preparations—sandbox processes like JVM attachments or C++ socket hooks—that require separate cleanup via blade revoke UID.

The RevokeCommand in cli/cmd/revoke.go handles type-specific detachment:

// cli/cmd/revoke.go L51-71
switch record.ProgramType {
case PrepareJvmType:
    response = jvm.Detach(ctx, record.Port)
case PrepareCPlusType:
    response = cplus.Revoke(ctx, record.Port)
case PrepareK8sType:
    response = channel.Run(ctx, "kubectl", "delete ns chaosblade")
}

Upon successful detachment, the preparation status is updated to Revoked:

// cli/cmd/revoke.go L78-84
if response.Success {
    checkError(GetDS().UpdatePreparationRecordByUid(uid, Revoked, ""))
}

This bifurcated approach—destroy for experiments and revoke for preparations—ensures that both injected faults and their underlying execution environments are properly recovered.

Summary

• ChaosBlade uses a destroy flag (spec.SetDestroyFlag) injected into the execution context to signal rollback mode to executors, maintaining clean separation between injection and cleanup logic.
• The destroyExperiment function in cli/cmd/destroy.go orchestrates the rollback by invoking the original executor, then updates the SQLite database status to Destroyed via UpdateExperimentModelByUid.
• The --force-remove flag provides safety nets by calling kubernetes.RemoveFinalizer for stuck Kubernetes resources and DeleteExperimentModelByUid for database records when standard cleanup fails.
• Preparations (JVM, C++ sandboxes) are cleaned up via blade revoke, which updates records to Revoked status in a separate code path at cli/cmd/revoke.go.
• All cleanup operations preserve an audit trail through state transitions in the lightweight SQLite database, ensuring observability of the entire experiment lifecycle.

Frequently Asked Questions

What happens if blade destroy is called on a UID that doesn't exist in the database?

If the experiment record is missing—such as when a Kubernetes experiment was created directly via the API rather than the CLI—DestroyCommand.runDestroyWithUid falls back to destroyAndRemoveK8sExperimentWithoutRecordByForceFlag. This path attempts to clean up the Kubernetes ChaosBlade custom resource directly without relying on local database state, though it requires the --force-remove flag to guarantee deletion.

How does ChaosBlade ensure that the same executor handles both creation and destruction?

The framework retrieves the executor instance associated with the experiment's target type from the registry and passes it to destroyExperiment. By injecting the spec.DestroyFlag into the context before calling executor.Exec, the same code path that injected the fault recognizes the flag and executes rollback-specific logic, ensuring the exact reverse operations are performed for every fault type.

What is the difference between blade destroy and blade revoke?

blade destroy terminates active experiments (network delays, CPU stress, etc.) and updates the experiment status to Destroyed, whereas blade revoke cleans up preparations—the sandbox environments like JVM attachments or C++ agents required for certain fault types. Destroy handles the fault itself; revoke handles the infrastructure that enabled the fault.

When should I use the --force-remove flag?

Use --force-remove when an experiment's executor has crashed, a Kubernetes finalizer is preventing CR deletion, or the database record is corrupted and blocking cleanup. This flag skips error checks for the destroy executor and directly invokes kubernetes.RemoveFinalizer for cluster resources and DeleteExperimentModelByUid for the local database entry, ensuring complete resource recovery even in failure scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}