How ChaosBlade Handles Concurrent Experiments on the Same Target

ChaosBlade manages concurrent experiments through UID-level isolation backed by SQLite unique constraints, allowing multiple experiments to run simultaneously on the same target without state interference.

The chaosblade-io/chaosblade project treats every chaos experiment as an independent unit identified by a globally unique UID. When operators run blade create commands targeting the same container, pod, or host, the system leverages SQLite's serialization guarantees and schema-level uniqueness enforcement to prevent metadata conflicts. This architecture ensures that concurrent experiments on the same target remain isolated in the database while allowing simultaneous execution of chaos actions.

UID Generation and Collision Handling

Every experiment begins with the generation of a 32-character unique identifier that serves as the primary key for lifecycle management.

Recursive UID Generation with Collision Detection

In cli/cmd/command.go (lines 22‑34), the baseCommand.generateUid() method implements a collision-resistant generation strategy:

func (bc *baseCommand) generateUid() (string, error) {
    uid, err := util.GenerateUid()
    if err != nil { return "", err }
    model, err := GetDS().QueryExperimentModelByUid(uid)
    if err != nil { return "", err }
    if model == nil { return uid, nil }
    return bc.generateUid() // retry on collision
}

This function calls util.GenerateUid() and immediately queries the database via QueryExperimentModelByUid(). If the UID already exists—possible under extreme concurrency—it recursively generates a new one until a unique value is obtained.

Database-Level Uniqueness Constraints

The SQLite schema in data/experiment.go (lines 71‑75) enforces uniqueness at the storage layer:

uid VARCHAR(32) UNIQUE

This constraint guarantees that no two experiments can share the same UID even during concurrent inserts. The InsertExperimentModel function (lines 42‑58) attempts the insertion, and SQLite's atomic commit semantics ensure that only the first concurrent caller succeeds for any given UID.

Experiment Metadata Persistence

Once a unique UID is secured, ChaosBlade records the experiment metadata in a centralized SQLite database named chaosblade.dat.

Recording Experiment Models

The recordExpModel method in cli/cmd/command.go (lines 95‑111) constructs an ExperimentModel struct and persists it:

func (bc *baseCommand) recordExpModel(commandPath string, expModel *spec.ExpModel) (*data.ExperimentModel, *spec.Response) {
    uid := expModel.ActionFlags[UidFlag]
    if uid == "" { uid, err = bc.generateUid() ... }

    flagsInline := spec.ConvertExpMatchersToString(expModel, func() map[string]spec.Empty { return make(map[string]spec.Empty) })
    now := time.Now().Format(time.RFC3339Nano)
    cmd, sub, _ := parseCommandPath(commandPath)

    commandModel := &data.ExperimentModel{
        Uid: uid, Command: cmd, SubCommand: sub,
        Flag: flagsInline, Status: Created,
        CreateTime: now, UpdateTime: now,
    }
    _ = GetDS().InsertExperimentModel(commandModel) // INSERT INTO experiment …
    return commandModel, spec.ReturnSuccess(uid)
}

This captures the command, sub-command, flags, and initial Created status, ensuring every experiment has an independent database row.

Isolated Status Transitions

Throughout the experiment lifecycle, status updates target specific UIDs via UpdateExperimentModelByUid in data/experiment.go (lines 64‑78). Whether transitioning to Success, Error, or Destroyed, each update specifies the UID:

func (dc *DestroyCommand) destroyExperiment(uid string, executor spec.Executor, expModel *spec.ExpModel) error {
    // … executor runs …
    checkError(GetDS().UpdateExperimentModelByUid(uid, Destroyed, ""))
    return nil
}

Because updates are UID-scoped, concurrent experiments on the same target never overwrite each other’s state.

Database Concurrency Architecture

ChaosBlade relies on SQLite's internal concurrency model to serialize writes while allowing parallel reads.

Singleton Connection Management

The data/source.go file (lines 12‑20, 45‑58) implements a singleton database handle using sync.Once:

var (
    source *Source
    once   sync.Once
)

func GetSource() *Source {
    once.Do(func() {
        source = &Source{}
        source.open()
    })
    return source
}

The once.Do initialization guarantees a single *sql.DB handle per process, preventing race conditions during database file opening and ensuring that all goroutines share the same connection pool.

Write Serialization and Target Locking Behavior

SQLite serializes write operations at the database level. Because each experiment inserts and updates its own unique row, contention is limited to the brief moment of UID generation and insertion. The system does not implement a global lock per target (e.g., per container or host). Consequently, two experiments can execute simultaneously on the same target, with each maintaining independent lifecycle tracking via its UID.

Practical Examples

Running multiple experiments against the same target demonstrates the isolation:

# Start two experiments that target the same container `myapp`

blade create cpu load --cpu-percent 60 --target myapp

# → returns UID: 3f2e7a1d-9c4b-4f9b-a2c1-5b9e7d0f1a2b

blade create mem fill --mem-percent 50 --target myapp

# → returns UID: 7a9c3b0e-1d5f-4e2a-8b6c-3d4f9e0a6b7c

Both commands return distinct UIDs and create independent rows in the experiment table. The CPU load and memory fill actions execute concurrently, but their metadata remains isolated.

Summary

• UID-level isolation: Every experiment receives a unique 32-character identifier generated recursively to avoid collisions.
• SQLite unique constraints: The uid VARCHAR(32) UNIQUE schema prevents duplicate inserts even under concurrent load.
• Row-scoped updates: Status transitions use UpdateExperimentModelByUid, ensuring experiments never overwrite each other’s state.
• No global target locks: ChaosBlade allows simultaneous execution on the same target, delegating conflict prevention for the actual chaos actions (CPU, network, etc.) to the operator or the underlying implementation.

Frequently Asked Questions

Can two experiments run simultaneously on the same container?

Yes. ChaosBlade does not enforce mutual exclusion at the target level. Two experiments targeting the same container will receive unique UIDs and execute concurrently, though the operator should ensure the chaos actions themselves are compatible.

How does ChaosBlade prevent duplicate UIDs during concurrent creation?

The generateUid() function in cli/cmd/command.go checks for existing UIDs via QueryExperimentModelByUid() and recurses if a collision is detected. Additionally, the SQLite unique constraint on the uid column provides a hard guarantee at the database layer.

Does ChaosBlade stop conflicting chaos actions on the same target?

No. The framework tracks experiment metadata independently and does not validate whether a new experiment conflicts with an already-running action on the same target. The system assumes idempotence or operator-level coordination for conflicting actions like network partitioning.

What database handles the experiment state?

ChaosBlade uses an embedded SQLite database stored in chaosblade.dat, accessed through a singleton connection managed in data/source.go. SQLite's write serialization ensures that concurrent inserts and updates remain consistent without requiring external locking mechanisms.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}