# CubeSandbox Configuration for Storage Options: TOML Settings, Architecture, and Backend Setup

> Learn to configure CubeSandbox storage options using TOML. Explore cubecow engine, pool types, paths, and logging settings for robust backend setup.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-13

---

**CubeSandbox provides a pluggable storage subsystem configured via TOML that supports copy-on-write backends through the cubecow engine, with settings for pool types, paths, and logging defined in the `Config` struct.**

CubeSandbox (TencentCloud/CubeSandbox) implements a flexible storage layer within its Cubelet component, allowing operators to customize volume backends, pool behavior, and persistence settings through a declarative configuration file. The storage plugin, registered as an internal component via `constants.InternalPlugin` in [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go), parses TOML settings to initialize either standard block storage or reflink-based copy-on-write pools. Understanding the available CubeSandbox configuration for storage options enables precise control over disk allocation, metadata management, and the Rust-backed cubecow engine initialization.

## Core Storage Architecture

The storage subsystem follows a modular design where a central plugin coordinates between configuration parsing, pool management, and backend-specific volume operations.

### Component Overview

| Component | Role | Key Source File |
| --- | --- | --- |
| **plugin.go** | Registers the plugin via `registry.Register`, parses TOML configuration, validates external dependencies, and boots the storage engine. | [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go) |
| **local.go** | Implements the `Storage` interface, manages per-sandbox storage entries, persists metadata in a local database, and tracks storage lifecycle. | [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go) |
| **pool.go** | Abstracts block-device pools, handling allocation, reclamation, and metrics for both standard and reflink-based storage. | [`Cubelet/storage/pool.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool.go) |
| **cubecow_engine.go** | Wraps the Rust-implemented `cubecow` copy-on-write engine, generating JSON payloads and initializing the engine via `cubecow.InitWithoutLoggingFromJSON`. | [`Cubelet/storage/cubecow_engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/cubecow_engine.go) |
| **cubecow_volume_manager.go** | Formats reflink-backed files as ext4, mounts them, and prepares them for use as default-medium volumes. | [`Cubelet/storage/cubecow_volume_manager.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/cubecow_volume_manager.go) |

### Initialization Flow

The storage plugin initializes through a strict sequence defined in [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go):

1. **Registration** – The plugin registers itself via `registry.Register` (lines 53-62), establishing it as an internal plugin.
2. **Path Resolution** – `RootPath` defaults to the state directory (`plugins.PropertyStateDir`); `DataPath` defaults to `RootPath` or receives the suffix `io.cubelet.internal.v1.storage`.
3. **Pool Type Detection** – When `PoolType` is set to `copy_reflink`, the `checkPoolType` function attempts to create a temporary ext4 image using reflink; on failure, it falls back to `cp_type`.
4. **Backend Validation** – For `StorageBackend = "cubecow"`, the plugin validates required external binaries (`mkfs.ext4`, `mount`, `umount`, `losetup`) through `validateCowStartupDeps`.
5. **Engine Initialization** – `initCowEngine` builds a JSON payload via `BuildCowInitJSON` containing `kind = "reflink"` and an auto-generated `root_dir`, then invokes `cubecow.InitWithoutLoggingFromJSON` to create the engine instance stored in `localStorage.cowEngine`.

## Configuration Schema and Options

The TOML schema mirrors the `Config` struct defined in [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go) (lines 56-97). All storage settings reside under the `[storage]` table.

### TOML Configuration Fields

| TOML Key | Type | Description | Default / Derivation |
| --- | --- | --- | --- |
| `root_path` | `string` | Directory for persistent Cubelet state (e.g., `/var/lib/cubelet`). | Derived from `plugins.PropertyStateDir` if empty. |
| `data_path` | `string` | Base directory for storage data. | Defaults to `root_path` unless overridden, appending `<plugin>.<id>` suffix when customized. |
| `disksize` | `string` | Desired size for default-medium volumes (e.g., `10Gi`). | No default; required for new volumes. |
| `warningPercent` | `int64` | Percentage of free space at which to emit warnings. | No default. |
| `pool_default_format_size_list` | `[]string` | Sizes to pre-create as template images (e.g., `["1Gi","10Gi"]`). | Empty slice. |
| `base_disk_uuid` | `string` | UUID embedded in base ext4 images for identification. | Empty. |
| `pool_size` | `int` | Maximum template images kept in the pool. | Zero (unlimited). |
| `pool_worker_num` | `int` | Concurrent workers for pool maintenance. | Zero (single-threaded). |
| `pool_type` | `string` | Backend type: `cp_type` (standard) or `copy_reflink` (reflink COW). | `cp_type` |
| `pool_trigger_interval_in_ms` | `int` | Interval for periodic pool maintenance tasks. | Zero (disabled). |
| `pool_trigger_burst` | `int` | Burst size for pool-trigger operations. | Zero. |
| `fadvise_size` | `int` | Size hint for `posix_fadvise` calls on allocated files. | Zero (no hint). |
| `disable_disk_check` | `bool` | Skip sanity checks on underlying block devices. | `false` |
| `free_blocks_threshold` | `int32` | Minimum free blocks before pool is considered "low". | Zero. |
| `free_inodes_threshold` | `int32` | Minimum free inodes before pool is considered "low". | Zero. |
| `reconcile_interval` | `duration` | Frequency of storage reconciler runs for stale entry cleanup. | Zero (no periodic reconcile). |
| `storage_backend` | `string` | Backend identifier: must be `"cubecow"` (the only supported value). | Must be explicitly set to `cubecow`. |
| `cmd_timeout` | `duration` | Timeout for external utilities invoked in [`storage/shell.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/storage/shell.go). | `defaultCmdTimeout` (3 seconds). |

### The `cow` Logging Sub-table

The `cow` table configures the Rust cubecow engine's logging behavior. Only the `log` sub-block is user-configurable; the backend block is auto-populated by `PrepareCowInlineConfig`.

```toml
[cow.log]
level = "info"      # "debug", "info", "warn"

format = "json"     # "text" or "json"

file = "/var/log/cubecow.log"
rotation = "daily"  # "daily" or "size:10M"

```

The backend configuration containing `kind = "reflink"` and `root_dir` is generated automatically during initialization.

## Practical Configuration Examples

### Sample TOML Configuration

This configuration enables reflink-based copy-on-write storage with custom logging and a 20Gi default volume size:

```toml
[storage]
root_path = "/var/lib/cubelet"
data_path = "/var/lib/cubelet/io.cubelet.internal.v1.storage"
disksize = "20Gi"
warningPercent = 10
pool_type = "copy_reflink"
storage_backend = "cubecow"
cmd_timeout = "5s"

[cow.log]
level = "info"
format = "text"
file = "/var/log/cubecow.log"
rotation = "size:10M"

```

### Programmatic Config Initialization

When integrating with the Cubelet API directly, load and prepare the configuration as implemented in `initCowEngine`:

```go
import (
    "github.com/tencentcloud/CubeSandbox/Cubelet/storage"
    "github.com/tencentcloud/CubeSandbox/Cubelet/internal/tomlext"
)

// Load Config from TOML into cfg variable
err := cfg.PrepareCowInlineConfig()
if err != nil {
    // Handle configuration preparation error
}

payload, err := cfg.BuildCowInitJSON()
if err != nil {
    // Handle JSON build error
}

// Initialize the cubecow engine
engine, src, err := cubecow.InitWithoutLoggingFromJSON(string(payload))
if err != nil {
    // Handle engine initialization failure
}
// Engine is now ready for volume operations

```

### Creating Default-Medium Volumes

After engine initialization, create volumes through the local storage interface:

```go
vol, err := localStorage.CreateDefaultMediumVolume(ctx, sandboxID, "10Gi")
if err != nil {
    log.Fatalf("Failed to create volume: %v", err)
}
fmt.Printf("Created reflink-backed volume at %s\n", vol.Path)

```

This method invokes [`cubecow_volume_manager.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubecow_volume_manager.go) to format the reflink file as ext4 and mount it for sandbox use.

## Summary

- **CubeSandbox storage configuration** centers on the `Config` struct in [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go), supporting TOML-based customization of paths, pool behavior, and backend selection.
- **Two pool types** are available: standard block storage (`cp_type`) and reflink-based copy-on-write (`copy_reflink`), with the latter requiring `mkfs.ext4`, `mount`, `umount`, and `losetup` utilities.
- **The cubecow backend** is the default and recommended engine, implemented in Rust and initialized via JSON payloads built by `BuildCowInitJSON` and `PrepareCowInlineConfig`.
- **Initialization sequence** includes plugin registration, path resolution, pool type validation, dependency checking, and engine creation through `cubecow.InitWithoutLoggingFromJSON`.
- **Configuration files** typically reside alongside [`cubelet.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet.toml), with storage settings nested under the `[storage]` table and optional `[cow.log]` sub-table for engine logging.

## Frequently Asked Questions

### What is the difference between `cp_type` and `copy_reflink` pool types?

**`cp_type`** uses standard block device copying for volume creation, while **`copy_reflink`** leverages filesystem reflink capabilities for copy-on-write operations. The reflink option requires ext4 with reflink support and significantly reduces storage overhead and creation time for duplicate volumes. When `pool_type` is set to `copy_reflink`, the system validates support by attempting to create a temporary ext4 image before falling back to `cp_type` on failure.

### How do I configure logging for the cubecow storage backend?

Add a `[cow.log]` section to your TOML configuration with `level`, `format`, `file`, and `rotation` fields. Valid levels include `debug`, `info`, and `warn`; formats are `text` or `json`. The rotation setting accepts `daily` or size-based patterns like `size:10M`. These settings are processed by `PrepareCowInlineConfig` and embedded into the JSON payload passed to the Rust engine during initialization.

### Why does my storage initialization fail with "missing external dependencies"?

The cubecow backend validates that four specific binaries are present in `PATH`: `mkfs.ext4`, `mount`, `umount`, and `losetup`. This check occurs in `validateCowStartupDeps` within [`Cubelet/storage/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/plugin.go). Ensure these utilities are installed and accessible to the Cubelet process, particularly when using privileged container runtimes or minimal base images.

### Can I change the storage configuration without restarting Cubelet?

**No.** The storage plugin reads configuration during initialization in `initCowEngine` and `localStorage` setup. Changes to `root_path`, `pool_type`, or `storage_backend` require a restart to re-trigger the `registry.Register` sequence and pool initialization logic. However, dynamic parameters like `warningPercent` thresholds may be adjustable through the Cubelet API depending on your specific build configuration.