CubeSandbox Configuration for Storage Options: TOML Settings, Architecture, and Backend Setup

CubeSandbox provides a pluggable storage subsystem configured via TOML that supports copy-on-write backends through the cubecow engine, with settings for pool types, paths, and logging defined in the Config struct.

CubeSandbox (TencentCloud/CubeSandbox) implements a flexible storage layer within its Cubelet component, allowing operators to customize volume backends, pool behavior, and persistence settings through a declarative configuration file. The storage plugin, registered as an internal component via constants.InternalPlugin in Cubelet/storage/plugin.go, parses TOML settings to initialize either standard block storage or reflink-based copy-on-write pools. Understanding the available CubeSandbox configuration for storage options enables precise control over disk allocation, metadata management, and the Rust-backed cubecow engine initialization.

Core Storage Architecture

The storage subsystem follows a modular design where a central plugin coordinates between configuration parsing, pool management, and backend-specific volume operations.

Component Overview

Component	Role	Key Source File
plugin.go	Registers the plugin via registry.Register, parses TOML configuration, validates external dependencies, and boots the storage engine.	Cubelet/storage/plugin.go
local.go	Implements the Storage interface, manages per-sandbox storage entries, persists metadata in a local database, and tracks storage lifecycle.	Cubelet/storage/local.go
pool.go	Abstracts block-device pools, handling allocation, reclamation, and metrics for both standard and reflink-based storage.	Cubelet/storage/pool.go
cubecow_engine.go	Wraps the Rust-implemented cubecow copy-on-write engine, generating JSON payloads and initializing the engine via cubecow.InitWithoutLoggingFromJSON.	Cubelet/storage/cubecow_engine.go
cubecow_volume_manager.go	Formats reflink-backed files as ext4, mounts them, and prepares them for use as default-medium volumes.	Cubelet/storage/cubecow_volume_manager.go

Initialization Flow

The storage plugin initializes through a strict sequence defined in Cubelet/storage/plugin.go:

1. Registration – The plugin registers itself via registry.Register (lines 53-62), establishing it as an internal plugin.
2. Path Resolution – RootPath defaults to the state directory (plugins.PropertyStateDir); DataPath defaults to RootPath or receives the suffix io.cubelet.internal.v1.storage.
3. Pool Type Detection – When PoolType is set to copy_reflink, the checkPoolType function attempts to create a temporary ext4 image using reflink; on failure, it falls back to cp_type.
4. Backend Validation – For StorageBackend = "cubecow", the plugin validates required external binaries (mkfs.ext4, mount, umount, losetup) through validateCowStartupDeps.
5. Engine Initialization – initCowEngine builds a JSON payload via BuildCowInitJSON containing kind = "reflink" and an auto-generated root_dir, then invokes cubecow.InitWithoutLoggingFromJSON to create the engine instance stored in localStorage.cowEngine.

Configuration Schema and Options

The TOML schema mirrors the Config struct defined in Cubelet/storage/plugin.go (lines 56-97). All storage settings reside under the [storage] table.

TOML Configuration Fields

TOML Key	Type	Description	Default / Derivation
root_path	string	Directory for persistent Cubelet state (e.g., /var/lib/cubelet).	Derived from plugins.PropertyStateDir if empty.
data_path	string	Base directory for storage data.	Defaults to root_path unless overridden, appending <plugin>.<id> suffix when customized.
disksize	string	Desired size for default-medium volumes (e.g., 10Gi).	No default; required for new volumes.
warningPercent	int64	Percentage of free space at which to emit warnings.	No default.
pool_default_format_size_list	[]string	Sizes to pre-create as template images (e.g., ["1Gi","10Gi"]).	Empty slice.
base_disk_uuid	string	UUID embedded in base ext4 images for identification.	Empty.
pool_size	int	Maximum template images kept in the pool.	Zero (unlimited).
pool_worker_num	int	Concurrent workers for pool maintenance.	Zero (single-threaded).
pool_type	string	Backend type: cp_type (standard) or copy_reflink (reflink COW).	cp_type
pool_trigger_interval_in_ms	int	Interval for periodic pool maintenance tasks.	Zero (disabled).
pool_trigger_burst	int	Burst size for pool-trigger operations.	Zero.
fadvise_size	int	Size hint for posix_fadvise calls on allocated files.	Zero (no hint).
disable_disk_check	bool	Skip sanity checks on underlying block devices.	false
free_blocks_threshold	int32	Minimum free blocks before pool is considered "low".	Zero.
free_inodes_threshold	int32	Minimum free inodes before pool is considered "low".	Zero.
reconcile_interval	duration	Frequency of storage reconciler runs for stale entry cleanup.	Zero (no periodic reconcile).
storage_backend	string	Backend identifier: must be "cubecow" (the only supported value).	Must be explicitly set to cubecow.
cmd_timeout	duration	Timeout for external utilities invoked in storage/shell.go.	defaultCmdTimeout (3 seconds).

The cow Logging Sub-table

The cow table configures the Rust cubecow engine's logging behavior. Only the log sub-block is user-configurable; the backend block is auto-populated by PrepareCowInlineConfig.

[cow.log]
level = "info"      # "debug", "info", "warn"

format = "json"     # "text" or "json"

file = "/var/log/cubecow.log"
rotation = "daily"  # "daily" or "size:10M"

The backend configuration containing kind = "reflink" and root_dir is generated automatically during initialization.

Practical Configuration Examples

Sample TOML Configuration

This configuration enables reflink-based copy-on-write storage with custom logging and a 20Gi default volume size:

[storage]
root_path = "/var/lib/cubelet"
data_path = "/var/lib/cubelet/io.cubelet.internal.v1.storage"
disksize = "20Gi"
warningPercent = 10
pool_type = "copy_reflink"
storage_backend = "cubecow"
cmd_timeout = "5s"

[cow.log]
level = "info"
format = "text"
file = "/var/log/cubecow.log"
rotation = "size:10M"

Programmatic Config Initialization

When integrating with the Cubelet API directly, load and prepare the configuration as implemented in initCowEngine:

import (
    "github.com/tencentcloud/CubeSandbox/Cubelet/storage"
    "github.com/tencentcloud/CubeSandbox/Cubelet/internal/tomlext"
)

// Load Config from TOML into cfg variable
err := cfg.PrepareCowInlineConfig()
if err != nil {
    // Handle configuration preparation error
}

payload, err := cfg.BuildCowInitJSON()
if err != nil {
    // Handle JSON build error
}

// Initialize the cubecow engine
engine, src, err := cubecow.InitWithoutLoggingFromJSON(string(payload))
if err != nil {
    // Handle engine initialization failure
}
// Engine is now ready for volume operations

Creating Default-Medium Volumes

After engine initialization, create volumes through the local storage interface:

vol, err := localStorage.CreateDefaultMediumVolume(ctx, sandboxID, "10Gi")
if err != nil {
    log.Fatalf("Failed to create volume: %v", err)
}
fmt.Printf("Created reflink-backed volume at %s\n", vol.Path)

This method invokes cubecow_volume_manager.go to format the reflink file as ext4 and mount it for sandbox use.

Summary

• CubeSandbox storage configuration centers on the Config struct in Cubelet/storage/plugin.go, supporting TOML-based customization of paths, pool behavior, and backend selection.
• Two pool types are available: standard block storage (cp_type) and reflink-based copy-on-write (copy_reflink), with the latter requiring mkfs.ext4, mount, umount, and losetup utilities.
• The cubecow backend is the default and recommended engine, implemented in Rust and initialized via JSON payloads built by BuildCowInitJSON and PrepareCowInlineConfig.
• Initialization sequence includes plugin registration, path resolution, pool type validation, dependency checking, and engine creation through cubecow.InitWithoutLoggingFromJSON.
• Configuration files typically reside alongside cubelet.toml, with storage settings nested under the [storage] table and optional [cow.log] sub-table for engine logging.

Frequently Asked Questions

What is the difference between cp_type and copy_reflink pool types?

cp_type uses standard block device copying for volume creation, while copy_reflink leverages filesystem reflink capabilities for copy-on-write operations. The reflink option requires ext4 with reflink support and significantly reduces storage overhead and creation time for duplicate volumes. When pool_type is set to copy_reflink, the system validates support by attempting to create a temporary ext4 image before falling back to cp_type on failure.

How do I configure logging for the cubecow storage backend?

Add a [cow.log] section to your TOML configuration with level, format, file, and rotation fields. Valid levels include debug, info, and warn; formats are text or json. The rotation setting accepts daily or size-based patterns like size:10M. These settings are processed by PrepareCowInlineConfig and embedded into the JSON payload passed to the Rust engine during initialization.

Why does my storage initialization fail with "missing external dependencies"?

The cubecow backend validates that four specific binaries are present in PATH: mkfs.ext4, mount, umount, and losetup. This check occurs in validateCowStartupDeps within Cubelet/storage/plugin.go. Ensure these utilities are installed and accessible to the Cubelet process, particularly when using privileged container runtimes or minimal base images.

Can I change the storage configuration without restarting Cubelet?

No. The storage plugin reads configuration during initialization in initCowEngine and localStorage setup. Changes to root_path, pool_type, or storage_backend require a restart to re-trigger the registry.Register sequence and pool initialization logic. However, dynamic parameters like warningPercent thresholds may be adjustable through the Cubelet API depending on your specific build configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}