How to Use Extra Kernel Cmdline Parameters for VM Customization in CubeSandbox

CubeSandbox supports custom kernel boot parameters through the cube.vm.kernel.cmdline.append annotation, which validates and injects JSON-formatted arguments into the KVM micro-VM's command line before guest memory initialization.

CubeSandbox builds a KVM micro-VM for each sandbox runtime, assembling the guest kernel command line dynamically within the hypervisor (VMM). You can customize this boot configuration—enabling kernel flags like net.ifnames=0 or debug consoles—without rebuilding the VM image by leveraging the annotation-based parameter injection system implemented in the CubeShim component.

How Kernel Command-Line Customization Works

The injection follows a strict four-phase pipeline that ensures internal stability while allowing flexible customization.

Base Command-Line Assembly

The foundation is built in hypervisor/vmm/src/vm.rs within the generate_cmdline() function (lines 38-51). This method creates a linux_loader::cmdline::Cmdline object and populates it with static payload settings, followed by internal parameters gathered from the device manager. These internal arguments include essential settings like quiet, highres=off, and clocksource=kvm-clock to ensure proper KVM guest operation.

Parsing the Annotation

CubeShim defines the annotation key cube.vm.kernel.cmdline.append in CubeShim/shim/src/sandbox/config.rs (lines 24-27). When Config::new() executes (lines 176-189), it parses this annotation using Utils::anno_to_obj::<Vec<String>>, expecting a JSON array of strings. Valid examples include ["net.ifnames=0","pci=noaer"] or ["console=ttyS0,115200n8"]. The parsed values are stored in Config.extra_kernel_params.

Conflict Detection

Before VM launch, CubeShim/shim/src/sandbox/sb.rs::build_vm_config() (lines 51-58) validates user-defined parameters against the existing command line via vc.check_cmdline_conflicts. If any extra parameter conflicts with an internal requirement, sandbox creation fails immediately with a descriptive error, preventing boot-time kernel panics.

Final Injection into Guest Memory

After passing validation, each custom parameter is appended via vc.add_cmdline(param) (lines 62-64 in sb.rs). The finalized command line is then written into the guest's boot memory by Vm::load_kernel in the VMM (hypervisor/vmm/src/vm.rs, lines 1022-1023) through the linux_loader::loader::load_cmdline mechanism.

Configuring Extra Kernel Parameters

You can supply custom boot arguments through YAML manifests or programmatically via the Go SDK.

YAML Template Configuration

Add the annotation to your Sandbox metadata:

apiVersion: cubesandbox.io/v1
kind: Sandbox
metadata:
  name: my-sandbox
  annotations:
    # Append kernel boot parameters

    cube.vm.kernel.cmdline.append: |
      ["net.ifnames=0", "pci=noaer", "console=ttyS0,115200n8"]
spec:
  # … other sandbox spec …

Go SDK Implementation

The Cubelet SDK exports the annotation constant for programmatic access:

import (
    cubesandbox "github.com/TencentCloud/CubeSandbox/Cubelet/pkg/client"
    "github.com/TencentCloud/CubeSandbox/Cubelet/pkg/constants"
)

sandbox := &cubesandbox.Sandbox{
    Metadata: &cubesandbox.Metadata{
        Name: "my-sandbox",
        Annotations: map[string]string{
            constants.AnnotationVMKernelCmdlineAppend: `["net.ifnames=0","pci=noaer"]`,
        },
    },
    // … spec …
}
client.CreateSandbox(context.Background(), sandbox)

Verifying the Effective Command Line

After the sandbox starts, inspect the effective boot parameters through CubeShim's request logs:

# Find the sandbox ID first

SID=$(cubecli get sandbox my-sandbox -o jsonpath='{.metadata.uid}')

# Grep the kernel boot line from the shim request log

grep "\"InstanceId\":\"$SID\"" /data/log/CubeShim/cube-shim-req.log | \
  jq -r '.LogContent' | grep "console="

This confirms that your extra kernel cmdline parameters were successfully appended to the guest kernel.

Key Source Files and Functions

Understanding these implementation details helps troubleshoot injection failures:

• CubeShim/shim/src/sandbox/config.rs – Declares ANNO_VM_KERNEL_CMDLINE_APPEND and parses the annotation into extra_kernel_params using Utils::anno_to_obj::<Vec<String>>.
• CubeShim/shim/src/sandbox/sb.rs – Contains build_vm_config() which validates conflicts via vc.check_cmdline_conflicts and injects parameters using vc.add_cmdline().
• hypervisor/vmm/src/vm.rs – Houses generate_cmdline() for base assembly and Vm::load_kernel for final memory writing via linux_loader.
• hypervisor/vmm/src/device_manager.rs – Provides internal command-line fragments (console, clocksource) that are merged before custom parameters.
• Cubelet/pkg/constants/const.go – Exports AnnotationVMKernelCmdlineAppend for SDK consumers.

Summary

• CubeSandbox uses the cube.vm.kernel.cmdline.append annotation to accept extra kernel cmdline parameters as a JSON string array.
• The CubeShim component validates parameters against internal requirements in sb.rs::build_vm_config() before injection.
• Valid parameters are appended after internal settings and loaded into guest memory via linux_loader in vm.rs::load_kernel.
• You can configure parameters via YAML annotations or the Go SDK constant AnnotationVMKernelCmdlineAppend.
• Use cubecli and CubeShim logs to verify the final effective command line inside the running micro-VM.

Frequently Asked Questions

What format should the cube.vm.kernel.cmdline.append value use?

The annotation requires a JSON array of strings, such as ["param1=value1","param2"]. Invalid JSON or non-string array formats will fail parsing during Config::new() in CubeShim/shim/src/sandbox/config.rs, preventing sandbox creation.

What happens if my custom parameter conflicts with internal ones?

CubeSandbox executes strict conflict detection in sb.rs::build_vm_config(). If your parameter duplicates or contradicts an internal setting (like clocksource), the sandbox creation fails immediately with a clear error message before the VM starts.

Can I override existing kernel parameters or only append new ones?

The current implementation only appends parameters. If you need to override an internal setting, you must ensure the internal value is not set by the device manager, or rely on the kernel's "last value wins" behavior for specific boot flags. The conflict checker prevents dangerous overrides that could destabilize the KVM micro-VM.

Where is the final command line physically stored in the VM?

The finalized command line is written into the guest's physical memory by Vm::load_kernel in hypervisor/vmm/src/vm.rs (lines 1022-1023) using linux_loader::loader::load_cmdline, making it available to the guest kernel at boot time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}