How Soft Dirty Pages Enable Efficient Incremental Snapshots in CubeSandbox

CubeSandbox uses soft dirty pages to track per-snapshot memory modifications, generating minimal incremental deltas by intersecting write-tracked pages with copy-on-write anonymous pages.

CubeSandbox, TencentCloud's hypervisor, implements a sophisticated memory tracking system that leverages soft dirty pages to optimize incremental snapshot creation. This mechanism provides fine-grained visibility into guest memory modifications since the last snapshot, working alongside copy-on-write tracking to ensure consistency while minimizing delta size. When the host kernel supports CONFIG_MEM_SOFT_DIRTY, the VMM can track true per-snapshot deltas rather than accumulating all pages touched since boot.

Understanding Soft Dirty Pages vs. Anonymous Page Tracking

CubeSandbox employs two complementary mechanisms to identify which guest memory pages to write into an incremental snapshot:

Mechanism	What It Tracks	Usage
pagemap_anon	All pages that have become anonymous (Copy-on-Write) after restore. These pages may differ from the base image and must be examined for every snapshot.	Always available; works on any kernel.
soft dirty	Only pages written since the last clear_soft_dirty() call. The kernel clears bit 55 in /proc/self/pagemap on clear_refs(4), then sets it on subsequent writes.	Requires CONFIG_MEM_SOFT_DIRTY=y. Provides a true delta that resets each snapshot cycle.

The soft dirty mechanism tracks modifications at the PTE (Page Table Entry) level using bit 55 of the kernel's pagemap interface. Unlike the anonymous page tracker, which accumulates all CoW pages indefinitely, soft dirty pages provide a clean slate after each snapshot window.

The Soft Dirty Page Lifecycle in CubeSandbox

The implementation in hypervisor/vmm/src/soft_dirty.rs and hypervisor/vmm/src/memory_manager.rs follows a strict lifecycle to ensure snapshot consistency.

1. Probing and Arming the Tracker

During the first snapshot (base image creation), the VMM probes kernel support and initializes the tracking window. The clear_soft_dirty() function writes to clear_refs(4), which both tests for CONFIG_MEM_SOFT_DIRTY support and resets every page's soft-dirty bit.

// Probe kernel for soft‑dirty support; this also clears the bitmap.
if probe_soft_dirty_support() {
    // Tracker is now armed for the next window.
    self.soft_dirty_armed.store(true, Ordering::Release);
}

Source: send_soft_dirty_memory first-time path in hypervisor/vmm/src/memory_manager.rs (lines 21‑24).

2. Capturing the Delta Bitmap

Before writing subsequent incremental snapshots, the VMM reads the soft-dirty bitmap for each memory region using get_soft_dirty_pages. This function examines bit 55 in the pagemap to identify pages modified since the previous snapshot.

let dirty_pages = get_soft_dirty_pages(host_addr as u64, length)?;

Source: get_soft_dirty_pages implementation in hypervisor/vmm/src/soft_dirty.rs (lines 76‑48).

3. Intersecting with Anonymous Pages

To ensure correctness, the VMM calls filter_memory_ranges_by_anon_and_soft_dirty, which performs a bitwise intersection between the anonymous page set (CoW pages) and the soft-dirty set. Only pages that are both anonymous and soft-dirty are selected for the delta.

let anon_pages = get_anon_pages(host_addr as u64, length)?;
let must_save: Vec<bool> = anon_pages.iter()
    .zip(dirty_pages.iter())
    .map(|(&a, &d)| a && d)
    .collect();

Source: Intersection logic in hypervisor/vmm/src/soft_dirty.rs (lines 84‑88).

4. Writing the Delta and Re-Arming

The filtered pages overlay onto the existing base snapshot file. After writing the delta, the VMM must re-arm the tracker by calling clear_soft_dirty() again. If this call fails, the system silently disables soft-dirty tracking and falls back to the pagemap_anon method.

for range in filtered_ranges.regions() {
    let file_off = Self::calculate_file_offset_for_gpa(range.gpa, range.length, &gpa_to_file_offset)?;
    self.save_range_to_file(&memory_file, range, file_off)?;
}

// Re-arm for next window
if let Err(e) = clear_soft_dirty() {
    self.soft_dirty_armed.store(false, Ordering::Release);
}

Source: Overlay loop and re-arm handling in hypervisor/vmm/src/memory_manager.rs (lines 86‑90 and 98‑107).

Key Implementation Files

File	Purpose
hypervisor/vmm/src/soft_dirty.rs	Implements probing, clearing (clear_soft_dirty), bitmap extraction (get_soft_dirty_pages), and filtering logic.
hypervisor/vmm/src/memory_manager.rs	Coordinates snapshot creation, decides whether to use the soft-dirty path, and performs delta overlays.
hypervisor/vmm/src/pagemap_anon.rs	Provides CoW-only filtering used as fallback and for intersection with soft-dirty sets.

Performance and Consistency Benefits

True Incremental Snapshots: Because clear_soft_dirty() resets the dirty bits after each snapshot, the VMM records only pages that changed in the current window. This yields significantly smaller deltas than the pagemap_anon approach, which accumulates all CoW pages ever touched. The end-to-end test test_filter_memory_ranges_by_soft_dirty_end_to_end in hypervisor/vmm/src/soft_dirty.rs (lines 31‑41) verifies that only newly written pages appear in subsequent windows.

Controlled Pause Time: The primary cost is the kernel PTE walk during clear_refs(4). By invoking this only once per snapshot cycle, CubeSandbox limits pause time to a few hundred milliseconds even for multi-gigabyte VMs. The implementation logs these durations in hypervisor/vmm/src/soft_dirty.rs (lines 66‑71) for operational correlation.

Graceful Degradation: If the kernel lacks CONFIG_MEM_SOFT_DIRTY, the VMM automatically falls back to anonymous page tracking without error, ensuring compatibility across all Linux kernels. This probe and fallback logic resides in hypervisor/vmm/src/soft_dirty.rs (lines 21‑26).

Summary

• Soft dirty pages provide per-snapshot, fine-grained tracking of guest memory modifications via bit 55 in the kernel pagemap.
• CubeSandbox generates minimal deltas by intersecting soft-dirty pages with anonymous (CoW) pages, ensuring only relevant modified pages are written.
• The clear_soft_dirty() lifecycle (probe → arm → filter → re-arm) ensures consistent delta computation while limiting VM pause time.
• The system gracefully degrades to pagemap_anon tracking when kernel support is unavailable, maintaining functionality across diverse host environments.

Frequently Asked Questions

What happens if the host kernel does not support soft dirty pages?

If CONFIG_MEM_SOFT_DIRTY is disabled, the initial probe in clear_soft_dirty() fails, and the VMM automatically disables the soft-dirty path. It falls back to using only pagemap_anon tracking, which still guarantees correctness but produces larger incremental snapshots because it includes all pages that have ever been touched via Copy-on-Write.

Why does CubeSandbox intersect soft dirty pages with anonymous pages?

The intersection ensures that only pages that are both anonymous (indicating they have diverged from the base image via CoW) and soft-dirty (indicating they were written in the current snapshot window) are included in the delta. This filtering prevents writing shared pages that have not actually changed and maintains snapshot consistency by respecting the memory hierarchy.

How does soft dirty tracking affect VM performance during snapshots?

The primary performance impact occurs during the clear_refs(4) call, which walks all PTEs to clear soft-dirty bits. CubeSandbox minimizes this overhead by invoking the operation only once per snapshot cycle, typically resulting in pause times of a few hundred milliseconds for large VMs. The resulting smaller delta files also reduce I/O bandwidth and storage requirements compared to the anonymous-only approach.

What is the difference between soft dirty and pagemap_anon tracking?

Soft dirty tracking uses kernel PTE bit 55 to identify pages written since the last clear_soft_dirty() call, providing a true delta that resets each cycle. Pagemap_anon tracking identifies all pages that have become anonymous (Copy-on-Write) since restore, which accumulates indefinitely. Soft dirty produces smaller, more precise incremental snapshots, while pagemap_anon provides broader compatibility as a fallback mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}