# How Soft Dirty Pages Enable Efficient Incremental Snapshots in CubeSandbox

> Discover how CubeSandbox's soft dirty pages optimize incremental snapshots by tracking memory modifications and creating efficient deltas for enhanced performance.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-16

---

**CubeSandbox uses soft dirty pages to track per-snapshot memory modifications, generating minimal incremental deltas by intersecting write-tracked pages with copy-on-write anonymous pages.**

CubeSandbox, TencentCloud's hypervisor, implements a sophisticated memory tracking system that leverages **soft dirty pages** to optimize incremental snapshot creation. This mechanism provides fine-grained visibility into guest memory modifications since the last snapshot, working alongside copy-on-write tracking to ensure consistency while minimizing delta size. When the host kernel supports `CONFIG_MEM_SOFT_DIRTY`, the VMM can track true per-snapshot deltas rather than accumulating all pages touched since boot.

## Understanding Soft Dirty Pages vs. Anonymous Page Tracking

CubeSandbox employs two complementary mechanisms to identify which guest memory pages to write into an incremental snapshot:

| Mechanism | What It Tracks | Usage |
|-----------|----------------|-------|
| **pagemap_anon** | All pages that have become anonymous (Copy-on-Write) after restore. These pages may differ from the base image and must be examined for every snapshot. | Always available; works on any kernel. |
| **soft dirty** | Only pages written since the last `clear_soft_dirty()` call. The kernel clears bit 55 in `/proc/self/pagemap` on `clear_refs(4)`, then sets it on subsequent writes. | Requires `CONFIG_MEM_SOFT_DIRTY=y`. Provides a true delta that resets each snapshot cycle. |

The **soft dirty** mechanism tracks modifications at the PTE (Page Table Entry) level using bit 55 of the kernel's pagemap interface. Unlike the anonymous page tracker, which accumulates all CoW pages indefinitely, soft dirty pages provide a clean slate after each snapshot window.

## The Soft Dirty Page Lifecycle in CubeSandbox

The implementation in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) and [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) follows a strict lifecycle to ensure snapshot consistency.

### 1. Probing and Arming the Tracker

During the first snapshot (base image creation), the VMM probes kernel support and initializes the tracking window. The `clear_soft_dirty()` function writes to `clear_refs(4)`, which both tests for `CONFIG_MEM_SOFT_DIRTY` support and resets every page's soft-dirty bit.

```rust
// Probe kernel for soft‑dirty support; this also clears the bitmap.
if probe_soft_dirty_support() {
    // Tracker is now armed for the next window.
    self.soft_dirty_armed.store(true, Ordering::Release);
}

```

*Source:* `send_soft_dirty_memory` first-time path in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) (lines 21‑24).

### 2. Capturing the Delta Bitmap

Before writing subsequent incremental snapshots, the VMM reads the soft-dirty bitmap for each memory region using `get_soft_dirty_pages`. This function examines bit 55 in the pagemap to identify pages modified since the previous snapshot.

```rust
let dirty_pages = get_soft_dirty_pages(host_addr as u64, length)?;

```

*Source:* `get_soft_dirty_pages` implementation in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) (lines 76‑48).

### 3. Intersecting with Anonymous Pages

To ensure correctness, the VMM calls `filter_memory_ranges_by_anon_and_soft_dirty`, which performs a bitwise intersection between the anonymous page set (CoW pages) and the soft-dirty set. Only pages that are both **anonymous** and **soft-dirty** are selected for the delta.

```rust
let anon_pages = get_anon_pages(host_addr as u64, length)?;
let must_save: Vec<bool> = anon_pages.iter()
    .zip(dirty_pages.iter())
    .map(|(&a, &d)| a && d)
    .collect();

```

*Source:* Intersection logic in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) (lines 84‑88).

### 4. Writing the Delta and Re-Arming

The filtered pages overlay onto the existing base snapshot file. After writing the delta, the VMM must re-arm the tracker by calling `clear_soft_dirty()` again. If this call fails, the system silently disables soft-dirty tracking and falls back to the `pagemap_anon` method.

```rust
for range in filtered_ranges.regions() {
    let file_off = Self::calculate_file_offset_for_gpa(range.gpa, range.length, &gpa_to_file_offset)?;
    self.save_range_to_file(&memory_file, range, file_off)?;
}

// Re-arm for next window
if let Err(e) = clear_soft_dirty() {
    self.soft_dirty_armed.store(false, Ordering::Release);
}

```

*Source:* Overlay loop and re-arm handling in [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) (lines 86‑90 and 98‑107).

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) | Implements probing, clearing (`clear_soft_dirty`), bitmap extraction (`get_soft_dirty_pages`), and filtering logic. |
| [`hypervisor/vmm/src/memory_manager.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/memory_manager.rs) | Coordinates snapshot creation, decides whether to use the soft-dirty path, and performs delta overlays. |
| [`hypervisor/vmm/src/pagemap_anon.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/pagemap_anon.rs) | Provides CoW-only filtering used as fallback and for intersection with soft-dirty sets. |

## Performance and Consistency Benefits

**True Incremental Snapshots**: Because `clear_soft_dirty()` resets the dirty bits after each snapshot, the VMM records only pages that changed in the current window. This yields significantly smaller deltas than the `pagemap_anon` approach, which accumulates all CoW pages ever touched. The end-to-end test `test_filter_memory_ranges_by_soft_dirty_end_to_end` in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) (lines 31‑41) verifies that only newly written pages appear in subsequent windows.

**Controlled Pause Time**: The primary cost is the kernel PTE walk during `clear_refs(4)`. By invoking this only once per snapshot cycle, CubeSandbox limits pause time to a few hundred milliseconds even for multi-gigabyte VMs. The implementation logs these durations in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) (lines 66‑71) for operational correlation.

**Graceful Degradation**: If the kernel lacks `CONFIG_MEM_SOFT_DIRTY`, the VMM automatically falls back to anonymous page tracking without error, ensuring compatibility across all Linux kernels. This probe and fallback logic resides in [`hypervisor/vmm/src/soft_dirty.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/vmm/src/soft_dirty.rs) (lines 21‑26).

## Summary

- **Soft dirty pages** provide per-snapshot, fine-grained tracking of guest memory modifications via bit 55 in the kernel pagemap.
- CubeSandbox generates minimal deltas by **intersecting** soft-dirty pages with anonymous (CoW) pages, ensuring only relevant modified pages are written.
- The **clear_soft_dirty()** lifecycle (probe → arm → filter → re-arm) ensures consistent delta computation while limiting VM pause time.
- The system **gracefully degrades** to `pagemap_anon` tracking when kernel support is unavailable, maintaining functionality across diverse host environments.

## Frequently Asked Questions

### What happens if the host kernel does not support soft dirty pages?

If `CONFIG_MEM_SOFT_DIRTY` is disabled, the initial probe in `clear_soft_dirty()` fails, and the VMM automatically disables the soft-dirty path. It falls back to using only `pagemap_anon` tracking, which still guarantees correctness but produces larger incremental snapshots because it includes all pages that have ever been touched via Copy-on-Write.

### Why does CubeSandbox intersect soft dirty pages with anonymous pages?

The intersection ensures that only pages that are both **anonymous** (indicating they have diverged from the base image via CoW) and **soft-dirty** (indicating they were written in the current snapshot window) are included in the delta. This filtering prevents writing shared pages that have not actually changed and maintains snapshot consistency by respecting the memory hierarchy.

### How does soft dirty tracking affect VM performance during snapshots?

The primary performance impact occurs during the `clear_refs(4)` call, which walks all PTEs to clear soft-dirty bits. CubeSandbox minimizes this overhead by invoking the operation only once per snapshot cycle, typically resulting in pause times of a few hundred milliseconds for large VMs. The resulting smaller delta files also reduce I/O bandwidth and storage requirements compared to the anonymous-only approach.

### What is the difference between soft dirty and pagemap_anon tracking?

**Soft dirty** tracking uses kernel PTE bit 55 to identify pages written since the last `clear_soft_dirty()` call, providing a true delta that resets each cycle. **Pagemap_anon** tracking identifies all pages that have become anonymous (Copy-on-Write) since restore, which accumulates indefinitely. Soft dirty produces smaller, more precise incremental snapshots, while pagemap_anon provides broader compatibility as a fallback mechanism.