How the Mito2 Compaction Scheduler Manages SST Files in GreptimeDB

The Mito2 compaction scheduler orchestrates SST file merges by tracking per-region compaction status, deduplicating concurrent requests, selecting strategies via the picker, and executing tasks locally or remotely while enforcing memory limits.

The Mito2 engine in GreptimeDB stores time-series data in SST (Sorted-String-Table) files that accumulate over time and require periodic merging to maintain query performance and reclaim storage space. The CompactionScheduler in src/mito2/src/compaction.rs serves as the central orchestrator that determines when and how to compact these files, coordinating the entire lifecycle of compaction tasks across the distributed engine.

Core Components of the Compaction Scheduler

The CompactionScheduler Struct

The scheduler maintains the state necessary to coordinate compactions across all regions:

pub(crate) struct CompactionScheduler {
    scheduler: SchedulerRef,
    region_status: HashMap<RegionId, CompactionStatus>,
    request_sender: Sender<WorkerRequestWithTime>,
    cache_manager: CacheManagerRef,
    engine_config: Arc<MitoConfig>,
    memory_manager: Arc<CompactionMemoryManager>,
    memory_policy: OnExhaustedPolicy,
    listener: WorkerListener,
    plugins: Plugins,
}

Source: [src/mito2/src/compaction.rs lines 101-116](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/compaction.rs#L101-L116)

Per-Region Status Tracking

The region_status field stores a HashMap<RegionId, CompactionStatus> that tracks ongoing compactions. This map enables the scheduler to manage concurrent requests, queue pending manual compactions, and maintain waiters for each region's compaction lifecycle.

Scheduling Flow and Request Handling

Entry Points and Triggers

The schedule_compaction method serves as the primary entry point for all compaction requests:

pub(crate) async fn schedule_compaction(
    &mut self,
    region_id: RegionId,
    compact_options: compact_request::Options,
    version_control: &VersionControlRef,
    access_layer: &AccessLayerRef,
    waiter: OptionOutputTx,
    manifest_ctx: &ManifestContextRef,
    schema_metadata_manager: SchemaMetadataManagerRef,
    max_parallelism: usize,
) -> Result<()> { … }

Source: [src/mito2/src/compaction.rs lines 42-54](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/compaction.rs#L42-L54)

This method is invoked from multiple worker handlers throughout the codebase:

• src/mito2/src/worker/handle_flush.rs – triggers compaction after flush completion
• src/mito2/src/worker/handle_compaction.rs – handles manual compaction requests from SQL
• src/mito2/src/worker/handle_truncate.rs and handle_drop.rs – manage region lifecycle events

Example from flush handling:

self.schedule_compaction(&region).await;

Source: [src/mito2/src/worker/handle_flush.rs line 322](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/worker/handle_flush.rs#L322)

Deduplication and Waiter Merging

When schedule_compaction detects an existing entry in region_status, it invokes merge_waiter to deduplicate concurrent requests. Regular compaction requests merge their waiters into the existing status, while manual requests using StrictWindow options are stored as PendingCompaction to execute after the current task completes.

Staging Mode Bypass

The scheduler checks for RegionRoleState::Leader(RegionLeaderState::Staging) and returns early to bypass compaction during region initialization, preventing interference with staging operations.

Strategy Selection and Execution

Picker Integration

The scheduler delegates strategy selection to new_picker:

let picker = new_picker(
    &options,
    &dynamic_compaction_opts,
    request.current_version.options.append_mode,
    Some(self.engine_config.max_background_compactions),
);

Source: [src/mito2/src/compaction.rs lines 29-34](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/compaction.rs#L29-L34)

This function selects between TWCS (Time-Window-Compaction-Strategy), level-based compaction, or TTL-based purge based on region options, schema metadata from schema_metadata_manager, and runtime configuration.

Local vs Remote Execution

The scheduler supports flexible execution modes:

Local execution submits CompactionTaskImpl to the generic SchedulerRef:

self.scheduler
    .schedule(Box::pin(async move {
        INFLIGHT_COMPACTION_COUNT.inc();
        task.run().await;
        INFLIGHT_COMPACTION_COUNT.dec();
    }))

Source: [src/mito2/src/compaction.rs lines 71-77](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/compaction.rs#L71-L77)

Remote execution uses the RemoteJobScheduler plugin when remote_compaction is enabled:

if let Some(remote_job_scheduler) = &self.plugins.get::<RemoteJobSchedulerRef>() {
    // schedule remote job …
}

Source: [src/mito2/src/compaction.rs lines 92-104](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/compaction.rs#L92-L104)

If remote execution fails and fallback_to_local is true, the scheduler automatically reverts to local execution.

Memory Management

The CompactionMemoryManager from src/compaction/memory_manager.rs regulates resource consumption. Each compaction task is wrapped in a CompactionMemoryGuard. When memory limits are reached, the OnExhaustedPolicy determines whether to block the task until resources free up, reject the request, or drop it entirely, preventing OOM errors during large SST merges.

Lifecycle and Completion Handling

Task Completion Callbacks

Upon completion, on_compaction_finished processes the results:

• If a pending manual compaction exists, it is immediately scheduled
• Otherwise, the scheduler attempts to schedule the next regular compaction
• If the picker returns None, the region is removed from region_status

Region Lifecycle Events

The scheduler handles state changes through specific callbacks:

• on_region_dropped – removes status and fails waiters
• on_region_closed – cleans up pending compactions
• on_region_truncated – resets compaction state
• on_compaction_failed – handles task failures and cleanup

Summary

• The CompactionScheduler in src/mito2/src/compaction.rs acts as the central coordinator for all SST file merging operations in the Mito2 engine.
• It maintains per-region status tracking via region_status: HashMap<RegionId, CompactionStatus> to manage concurrent requests and deduplicate work.
• Request deduplication merges waiters for regular compactions and queues manual requests (StrictWindow) as pending tasks.
• Strategy selection delegates to new_picker, choosing between TWCS, level-based, or TTL strategies based on region options and runtime configuration.
• Execution flexibility supports both local execution via CompactionTaskImpl and remote execution via the RemoteJobScheduler plugin, with automatic fallback capabilities.
• Memory safety is enforced through CompactionMemoryManager and OnExhaustedPolicy to prevent OOM during large merges.
• Lifecycle management ensures proper cleanup through callbacks like on_compaction_finished, on_region_dropped, and on_region_closed.

Frequently Asked Questions

What triggers a compaction in Mito2?

Compactions are triggered automatically after flush operations complete in src/mito2/src/worker/handle_flush.rs, through manual SQL commands like ALTER TABLE ... COMPACT handled in src/mito2/src/worker/handle_compaction.rs, or via periodic background checks that evaluate SST file accumulation against configured thresholds.

How does the scheduler handle concurrent compaction requests?

The scheduler uses a deduplication mechanism via the region_status HashMap. When schedule_compaction is called for a region already being compacted, it invokes merge_waiter to combine the new request with the existing task. Regular requests merge their waiters, while manual requests (StrictWindow) are stored as pending compactions to execute after the current task finishes.

What is the difference between local and remote compaction execution?

Local execution submits CompactionTaskImpl directly to the internal SchedulerRef async runtime, running the merge process on the same node. Remote execution delegates the task to a RemoteJobScheduler plugin when remote_compaction is enabled in region options, allowing distributed processing. If remote scheduling fails and fallback_to_local is true, the scheduler automatically reverts to local execution.

How does memory management prevent OOM during compaction?

The scheduler integrates CompactionMemoryManager from src/compaction/memory_manager.rs to track resource usage. Each compaction task is wrapped in a CompactionMemoryGuard. When memory limits are reached, the OnExhaustedPolicy determines whether to block the task until resources free up, reject the request, or drop it entirely, ensuring the system remains stable during large SST merges.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}