How Garbage Collection Works in GreptimeDB to Clean Up Obsolete Data

GreptimeDB uses a coordinated two-phase garbage collection pipeline where the meta-service schedules cleanup tasks and datanodes execute safe deletion of obsolete SST and index files using manifest comparisons and configurable lingering periods.

The greptimeteam/greptimedb storage engine accumulates immutable SST files and index files during normal operations. To reclaim disk space without compromising data integrity, the system implements a distributed GC pipeline that coordinates between the meta-service scheduler and per-datanode workers. This architecture ensures that files still referenced by active queries or temporary operations are never deleted, even during region migrations or splits.

The Two-Phase GC Architecture

Garbage collection in GreptimeDB operates as a distributed transaction across the cluster. The meta-service acts as the coordinator, selecting candidate regions and building a consistent view of file references, while each datanode runs a LocalGcWorker that performs the actual deletion safely. This separation prevents race conditions between compaction, queries, and cleanup operations.

Phase 1: Meta-Service Scheduling

GcScheduler and Periodic Ticks

The GC orchestration begins in src/meta-srv/src/gc/scheduler.rs with the GcScheduler struct. A background ticker fires at TICKER_INTERVAL = 5 min (defined in src/meta-srv/src/gc/options.rs) to evaluate tables for cleanup. The scheduler selects regions based on criteria such as SST file count and total size, then creates a BatchGcProcedure to manage the distributed state machine.

// From src/meta-srv/src/gc/scheduler.rs
pub struct GcScheduler {
    ticker: GcTicker,
    options: GcSchedulerOptions,
}

GC runs only when GcSchedulerOptions.enable is true. The scheduler respects cooldown periods and concurrency limits to avoid overwhelming the cluster.

BatchGcProcedure State Machine

The BatchGcProcedure (defined in src/meta-srv/src/gc/procedure.rs) executes a four-step state machine:

1. Discovery: Locates region routes across the cluster.
2. Reference Gathering: Fetches FileRefsManifest from all datanodes to determine which files are currently in use by running queries.
3. Instruction Broadcast: Sends GcRegions instructions (defined in src/common_meta/src/instruction.rs) to each datanode containing the region list, consolidated file references, and whether to perform a full object-store listing.
4. Finalization: Merges per-datanode GcReport responses, updates region-repartition metadata, and records completion metrics.

Manual GC Triggering

Administrators can bypass the periodic ticker using the SQL command:

ADMIN GC;

This sends an Event::Manually through the same ticker channel (scheduler.rs lines 52-55), immediately spawning a BatchGcProcedure without waiting for the next 5-minute interval.

Phase 2: Datanode Execution

GcRegionsHandler Reception

When a datanode receives the GcRegions instruction via heartbeat, the GcRegionsHandler in src/datanode/src/heartbeat/handler/gc_worker.rs registers a GC task and instantiates a LocalGcWorker from src/mito2/src/gc.rs. This worker executes the actual deletion logic while respecting safety constraints.

LocalGcWorker Decision Logic

The LocalGcWorker determines file eligibility through the should_delete_file predicate (lines 60-78 of gc.rs). A file is deletable only when all the following conditions hold:

• Not in manifest: The file does not appear in the current region manifest's live file set.
• Not referenced: The file is absent from the temporary FileRefsManifest sent by the meta-service (indicating no active queries use it).
• Lingering period elapsed: For files with known expulsion timestamps, the lingering_time (default 60 seconds) must pass. For files with unknown expulsion times, the unknown_file_lingering_time (default 1 hour) applies.

// Conceptual logic from src/mito2/src/gc.rs
fn should_delete_file(&self, file: &RemovedFile) -> bool {
    !in_manifest && !in_tmp_ref && (is_linger_elapsed || is_eligible_for_delete)
}

Fast Mode and Full-Listing Mode

The worker operates in two distinct modes controlled by the full_file_listing flag:

• Fast mode (full_file_listing == false): Examines only the removed_files list from the manifest delta. This avoids expensive object-store API calls and handles the common case of post-compaction cleanup.
• Full-listing mode (full_file_listing == true): Performs a complete directory scan via list_from_object_store to identify orphan files—data that exists in storage but is unreferenced by any manifest. This mode runs periodically based on full_file_listing_interval (typically 24 hours).

Safe Deletion and Manifest Cleanup

Once the worker builds a Vec<RemovedFile> via list_to_be_deleted_files, it executes deletions through the store API:

// From src/mito2/src/gc.rs
self.delete_files(region_id, &deletable_files).await?;
self.update_manifest_removed_files(region, deletable_files.clone()).await?;

The delete_files helper removes both Parquet SST files and associated index files. After successful object-store deletion, the worker calls clear_deleted_files on the region manifest to prune the removed_files list, preventing duplicate deletion attempts.

Concurrency and Safety Limits

To prevent GC from saturating datanode resources, src/mito2/src/gc.rs implements two control mechanisms:

• Job Limiter (GcLimiter): Caps concurrent GC jobs per datanode at 4 by default.
• Lister Concurrency (max_concurrent_lister_per_gc_job): Limits parallel directory listings to 32 threads during full-listing scans.

These values are configurable via the GcConfig struct (lines 44-50 of gc.rs):

pub struct GcConfig {
    pub enable: bool,
    pub lingering_time: Duration,
    pub unknown_file_lingering_time: Duration,
    pub max_concurrent_gc_jobs: usize, // default 4
    pub max_concurrent_lister_per_gc_job: usize, // default 32
}

Configuration and Observability

TOML Configuration Options

Enable and tune GC behavior in greptime.toml using the following fragments:

[mito]
gc = { 
    enable = true, 
    lingering_time = "60s", 
    unknown_file_lingering_time = "1h",
    max_concurrent_gc_jobs = 4,
    max_concurrent_lister_per_gc_job = 32
}

[meta]
gc = { 
    enable = true, 
    full_file_listing_interval = "24h" 
}

Sources: GcConfig defaults (gc.rs lines 30-64) and GcSchedulerOptions (options.rs lines 72-90).

Prometheus Metrics

The GC worker exposes detailed metrics defined in src/mito2/src/metrics.rs:

• gc_runs_total{mode="fast"} or gc_runs_total{mode="full_listing"} – Total GC execution count by mode.
• gc_files_deleted_total{type="parquet"} – Count of successfully removed Parquet files.
• gc_errors_total – Counter of failures during manifest reading, listing, or deletion.

These counters are incremented by LocalGcWorker during the cleanup cycle and can be scraped for alerting on failed GC runs or orphan file accumulation.

Summary

• Meta-service coordination: The GcScheduler drives cleanup every 5 minutes via BatchGcProcedure, gathering consistent file references before issuing GcRegions commands.
• Datanode safety: LocalGcWorker in src/mito2/src/gc.rs applies multi-layer checks (manifest absence, temporary reference clearance, lingering periods) before deleting any file.
• Dual-mode operation: Fast mode handles routine post-compaction cleanup efficiently, while periodic full-listing mode scavenges orphan files from the object store.
• Resource protection: Concurrency limits (GcLimiter, lister caps) and configurable timeouts prevent GC from impacting query performance.
• Administrative control: Use ADMIN GC; for manual triggers and Prometheus metrics for operational visibility.

Frequently Asked Questions

What prevents GreptimeDB GC from deleting files that are still being queried?

The LocalGcWorker receives a FileRefsManifest from the meta-service containing temporary references held by running queries. The should_delete_file function explicitly checks !in_tmp_ref, ensuring any file referenced by an active scan is retained regardless of its manifest status or age.

How can I manually trigger garbage collection outside the 5-minute ticker interval?

Execute the SQL command ADMIN GC; against the meta-service. This injects an Event::Manually into the GcTicker channel (src/meta-srv/src/gc/scheduler.rs), immediately spawning a BatchGcProcedure without waiting for the next periodic tick.

What is the difference between fast mode and full-listing mode in GreptimeDB garbage collection?

Fast mode examines only the removed_files list from the region manifest delta, avoiding object-store API costs and suitable for routine cleanup. Full-listing mode performs a complete directory scan via list_from_object_store to detect and remove orphan files that exist in storage but are unreferenced by any manifest, triggered by full_file_listing_interval (default 24 hours).

What are the default safety delays before GreptimeDB deletes an obsolete file?

Files known to be removed from the manifest require a 60-second lingering_time before deletion. Files with unknown expulsion timestamps (rare edge cases) use unknown_file_lingering_time of 1 hour. These delays provide a buffer against clock skew and temporary reference propagation delays.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}