# How Garbage Collection Works in GreptimeDB to Clean Up Obsolete Data

> Discover how GreptimeDB garbage collection cleans obsolete data. Learn about its two-phase pipeline, meta-service scheduling, datanode execution, and safe file deletion strategies.

- Repository: [Greptime/greptimedb](https://github.com/greptimeteam/greptimedb)
- Tags: internals
- Published: 2026-03-02

---

**GreptimeDB uses a coordinated two-phase garbage collection pipeline where the meta-service schedules cleanup tasks and datanodes execute safe deletion of obsolete SST and index files using manifest comparisons and configurable lingering periods.**

The `greptimeteam/greptimedb` storage engine accumulates immutable SST files and index files during normal operations. To reclaim disk space without compromising data integrity, the system implements a distributed **GC pipeline** that coordinates between the meta-service scheduler and per-datanode workers. This architecture ensures that files still referenced by active queries or temporary operations are never deleted, even during region migrations or splits.

## The Two-Phase GC Architecture

Garbage collection in GreptimeDB operates as a distributed transaction across the cluster. The **meta-service** acts as the coordinator, selecting candidate regions and building a consistent view of file references, while each **datanode** runs a `LocalGcWorker` that performs the actual deletion safely. This separation prevents race conditions between compaction, queries, and cleanup operations.

## Phase 1: Meta-Service Scheduling

### GcScheduler and Periodic Ticks

The GC orchestration begins in [`src/meta-srv/src/gc/scheduler.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/gc/scheduler.rs) with the `GcScheduler` struct. A background ticker fires at `TICKER_INTERVAL = 5 min` (defined in [`src/meta-srv/src/gc/options.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/gc/options.rs)) to evaluate tables for cleanup. The scheduler selects regions based on criteria such as SST file count and total size, then creates a `BatchGcProcedure` to manage the distributed state machine.

```rust
// From src/meta-srv/src/gc/scheduler.rs
pub struct GcScheduler {
    ticker: GcTicker,
    options: GcSchedulerOptions,
}

```

GC runs only when `GcSchedulerOptions.enable` is `true`. The scheduler respects cooldown periods and concurrency limits to avoid overwhelming the cluster.

### BatchGcProcedure State Machine

The `BatchGcProcedure` (defined in [`src/meta-srv/src/gc/procedure.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/gc/procedure.rs)) executes a four-step state machine:

1. **Discovery**: Locates region routes across the cluster.
2. **Reference Gathering**: Fetches `FileRefsManifest` from all datanodes to determine which files are currently in use by running queries.
3. **Instruction Broadcast**: Sends `GcRegions` instructions (defined in [`src/common_meta/src/instruction.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/common_meta/src/instruction.rs)) to each datanode containing the region list, consolidated file references, and whether to perform a full object-store listing.
4. **Finalization**: Merges per-datanode `GcReport` responses, updates region-repartition metadata, and records completion metrics.

### Manual GC Triggering

Administrators can bypass the periodic ticker using the SQL command:

```sql
ADMIN GC;

```

This sends an `Event::Manually` through the same ticker channel ([`scheduler.rs`](https://github.com/greptimeteam/greptimedb/blob/main/scheduler.rs) lines 52-55), immediately spawning a `BatchGcProcedure` without waiting for the next 5-minute interval.

## Phase 2: Datanode Execution

### GcRegionsHandler Reception

When a datanode receives the `GcRegions` instruction via heartbeat, the `GcRegionsHandler` in [`src/datanode/src/heartbeat/handler/gc_worker.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/datanode/src/heartbeat/handler/gc_worker.rs) registers a GC task and instantiates a `LocalGcWorker` from [`src/mito2/src/gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/gc.rs). This worker executes the actual deletion logic while respecting safety constraints.

### LocalGcWorker Decision Logic

The `LocalGcWorker` determines file eligibility through the `should_delete_file` predicate (lines 60-78 of [`gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/gc.rs)). A file is deletable only when all the following conditions hold:

- **Not in manifest**: The file does not appear in the current region manifest's live file set.
- **Not referenced**: The file is absent from the temporary `FileRefsManifest` sent by the meta-service (indicating no active queries use it).
- **Lingering period elapsed**: For files with known expulsion timestamps, the `lingering_time` (default **60 seconds**) must pass. For files with unknown expulsion times, the `unknown_file_lingering_time` (default **1 hour**) applies.

```rust
// Conceptual logic from src/mito2/src/gc.rs
fn should_delete_file(&self, file: &RemovedFile) -> bool {
    !in_manifest && !in_tmp_ref && (is_linger_elapsed || is_eligible_for_delete)
}

```

### Fast Mode and Full-Listing Mode

The worker operates in two distinct modes controlled by the `full_file_listing` flag:

- **Fast mode** (`full_file_listing == false`): Examines only the `removed_files` list from the manifest delta. This avoids expensive object-store API calls and handles the common case of post-compaction cleanup.
- **Full-listing mode** (`full_file_listing == true`): Performs a complete directory scan via `list_from_object_store` to identify **orphan files**—data that exists in storage but is unreferenced by any manifest. This mode runs periodically based on `full_file_listing_interval` (typically 24 hours).

### Safe Deletion and Manifest Cleanup

Once the worker builds a `Vec<RemovedFile>` via `list_to_be_deleted_files`, it executes deletions through the store API:

```rust
// From src/mito2/src/gc.rs
self.delete_files(region_id, &deletable_files).await?;
self.update_manifest_removed_files(region, deletable_files.clone()).await?;

```

The `delete_files` helper removes both Parquet SST files and associated index files. After successful object-store deletion, the worker calls `clear_deleted_files` on the region manifest to prune the `removed_files` list, preventing duplicate deletion attempts.

### Concurrency and Safety Limits

To prevent GC from saturating datanode resources, [`src/mito2/src/gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/gc.rs) implements two control mechanisms:

- **Job Limiter** (`GcLimiter`): Caps concurrent GC jobs per datanode at **4** by default.
- **Lister Concurrency** (`max_concurrent_lister_per_gc_job`): Limits parallel directory listings to **32** threads during full-listing scans.

These values are configurable via the `GcConfig` struct (lines 44-50 of [`gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/gc.rs)):

```rust
pub struct GcConfig {
    pub enable: bool,
    pub lingering_time: Duration,
    pub unknown_file_lingering_time: Duration,
    pub max_concurrent_gc_jobs: usize, // default 4
    pub max_concurrent_lister_per_gc_job: usize, // default 32
}

```

## Configuration and Observability

### TOML Configuration Options

Enable and tune GC behavior in [`greptime.toml`](https://github.com/greptimeteam/greptimedb/blob/main/greptime.toml) using the following fragments:

```toml
[mito]
gc = { 
    enable = true, 
    lingering_time = "60s", 
    unknown_file_lingering_time = "1h",
    max_concurrent_gc_jobs = 4,
    max_concurrent_lister_per_gc_job = 32
}

[meta]
gc = { 
    enable = true, 
    full_file_listing_interval = "24h" 
}

```

Sources: `GcConfig` defaults ([`gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/gc.rs) lines 30-64) and `GcSchedulerOptions` ([`options.rs`](https://github.com/greptimeteam/greptimedb/blob/main/options.rs) lines 72-90).

### Prometheus Metrics

The GC worker exposes detailed metrics defined in [`src/mito2/src/metrics.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/metrics.rs):

- `gc_runs_total{mode="fast"}` or `gc_runs_total{mode="full_listing"}` – Total GC execution count by mode.
- `gc_files_deleted_total{type="parquet"}` – Count of successfully removed Parquet files.
- `gc_errors_total` – Counter of failures during manifest reading, listing, or deletion.

These counters are incremented by `LocalGcWorker` during the cleanup cycle and can be scraped for alerting on failed GC runs or orphan file accumulation.

## Summary

- **Meta-service coordination**: The `GcScheduler` drives cleanup every 5 minutes via `BatchGcProcedure`, gathering consistent file references before issuing `GcRegions` commands.
- **Datanode safety**: `LocalGcWorker` in [`src/mito2/src/gc.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/mito2/src/gc.rs) applies multi-layer checks (manifest absence, temporary reference clearance, lingering periods) before deleting any file.
- **Dual-mode operation**: Fast mode handles routine post-compaction cleanup efficiently, while periodic full-listing mode scavenges orphan files from the object store.
- **Resource protection**: Concurrency limits (`GcLimiter`, lister caps) and configurable timeouts prevent GC from impacting query performance.
- **Administrative control**: Use `ADMIN GC;` for manual triggers and Prometheus metrics for operational visibility.

## Frequently Asked Questions

### What prevents GreptimeDB GC from deleting files that are still being queried?

The `LocalGcWorker` receives a `FileRefsManifest` from the meta-service containing temporary references held by running queries. The `should_delete_file` function explicitly checks `!in_tmp_ref`, ensuring any file referenced by an active scan is retained regardless of its manifest status or age.

### How can I manually trigger garbage collection outside the 5-minute ticker interval?

Execute the SQL command `ADMIN GC;` against the meta-service. This injects an `Event::Manually` into the `GcTicker` channel ([`src/meta-srv/src/gc/scheduler.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/gc/scheduler.rs)), immediately spawning a `BatchGcProcedure` without waiting for the next periodic tick.

### What is the difference between fast mode and full-listing mode in GreptimeDB garbage collection?

**Fast mode** examines only the `removed_files` list from the region manifest delta, avoiding object-store API costs and suitable for routine cleanup. **Full-listing mode** performs a complete directory scan via `list_from_object_store` to detect and remove **orphan files** that exist in storage but are unreferenced by any manifest, triggered by `full_file_listing_interval` (default 24 hours).

### What are the default safety delays before GreptimeDB deletes an obsolete file?

Files known to be removed from the manifest require a **60-second** `lingering_time` before deletion. Files with unknown expulsion timestamps (rare edge cases) use `unknown_file_lingering_time` of **1 hour**. These delays provide a buffer against clock skew and temporary reference propagation delays.