GreptimeDB Storage Engine Configuration Options for Compaction, Flush, and GC

GreptimeDB exposes TOML configuration keys in config/datanode.example.toml and config/metasrv.example.toml to govern background flush workers, compaction scheduling, and garbage collection throttling in the Mito storage engine.

GreptimeDB relies on the Mito region engine to manage time-series data persistence, with background processes that require careful tuning. Understanding the configuration options that control compaction, flush, and GC in the storage engine is critical for optimizing write throughput and disk utilization in production deployments. These TOML-based settings, parsed at startup from the datanode and meta-service configuration files, define worker pool sizes, memory limits, and deletion safety buffers as implemented in the greptimeteam/greptimedb source code.

Flush Configuration Options

Flush operations persist in-memory memtables to immutable SST files on disk. In config/datanode.example.toml, two keys control this behavior:

• max_background_flushes – Defines the size of the flush worker pool. This bounds the number of concurrent flush tasks that can write to disk simultaneously.
• auto_flush_interval – Specifies a duration (e.g., "15m") after which idle memtables are forced to disk, preventing data loss during low-write periods even if size thresholds are not met.

These parameters are consumed by src/region_engine/mito/flush.rs, which spawns the async workers that execute the flush tasks according to these limits.

Compaction Configuration Options

Compaction merges overlapping SST files to reclaim space and improve read performance. The datanode configuration file exposes four critical knobs that control this process:

• max_background_compactions – Sets the maximum number of concurrent compaction threads. Increasing this value accelerates merge operations on fast NVMe storage but raises CPU and I/O pressure.
• min_compaction_interval – Enforces a minimum delay (e.g., "5m") between compaction runs on the same region. The default is "0m", which allows aggressive compaction; setting a non-zero duration prevents thrashing on high-ingestion workloads.
• experimental_compaction_memory_limit – Caps memory allocation for compaction workers (e.g., "2GB"). A value of 0 or "unlimited" removes the restriction.
• experimental_compaction_on_exhausted – Determines behavior when the memory cap is reached: "wait" pauses the worker (default), while "fail" aborts the operation.

The compaction scheduler in src/region_engine/mito/compact.rs reads these values to throttle merge tasks and manage memory budgets.

Garbage Collection Configuration Options

GC removes unreferenced files after a safety period. This subsystem requires coordinated settings across both the datanode and meta-service to function correctly.

Datanode settings (in the region_engine.mito.gc section of config/datanode.example.toml):

• enable – Master toggle for the GC worker. Must be true for file cleanup to occur.
• lingering_time – Duration (e.g., "2m") to retain files after their last reference is dropped, allowing in-flight queries to complete before deletion.
• unknown_file_lingering_time – Safety buffer (e.g., "30m") for files whose expiration cannot be determined, such as those created during a crash recovery scenario.

Meta-service settings (in config/metasrv.example.toml):

• gc.enable – Global flag that must match the datanode's enable setting; otherwise GC is disabled silently.
• gc_cooldown_period – Minimum interval between GC cycles per region (e.g., "5m"), preventing metadata thrashing.

The GC worker implementation in src/region_engine/mito/gc.rs coordinates with the meta-service to respect these lingering periods and cooldowns.

Interaction Between Subsystems

These three subsystems operate in parallel but influence each other's resource consumption and scheduling.

Flush workers write memtables to SSTs, which increases the file count that compaction must later merge. Setting max_background_flushes and max_background_compactions too high can saturate disk bandwidth, while setting them too low creates backpressure during write spikes and allows SST file counts to grow unchecked.

The experimental_compaction_memory_limit acts as a circuit breaker during large merge operations. When compaction workers approach this limit, the experimental_compaction_on_exhausted policy determines whether the system pauses processing ("wait") or fails the operation ("fail"), protecting against OOM errors.

For GC to function, the region_engine.mito.gc.enable flag in the datanode must align with the meta-service's gc.enable. The lingering_time and gc_cooldown_period work together to ensure files remain accessible long enough for queries to finish while eventually being purged safely.

Configuration Examples

Below is a production-ready datanode configuration excerpt that enables aggressive background processing:

# config/datanode.example.toml

max_background_flushes = 4
auto_flush_interval = "15m"

max_background_compactions = 3
min_compaction_interval = "5m"
experimental_compaction_memory_limit = "2GB"
experimental_compaction_on_exhausted = "wait"

[region_engine.mito.gc]
enable = true
lingering_time = "2m"
unknown_file_lingering_time = "30m"

And the corresponding meta-service configuration:

# config/metasrv.example.toml

[gc]
enable = true
gc_cooldown_period = "5m"

Key Implementation Files

The configuration values above are parsed and enforced in the following Rust source files:

• src/region_engine/mito/mod.rs – Entry point that reads TOML sections and initializes the engine with the specified worker pools.
• src/region_engine/mito/flush.rs – Implements flush logic using max_background_flushes and auto_flush_interval.
• src/region_engine/mito/compact.rs – Contains the compaction scheduler that respects memory limits and throttling intervals.
• src/region_engine/mito/gc.rs – Houses the GC worker that processes lingering_time and coordinates with the meta-service.

Summary

• Flush settings (max_background_flushes, auto_flush_interval) control how quickly in-memory data becomes persistent SST files and how many concurrent flush workers are active.
• Compaction settings (max_background_compactions, min_compaction_interval, experimental_compaction_memory_limit) manage merge throughput, prevent thrashing through minimum intervals, and enforce memory safety via limits and exhaustion policies.
• GC settings require matching enable flags in both datanode (region_engine.mito.gc) and meta-service (gc) configurations, plus timing controls (lingering_time, unknown_file_lingering_time, gc_cooldown_period) to prevent premature deletion and metadata thrashing.
• All parameters are defined in config/datanode.example.toml and config/metasrv.example.toml, with implementations located in the src/region_engine/mito/ directory.

Frequently Asked Questions

What happens if the datanode and meta-service GC settings are mismatched?

If region_engine.mito.gc.enable in the datanode is true but gc.enable in the meta-service is false, garbage collection is disabled silently. Both flags must be explicitly set to true for the GC worker to delete unreferenced files.

How do I prevent compaction from consuming all available memory?

Set experimental_compaction_memory_limit to a value slightly below your system's available RAM (e.g., "4GB"). If workers exceed this budget, the experimental_compaction_on_exhausted setting (default "wait") pauses compaction until memory is freed rather than aborting the job or triggering an OOM kill.

What is the difference between lingering_time and unknown_file_lingering_time?

lingering_time applies to files whose expiration time is known precisely, keeping them for a short buffer (e.g., 2 minutes) after dereferencing to allow active queries to complete. unknown_file_lingering_time applies to files in an ambiguous state (e.g., after a crash), using a longer default (e.g., 30 minutes) to ensure safety before deletion.

Can I disable automatic flushing and trigger flushes manually?

While auto_flush_interval can be set to a very large value to minimize automatic flushes, the max_background_flushes pool must remain active to handle memtable size limits. Manual flush APIs exist in the engine, but disabling background flushes entirely risks unbounded memory growth and potential OOM errors when memtables fill up.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}