High Availability and Failover in GreptimeDB: A Technical Deep Dive

GreptimeDB achieves high availability through a region-level HA subsystem that combines Φ-accrual failure detection with automated failover orchestration to maintain service continuity during datanode failures.

The distributed time-series database greptimeteam/greptimedb implements a sophisticated failover architecture centered on regions—logical shards of tables distributed across datanodes. This system automatically detects failures, migrates regions to healthy nodes, and ensures data durability through remote write-ahead logging.

How Region-Level High Availability Works in GreptimeDB

GreptimeDB's HA stack operates through three tightly coupled layers that continuously monitor, detect, and remediate region failures. Each region represents a shard of table data residing on a specific datanode, making region health synonymous with data availability.

The Three-Layer HA Architecture

Layer	Responsibility	Core Implementation
Failure detection	Monitors region liveness through passive heartbeat analysis and active health checks	RegionSupervisor and RegionFailureDetector in src/meta-srv/src/region/supervisor.rs
Failover orchestration	Executes migration by selecting replacement nodes, transitioning region states, and coordinating close/open operations	RegionFailureHandler wired in src/meta-srv/src/metasrv/builder.rs
Configuration & control	Enables/disables HA, selects detection algorithms, and enforces safety guards	MetaSrvOptions fields in config/config.md

Failure Detection Mechanisms

The system employs a two-stage detection process to minimize false positives while ensuring rapid failure recognition.

Passive Detection via Heartbeat Analysis

Every datanode transmits heartbeats to the meta-service every 5 seconds containing RegionStat entries. The RegionSupervisor runs a Φ-accrual failure detector (Akka-style) over these heartbeat timestamps to calculate suspicion levels:

// src/meta-srv/src/metasrv/builder.rs lines 70-78
let failure_detector = RegionFailureDetector::new(options);
let supervisor = RegionSupervisor::new(
    failure_detector,
    heartbeat_receiver,
    region_lease_checker,
);

When heartbeat intervals exceed calculated thresholds, the detector flags the region as suspect but does not yet trigger failover.

Active Health Checks

Upon passive detection of a suspect region, the system initiates active detection by sending a direct health-check RPC to the datanode hosting the flagged region. Only a failed response from this targeted probe confirms the failure and triggers the failover procedure. This two-step approach prevents unnecessary migrations during temporary network partitions.

Region Failover Orchestration

Once a region failure is confirmed, RegionFailureHandler executes a coordinated migration through four distinct stages.

The Four-Stage Failover Flow

1. Candidate selection – The meta-service evaluates datanodes based on load metrics, tenant assignments, and historical CPU/IO usage to select optimal replacement candidates.

2. State transition – The failing region's state transitions to Passive. In this state, frontend nodes block writes while permitting reads to maintain query availability.

3. Close and open RPCs – Metasrv issues a close request to the failed datanode and open requests to candidates. The open operation reconstructs the region from the remote WAL (Bunshin):

// Conceptual flow from src/meta-srv/src/procedure/region_migration.rs
close_region(failed_datanode, region_id).await?;
open_region(candidate_datanode, region_id, remote_wal).await?;

4. Confirmation – Candidates report readiness through their regular heartbeats. Metasrv then transitions the region state to Active, lifting the write block and restoring full service.

State Management During Failover

The orchestration logic resides in RegionFailureHandler (instantiated in builder.rs lines 81-85), which embeds both the RegionSupervisor and HeartbeatAcceptor to react to liveness events in real-time. The handler ensures atomic state transitions and prevents split-brain scenarios during migration.

Safety Guards and Configuration

The HA subsystem includes critical safeguards to prevent data loss during automated failover operations.

Remote WAL Requirements

Region failover requires a remote WAL (such as Bunshin) to ensure data durability during migration. The system enforces this at startup through a hard assertion in the builder:

// src/meta-srv/src/metasrv/builder.rs lines 9-16
ensure!(
    options.allow_region_failover_on_local_wal,
    UnexpectedSnafu {
        msg: "Region failover requires remote WAL. \
              Set allow_region_failover_on_local_wal=true to override (unsafe)."
    }
);

Operating with a local WAL risks data loss during failover, as the failed node's local storage becomes inaccessible.

Configurable Failover Parameters

Administrators control HA behavior through MetaSrvOptions defined in config/config.md (lines 351-353):

Parameter	Description	Default
enable_region_failover	Master switch for the HA subsystem	false
allow_region_failover_on_local_wal	Permits failover with local WAL (unsafe)	false
region_failure_detector_initialization_delay	Grace period before detection starts	10m
phi	Φ-accrual threshold for failure suspicion	(configurable)

Setting allow_region_failover_on_local_wal = true requires explicit opt-in and generates warning logs due to potential data loss risks.

Summary

• GreptimeDB implements region-level high availability through a dedicated subsystem comprising failure detection, failover orchestration, and configuration layers.
• Φ-accrual failure detection combines passive heartbeat analysis with active health checks to minimize false positives while ensuring rapid failure recognition.
• Automated failover follows a four-stage process: candidate selection, passive state transition, coordinated close/open RPCs, and confirmation via heartbeat.
• Remote WAL is mandatory for safe failover; local WAL deployments require explicit unsafe opt-in via allow_region_failover_on_local_wal.
• Key source files include src/meta-srv/src/region/supervisor.rs for detection logic and src/meta-srv/src/metasrv/builder.rs for subsystem initialization.

Frequently Asked Questions

How does GreptimeDB detect datanode failures without generating false positives?

GreptimeDB uses a two-stage detection mechanism. First, the RegionFailureDetector applies Φ-accrual algorithms to regular heartbeats (passive detection). Only after this passive flag does the system send targeted health-check RPCs (active detection). This combination prevents unnecessary failovers during temporary network glitches while maintaining sensitivity to actual hardware failures.

What happens to write operations during a region failover?

When a region enters the Passive state during failover, frontend nodes immediately block write operations to that region while continuing to serve read requests. Writes resume automatically once the region transitions back to Active on the new datanode and confirms readiness via heartbeat. This ensures consistency while minimizing query downtime.

Can I enable region failover if my cluster uses local WAL storage?

Technically yes, by setting allow_region_failover_on_local_wal = true in your meta-service configuration, but this is strongly discouraged. Local WAL storage means region data resides only on the failed node's disk, making recovery impossible if the node is permanently lost. The system logs warnings when this unsafe configuration is active, and data loss is likely during failover scenarios.

Where is the failover orchestration logic implemented in the source code?

The core orchestration resides in src/meta-srv/src/metasrv/builder.rs (lines 68-85), which wires the RegionFailureHandler and RegionSupervisor into the Metasrv startup sequence. The actual migration procedures executing close/open RPCs are implemented in src/meta-srv/src/procedure/region_migration.rs, while failure detection algorithms live in src/meta-srv/src/region/supervisor.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}