# High Availability and Failover in GreptimeDB: A Technical Deep Dive

> Explore GreptimeDBs high availability and failover. Discover how its region-level HA subsystem ensures service continuity with advanced failure detection and automated orchestration.

- Repository: [Greptime/greptimedb](https://github.com/greptimeteam/greptimedb)
- Tags: deep-dive
- Published: 2026-03-02

---

**GreptimeDB achieves high availability through a region-level HA subsystem that combines Φ-accrual failure detection with automated failover orchestration to maintain service continuity during datanode failures.**

The distributed time-series database **greptimeteam/greptimedb** implements a sophisticated failover architecture centered on *regions*—logical shards of tables distributed across datanodes. This system automatically detects failures, migrates regions to healthy nodes, and ensures data durability through remote write-ahead logging.

## How Region-Level High Availability Works in GreptimeDB

GreptimeDB's HA stack operates through three tightly coupled layers that continuously monitor, detect, and remediate region failures. Each *region* represents a shard of table data residing on a specific datanode, making region health synonymous with data availability.

### The Three-Layer HA Architecture

| Layer | Responsibility | Core Implementation |
|-------|----------------|----------------------|
| **Failure detection** | Monitors region liveness through passive heartbeat analysis and active health checks | `RegionSupervisor` and `RegionFailureDetector` in [`src/meta-srv/src/region/supervisor.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/region/supervisor.rs) |
| **Failover orchestration** | Executes migration by selecting replacement nodes, transitioning region states, and coordinating close/open operations | `RegionFailureHandler` wired in [`src/meta-srv/src/metasrv/builder.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/metasrv/builder.rs) |
| **Configuration & control** | Enables/disables HA, selects detection algorithms, and enforces safety guards | `MetaSrvOptions` fields in [`config/config.md`](https://github.com/greptimeteam/greptimedb/blob/main/config/config.md) |

## Failure Detection Mechanisms

The system employs a two-stage detection process to minimize false positives while ensuring rapid failure recognition.

### Passive Detection via Heartbeat Analysis

Every datanode transmits heartbeats to the meta-service every 5 seconds containing `RegionStat` entries. The `RegionSupervisor` runs a **Φ-accrual failure detector** (Akka-style) over these heartbeat timestamps to calculate suspicion levels:

```rust
// src/meta-srv/src/metasrv/builder.rs lines 70-78
let failure_detector = RegionFailureDetector::new(options);
let supervisor = RegionSupervisor::new(
    failure_detector,
    heartbeat_receiver,
    region_lease_checker,
);

```

When heartbeat intervals exceed calculated thresholds, the detector flags the region as suspect but does not yet trigger failover.

### Active Health Checks

Upon passive detection of a suspect region, the system initiates *active detection* by sending a direct **health-check RPC** to the datanode hosting the flagged region. Only a failed response from this targeted probe confirms the failure and triggers the failover procedure. This two-step approach prevents unnecessary migrations during temporary network partitions.

## Region Failover Orchestration

Once a region failure is confirmed, `RegionFailureHandler` executes a coordinated migration through four distinct stages.

### The Four-Stage Failover Flow

1. **Candidate selection** – The meta-service evaluates datanodes based on load metrics, tenant assignments, and historical CPU/IO usage to select optimal replacement candidates.

2. **State transition** – The failing region's state transitions to `Passive`. In this state, frontend nodes block writes while permitting reads to maintain query availability.

3. **Close and open RPCs** – Metasrv issues a *close* request to the failed datanode and *open* requests to candidates. The open operation reconstructs the region from the remote WAL (Bunshin):

```rust
// Conceptual flow from src/meta-srv/src/procedure/region_migration.rs
close_region(failed_datanode, region_id).await?;
open_region(candidate_datanode, region_id, remote_wal).await?;

```

4. **Confirmation** – Candidates report readiness through their regular heartbeats. Metasrv then transitions the region state to `Active`, lifting the write block and restoring full service.

### State Management During Failover

The orchestration logic resides in `RegionFailureHandler` (instantiated in [`builder.rs`](https://github.com/greptimeteam/greptimedb/blob/main/builder.rs) lines 81-85), which embeds both the `RegionSupervisor` and `HeartbeatAcceptor` to react to liveness events in real-time. The handler ensures atomic state transitions and prevents split-brain scenarios during migration.

## Safety Guards and Configuration

The HA subsystem includes critical safeguards to prevent data loss during automated failover operations.

### Remote WAL Requirements

Region failover requires a **remote WAL** (such as Bunshin) to ensure data durability during migration. The system enforces this at startup through a hard assertion in the builder:

```rust
// src/meta-srv/src/metasrv/builder.rs lines 9-16
ensure!(
    options.allow_region_failover_on_local_wal,
    UnexpectedSnafu {
        msg: "Region failover requires remote WAL. \
              Set allow_region_failover_on_local_wal=true to override (unsafe)."
    }
);

```

Operating with a local WAL risks data loss during failover, as the failed node's local storage becomes inaccessible.

### Configurable Failover Parameters

Administrators control HA behavior through `MetaSrvOptions` defined in [`config/config.md`](https://github.com/greptimeteam/greptimedb/blob/main/config/config.md) (lines 351-353):

| Parameter | Description | Default |
|-----------|-------------|---------|
| `enable_region_failover` | Master switch for the HA subsystem | `false` |
| `allow_region_failover_on_local_wal` | Permits failover with local WAL (unsafe) | `false` |
| `region_failure_detector_initialization_delay` | Grace period before detection starts | `10m` |
| `phi` | Φ-accrual threshold for failure suspicion | (configurable) |

Setting `allow_region_failover_on_local_wal = true` requires explicit opt-in and generates warning logs due to potential data loss risks.

## Summary

- **GreptimeDB implements region-level high availability** through a dedicated subsystem comprising failure detection, failover orchestration, and configuration layers.
- **Φ-accrual failure detection** combines passive heartbeat analysis with active health checks to minimize false positives while ensuring rapid failure recognition.
- **Automated failover** follows a four-stage process: candidate selection, passive state transition, coordinated close/open RPCs, and confirmation via heartbeat.
- **Remote WAL is mandatory** for safe failover; local WAL deployments require explicit unsafe opt-in via `allow_region_failover_on_local_wal`.
- **Key source files** include [`src/meta-srv/src/region/supervisor.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/region/supervisor.rs) for detection logic and [`src/meta-srv/src/metasrv/builder.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/metasrv/builder.rs) for subsystem initialization.

## Frequently Asked Questions

### How does GreptimeDB detect datanode failures without generating false positives?

GreptimeDB uses a two-stage detection mechanism. First, the `RegionFailureDetector` applies Φ-accrual algorithms to regular heartbeats (passive detection). Only after this passive flag does the system send targeted health-check RPCs (active detection). This combination prevents unnecessary failovers during temporary network glitches while maintaining sensitivity to actual hardware failures.

### What happens to write operations during a region failover?

When a region enters the `Passive` state during failover, frontend nodes immediately block write operations to that region while continuing to serve read requests. Writes resume automatically once the region transitions back to `Active` on the new datanode and confirms readiness via heartbeat. This ensures consistency while minimizing query downtime.

### Can I enable region failover if my cluster uses local WAL storage?

Technically yes, by setting `allow_region_failover_on_local_wal = true` in your meta-service configuration, but this is strongly discouraged. Local WAL storage means region data resides only on the failed node's disk, making recovery impossible if the node is permanently lost. The system logs warnings when this unsafe configuration is active, and data loss is likely during failover scenarios.

### Where is the failover orchestration logic implemented in the source code?

The core orchestration resides in [`src/meta-srv/src/metasrv/builder.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/metasrv/builder.rs) (lines 68-85), which wires the `RegionFailureHandler` and `RegionSupervisor` into the Metasrv startup sequence. The actual migration procedures executing close/open RPCs are implemented in [`src/meta-srv/src/procedure/region_migration.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/procedure/region_migration.rs), while failure detection algorithms live in [`src/meta-srv/src/region/supervisor.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/meta-srv/src/region/supervisor.rs).