How GreptimeDB Manages Database Metadata Across Distributed Nodes

GreptimeDB centralizes database metadata in etcd and distributes it across nodes using the KvBackendCatalogManager, which combines etcd's strong consistency with local layered caching to provide low-latency catalog operations.

GreptimeDB (greptimeteam/greptimedb) is a cloud-native time-series database designed for distributed deployments. The system uses a sophisticated catalog architecture to manage database metadata across distributed nodes, ensuring all cluster members share a consistent view of schemas, tables, and routing information while maintaining high performance through intelligent caching.

Architecture Overview

Centralized Metadata Storage in etcd

All catalog metadata resides in a centralized key-value backend, typically etcd. This shared storage layer guarantees that every node in the cluster accesses the same metadata version. The KvBackend abstraction in src/common/meta/src/kv_backend/etcd.rs wraps etcd operations, providing linearizable reads and writes essential for distributed consistency.

The KvBackendCatalogManager Implementation

Each node runs a KvBackendCatalogManager that implements the CatalogManager trait defined in src/catalog/src/lib.rs. This stateless manager (apart from its cache registry) handles all catalog operations including database creation, schema listing, and existence checks by communicating with the shared etcd backend.

How Catalog Operations Work

Key Structure and Naming Conventions

Catalog metadata uses a structured key format in etcd. Database names are stored under the __catalog_name/<catalog> prefix. The implementation in src/common/meta/src/key/catalog_name.rs defines CatalogNameKey and provides CRUD operations for these entries.

Distributed Consistency Mechanisms

All catalog mutations flow through the KvBackend to etcd, which replicates changes across the cluster. When a node executes CREATE DATABASE, the KvBackendCatalogManager::create method writes to etcd, ensuring the transaction commits before returning. This guarantees that subsequent reads on any node will see the updated catalog state.

Local Caching for Performance

To reduce etcd round-trips, each node maintains a layered cache registry (LayeredCacheRegistryRef). The KvBackendCatalogManager caches table information, name-to-ID mappings, and routing data locally. When metadata changes, the cache invalidates or updates accordingly, allowing nodes to serve catalog queries without contacting etcd after the initial fetch.

System Tables and Metadata Merging

The catalog manager merges user-defined databases with system tables. The SystemCatalog structure in src/catalog/src/kvbackend/manager.rs provides static system tables like information_schema, pg_catalog, and numbers. These virtual tables are attached to the same manager and appear alongside user data without requiring etcd storage.

Practical Code Examples

Creating a New Database Catalog

use greptime_common::meta::key::catalog_name::CatalogNameKey;
use greptime_catalog::catalog::kvbackend::manager::KvBackendCatalogManager;

let manager = KvBackendCatalogManager { /* initialized with etcd backend */ };
let catalog_key = CatalogNameKey::new("analytics");
manager.create(catalog_key, true).await?;

This delegates to CatalogManager::create in src/common/meta/src/key/catalog_name.rs.

Checking Catalog Existence

let exists = manager.catalog_exists("analytics").await?;
println!("Analytics catalog exists? {}", exists);

The catalog_exists method (lines 46-52 of src/catalog/src/kvbackend/manager.rs) checks for the presence of the __catalog_name/analytics key in etcd.

Listing All Catalogs

let mut stream = manager.catalog_names().await?;
while let Some(name) = stream.next().await {
    println!("Catalog: {}", name?);
}

CatalogManager::catalog_names returns a BoxStream that queries keys with the __catalog_name/ prefix from catalog_name.rs.

Using the Unified CatalogManager Trait

async fn list_schemas<C: CatalogManager>(catalog_mgr: &C, catalog: &str) -> Result<Vec<String>> {
    catalog_mgr.schema_names(catalog, None).await
}

The trait abstraction in src/catalog/src/lib.rs (line 48) allows SQL planners and HTTP handlers to interact with metadata without knowing the backend implementation.

Key Source Files and Implementation Details

• src/common/meta/src/key/catalog_name.rs: Defines key formats (__catalog_name/<catalog>) and low-level catalog operations
• src/catalog/src/kvbackend/manager.rs: High-level KvBackendCatalogManager with caching and system table integration
• src/catalog/src/lib.rs: CatalogManager trait definition used by SQL planner and front-ends
• src/catalog/src/system_schema.rs: System table implementations for information_schema and pg_catalog
• src/meta-srv/src/bootstrap.rs: Meta service initialization and backend selection (etcd vs. in-memory)
• src/common/meta/src/kv_backend/etcd.rs: Etcd client implementation providing linearizable operations

Summary

• GreptimeDB stores all catalog metadata in etcd using a centralized key-value backend shared across the cluster
• The KvBackendCatalogManager provides a uniform API for catalog operations while remaining stateless except for local caching
• Catalog keys follow the __catalog_name/<catalog> format, implemented in catalog_name.rs
• Layered caching minimizes etcd queries while maintaining consistency through cache invalidation on updates
• The architecture supports pluggable backends (etcd for production, in-memory for testing) selected during meta-service startup

Frequently Asked Questions

How does GreptimeDB ensure consistency when multiple nodes modify catalog metadata?

All catalog writes go through etcd's linearizable transactions via the KvBackend abstraction. When one node creates a database using KvBackendCatalogManager::create, etcd replicates the change and guarantees that other nodes will see the update on subsequent reads, preventing split-brain scenarios.

What happens if the etcd cluster becomes unavailable?

Nodes can continue serving read operations for cached metadata using their local LayeredCacheRegistry, but catalog modifications will fail until etcd recovers. The system prioritizes availability for existing table queries over catalog changes during network partitions.

How are system tables like information_schema handled in the distributed catalog?

System tables are managed by the SystemCatalog structure attached to each KvBackendCatalogManager in kvbackend/manager.rs. These virtual tables are generated locally on each node and merged with user metadata from etcd, requiring no storage in the centralized backend.

Can GreptimeDB use alternatives to etcd for metadata storage?

Yes, the KvBackend trait allows pluggable implementations. While etcd is the default for production deployments, the system can use in-memory backends for testing, as configured in src/meta-srv/src/bootstrap.rs during MetaSrv::new initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}