# How the Metric Engine Works for Observability Data in GreptimeDB

> Discover how GreptimeDB's Metric Engine processes observability data. Learn how it handles high-cardinality workloads using the Mito storage backbone for efficient metric storage and retrieval.

- Repository: [Greptime/greptimedb](https://github.com/greptimeteam/greptimedb)
- Tags: internals
- Published: 2026-03-02

---

**The Metric Engine is a specialized storage layer that multiplexes many logical metric tables onto a single physical wide table, enabling high-cardinality observability workloads while reusing GreptimeDB's proven Mito storage backbone.**

The Metric Engine serves as the primary interface for ingesting and querying time-series metrics in GreptimeDB (`greptimeteam/greptimedb`). Unlike traditional table-per-metric approaches, this engine implements a thin wrapper architecture that translates logical table operations into efficient physical storage patterns, specifically optimized for Prometheus-style metrics and other observability data sources.

## Core Architecture and Concepts

The Metric Engine introduces a logical-to-physical mapping abstraction that separates user-visible table schemas from the underlying storage layout.

### Logical Regions and Physical Region Groups

A **Logical Region** represents a user-visible metric table created with `ENGINE = metric`. Rather than allocating dedicated storage files, logical regions share a **Physical Region Group** consisting of two underlying Mito regions:

- **Data Region**: Stores the actual time-series rows in a wide table format
- **Metadata Region**: Maintains schema definitions and logical-to-physical mappings as key-value pairs (e.g., keys formatted as `__table_<table_id>`)

This multiplexing approach is implemented in [`src/metric-engine/src/engine.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/engine.rs), where the `MetricEngine::create_regions` function handles the translation of `CREATE TABLE` statements into logical region registrations against existing physical tables.

### Internal Columns and TSID

Every physical metric table contains two reserved internal columns defined in [`src/metric-engine/src/data_region.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/data_region.rs):

- `__tsid`: A 64-bit hash of the tag set (labels) that identifies unique time series
- `__table_id`: The identifier linking rows to their logical table origin

These columns enable efficient grouping and filtering during query execution while maintaining logical isolation between multiplexed tables.

## Data Flow and Processing

The Metric Engine intercepts storage requests and transforms them to match the physical schema before delegating to the underlying Mito engine.

### Write Path and Row Modification

When processing `INSERT` operations, `MetricEngine::handle_request` in [`src/metric-engine/src/engine.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/engine.rs) delegates to `MetricEngineInner::put_region`. The write flow follows these steps:

1. **Row Transformation**: `RowModifier::modify_row` (in [`src/metric-engine/src/row_modifier.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/row_modifier.rs)) computes the `tsid` by hashing tag values and rewrites column IDs to match the physical schema
2. **Internal Column Injection**: The modifier appends `__tsid` and `__table_id` values to each row
3. **Mito Delegation**: The transformed request is forwarded to the **DataRegion** ([`src/metric-engine/src/data_region.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/data_region.rs)), which issues the actual `RegionRequest::Put` to the Mito engine

This transformation ensures that logically separate tables can coexist in the same physical wide table without collision.

### Read Path and Column Projection

For `SELECT` queries, the engine resolves the physical region ID via the metadata region, then delegates the scan to Mito. During result processing:

1. The scan returns rows containing internal columns (`__tsid`, `__table_id`)
2. The engine strips these columns before returning results to the client
3. Column projection ensures only requested logical columns are materialized

The `handle_query` implementation in [`src/metric-engine/src/engine.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/engine.rs) manages this delegation and projection logic.

### Schema Evolution

The Metric Engine supports limited schema alterations, specifically column addition. When `ALTER TABLE ADD COLUMN` executes:

1. `MetricEngine::alter_regions` validates the request
2. `DataRegion::add_columns` (in [`src/metric-engine/src/data_region.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/data_region.rs)) generates a `RegionAlterRequest`
3. The request adds columns to the physical schema with optional inverted or skipping indexes
4. Metadata region entries update the logical table schema mappings

This approach maintains consistency across all logical tables sharing the physical region.

## Region ID Management and Utilities

The Metric Engine implements a region ID grouping scheme to address multiple physical regions through a single logical identifier. Utility functions in [`src/metric-engine/src/utils.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/utils.rs) handle the bit-manipulation:

```rust
/// Convert a logical region id to its data region id.
pub fn to_data_region_id(logical_id: RegionId) -> RegionId {
    // High-order bits encode the group; low-order bits differentiate members
}

/// Convert to metadata region id.
pub fn to_metadata_region_id(logical_id: RegionId) -> RegionId { 
    // Bitmask operations to derive metadata region address
}

```

This grouping allows the engine to treat the data and metadata regions as a unified storage unit while maintaining separate physical storage structures.

## Optimizations for Observability Workloads

The Metric Engine implements several optimizations specifically targeting high-cardinality metric ingestion:

- **Tag Hashing (`tsid`)**: Reduces high-cardinality label dimensions to a 64-bit identifier, enabling efficient time-series grouping and aggregation without full tag comparisons
- **Wide Table Physical Layout**: Minimizes file I/O overhead by storing multiple logical tables in a single physical structure, reducing the file descriptor and compaction overhead that would occur with table-per-metric designs
- **Inverted and Skipping Indexes**: Optional per-column indexes (specified via `INDEX` clauses during table creation) accelerate tag-based filtering by leveraging Mito's existing index infrastructure

These features enable the engine to process millions of time-series rows per second while maintaining full SQL query capabilities, including `SELECT`, `GROUP BY`, and `COUNT` operations.

## Implementation Examples

The following Rust code demonstrates end-to-end metric engine usage, adapted from the integration tests in [`tests-integration/tests/sql.rs`](https://github.com/greptimeteam/greptimedb/blob/main/tests-integration/tests/sql.rs):

```rust
// 1️⃣ Create a physical metric table (the shared wide table)
let sql = r#"
CREATE TABLE phy (
    ts TIMESTAMP TIME INDEX,
    val DOUBLE
) ENGINE = metric WITH ("physical_metric_table" = "");
"#;
client.execute(sql).await?;

// 2️⃣ Create two logical metric tables that map onto `phy`
let create_logical = |name: &str, pk: &str| async move {
    let sql = format!(
        "CREATE TABLE {name} (
            ts TIMESTAMP TIME INDEX,
            val DOUBLE,
            {pk} STRING PRIMARY KEY
        ) ENGINE = metric WITH (\"on_physical_table\" = \"phy\");"
    );
    client.execute(&sql).await
};

create_logical("t1", "host").await?;
create_logical("t2", "job").await?;

// 3️⃣ Insert rows into a logical table – engine hashes tags into `__tsid`
let insert = r#"
INSERT INTO t1 (ts, val, host) VALUES
    ('2024-01-01 00:00:00', 1.23, 'host_a'),
    ('2024-01-01 00:01:00', 3.45, 'host_b');
"#;
client.execute(insert).await?;

// 4️⃣ Query – the engine strips internal columns before returning
let rows = client
    .query("SELECT host, val FROM t1 ORDER BY ts")
    .await?
    .collect::<Vec<_>>()
    .await;
assert_eq!(rows.len(), 2);

```

These SQL operations work identically via GreptimeDB's HTTP and MySQL-compatible endpoints, as validated in [`tests-integration/tests/http.rs`](https://github.com/greptimeteam/greptimedb/blob/main/tests-integration/tests/http.rs).

## Summary

- The **Metric Engine** multiplexes many logical metric tables onto a single physical wide table, reducing storage overhead while maintaining logical isolation.
- **Logical regions** map to **physical region groups** (data + metadata regions) via the Mito engine, with internal columns `__tsid` and `__table_id` enabling time-series identification.
- **Write operations** transform rows through `RowModifier::modify_row` to compute tag hashes and adjust column IDs before delegating to the underlying data region.
- **Read operations** resolve physical region IDs via the metadata region, execute scans through Mito, and strip internal columns before returning results.
- **Schema evolution** supports column addition through `DataRegion::add_columns`, maintaining consistency across all logical tables sharing the physical region.

## Frequently Asked Questions

### How does the Metric Engine handle high-cardinality time series?

The Metric Engine addresses high-cardinality workloads by hashing tag sets into a 64-bit `__tsid` identifier during ingestion. This transformation occurs in `RowModifier::modify_row` within [`src/metric-engine/src/row_modifier.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/row_modifier.rs), reducing complex label dimensions to a single integer that enables efficient grouping and aggregation without expensive string comparisons.

### What is the difference between a logical region and a physical region in the Metric Engine?

A **logical region** represents a user-visible metric table created with `ENGINE = metric`, while a **physical region group** consists of two underlying Mito regions: a data region storing actual rows and a metadata region storing schema mappings. The logical region has no dedicated storage files; instead, it shares the physical wide table with other logical tables, using `__table_id` to isolate rows as implemented in [`src/metric-engine/src/data_region.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/data_region.rs).

### Can I alter the schema of an existing metric table?

The Metric Engine supports limited schema alterations, specifically **column addition**. When executing `ALTER TABLE ADD COLUMN`, the `MetricEngine::alter_regions` method delegates to `DataRegion::add_columns` in [`src/metric-engine/src/data_region.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/metric-engine/src/data_region.rs), which generates a `RegionAlterRequest` to modify the physical schema. This operation affects all logical tables sharing the physical region, maintaining consistency across the multiplexed storage layout.

### How does the Metric Engine optimize query performance for observability data?

The engine implements several optimizations targeting metric workloads: **tag hashing** (`tsid`) reduces high-cardinality dimensions to 64-bit identifiers for efficient aggregation; **inverted and skipping indexes** specified during table creation accelerate tag-based filtering; and the **wide table physical layout** minimizes file I/O overhead by storing multiple logical tables in a single physical structure. These features, combined with Mito's compaction and indexing infrastructure, enable ingestion of millions of rows per second while supporting full SQL analytics.