How Different Index Types (Inverted, Fulltext, Skipping, Vector) Improve Query Performance in GreptimeDB

GreptimeDB uses four specialized index types—inverted, fulltext, skipping, and vector—to prune SST files early and reduce I/O by converting full table scans into targeted bitmap-filtered reads or graph traversals.

GreptimeDB stores time-series data in immutable SST files, and during memtable flushing, the engine generates auxiliary index structures that enable aggressive data pruning. Each index type targets specific predicate patterns, allowing the query optimizer to skip irrelevant data blocks entirely. According to the greptimeteam/greptimedb source code, these indexes are stored before column data in the same blob to enable memory-mapped access without reading the underlying rows.

Inverted Index: Bitmap-Based Filtering for Equality and Range Predicates

The inverted index accelerates exact match (=) and range (>, <, BETWEEN) queries on primitive columns by mapping distinct values to row positions using a finite-state-transducer (FST).

During SST creation, the SingleIndexWriter in src/index/src/inverted_index/format/writer/single.rs builds the FST while the InvertedIndexer in src/mito2/src/sst/index/inverted_index/creator.rs attaches the index to the column metadata. Each distinct value is paired with a bitmap indicating which rows contain that value.

When executing a query like WHERE id = 42, the InvertedIndexApplier loads the memory-mapped FST, extracts the bitmap for value 42, and restricts the scan to only those row offsets. This transforms a full-table scan into a bitmap-filtered read, drastically reducing I/O.

Fulltext Index: Token-Level Text Search

For string matching operations like LIKE, MATCH, or CONTAINS, GreptimeDB provides a fulltext index with two backend options: Bloom (pure Rust, fast builds) and Tantivy (feature-rich inverted text engine).

The FulltextIndexCreator in src/mito2/src/sst/index/fulltext_index/creator.rs handles index creation, storing token-to-row mappings. When a query uses MATCH(content, 'error'), the optimizer invokes the fulltext applier to retrieve a bitmap of candidate rows containing the token. This avoids full column scans and enables token-level filtering.

Column options for fulltext indexes are defined in src/datatypes/src/schema/column_schema.rs, where the FULLTEXT_KEY metadata constant marks columns with this index type.

Skipping Index: Bloom Filters for Early Exit

The skipping index (Bloom filter) enables rapid elimination of SST files that definitely do not contain queried values. This is critical for partition pruning and early-exit scans.

During flushing, BloomFilterCreator in src/mito2/src/sst/index/bloom_filter/creator.rs builds a probabilistic bitmap for each indexed column. The SkippingIndexOptions struct in src/datatypes/src/schema/column_schema.rs controls granularity and false-positive rates.

When a predicate like WHERE tags = 'urgent' arrives, the executor checks the Bloom filter first. If the filter reports the value is definitely absent, the entire SST or region is skipped, eliminating unnecessary disk reads.

Vector Index: HNSW Graphs for Approximate Nearest Neighbor Search

For high-dimensional vector data (VECTOR type), GreptimeDB implements HNSW (Hierarchical Navigable Small World) graphs to support approximate nearest-neighbor (ANN) search.

The HnswVectorIndexCreator in src/index/src/vector/create.rs builds the graph structure during SST creation. When a query uses VECTOR_DISTANCE with a LIMIT clause, the optimizer creates a VectorSearchState (defined in src/query/src/optimizer/scan_hint/vector_search.rs) that triggers a graph traversal.

Instead of computing distances across the entire table, the index visits only a bounded candidate set (controlled by parameters like ef=200), performing expensive distance calculations on a tiny fraction of vectors.

Metadata and Storage Architecture

Index availability is tracked via metadata constants in src/datatypes/src/schema/column_schema.rs:

pub const INVERTED_INDEX_KEY: &str = "greptime:inverted_index";
pub const FULLTEXT_KEY: &str = "greptime:fulltext";
pub const SKIPPING_INDEX_KEY: &str = "greptime:skipping_index";
pub const VECTOR_INDEX_KEY: &str = "greptime:vector_index";

Each index is stored before the actual column data in the SST blob, enabling memory-mapped access without loading data blocks. The meta structs (e.g., InvertedIndexMeta, FulltextIndexMeta) record offsets and sizes for fast random access.

Practical Usage Examples

Creating Tables with Multiple Index Types

You can define all four index types in a single CREATE TABLE statement:

CREATE TABLE demo (
    id      BIGINT,
    name    STRING,
    content STRING,
    tags    STRING,
    emb     VECTOR(128),
    PRIMARY KEY (id)
) WITH (
    INVERTED INDEX (id, name),
    FULLTEXT INDEX (content) USING BLOOM,
    SKIPPING INDEX (tags) GRANULARITY 10240 FALSE_POSITIVE_RATE 0.01,
    VECTOR INDEX (emb) ENGINE HNSW METRIC COSINE CONNECTIVITY 16 EXPANSION_ADD 200
);

The parser maps these options to internal structs like FulltextOptions, SkippingIndexOptions, and VectorIndexOptions, storing them in the column metadata using the keys shown above.

Query Execution Patterns

Different predicates trigger different index appliers:

1. Inverted index: WHERE id = 42 loads the FST and filters rows via bitmap.
2. Fulltext index: WHERE MATCH(content, 'error') queries the Bloom or Tantivy backend for matching row bitmaps.
3. Skipping index: WHERE tags = 'urgent' checks the Bloom filter first; if negative, the SST is skipped entirely.
4. Vector index: ORDER BY VECTOR_DISTANCE(emb, [...]) LIMIT 10 uses HNSW graph traversal to find candidate vectors before exact distance calculation.

Rust API Example

To manually apply an inverted index from Rust code:

use greptime_proto::v1::index::InvertedIndexMeta;
use greptime_db::sst::index::inverted_index::applier::InvertedIndexApplier;

// Assuming `sst_reader` is an opened SST file containing column `name`
let meta: InvertedIndexMeta = sst_reader.read_inverted_meta("name")?;
let mut applier = InvertedIndexApplier::new(&meta, &sst_reader)?;
let bitmap = applier.apply_eq(b"alice")?;  // Rows where name = "alice"

Similar appliers exist for fulltext (FulltextIndexApplier), skipping (BloomFilterApplier), and vector (VectorIndexApplier) indexes.

Summary

• Inverted indexes use FST-backed bitmaps to accelerate equality and range predicates on primitive columns.
• Fulltext indexes provide Bloom or Tantivy backends for token-level text search without full column scans.
• Skipping indexes employ Bloom filters to eliminate entire SST files when values are definitely absent.
• Vector indexes leverage HNSW graphs to perform approximate nearest-neighbor search on high-dimensional data with bounded candidate sets.
• All indexes are stored before data blocks in SST files, enabling memory-mapped access and efficient bitmap-based pruning via the ScanHint optimizer framework.

Frequently Asked Questions

What is the difference between inverted and fulltext indexes in GreptimeDB?

Inverted indexes are optimized for exact matches and range queries on structured primitive data using FST-backed bitmaps, while fulltext indexes are designed for unstructured text search using tokenization (Bloom or Tantivy backends). Use inverted indexes for IDs or categorical values, and fulltext indexes for log messages or document content.

How does the skipping index reduce storage I/O?

The skipping index uses Bloom filters to test whether a value might exist in an SST file before reading it. If the filter indicates the value is definitely not present, the query engine skips the entire file or region, avoiding disk I/O for data blocks that cannot contain matching rows.

Can I use multiple index types on the same column?

Yes, GreptimeDB allows combining index types. For example, a string column can have both an inverted index for exact equality checks and a fulltext index for pattern matching. The query optimizer selects the appropriate index based on the predicate type.

When should I choose Bloom over Tantivy for fulltext indexing?

Choose Bloom when you need fast index builds and simple token presence checks with lower resource overhead. Choose Tantivy when you require advanced text search features like phrase queries, stemming, or relevance scoring. Bloom is pure Rust and faster to construct, while Tantivy provides full-featured inverted text capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}