# How the GreptimeDB Index Module Provides Full-Text and Vector Search Capabilities

> Explore how the GreptimeDB index module delivers powerful full-text and vector search using Tantivy and USearch. Discover unified index storage with Puffin blobs and standardized metadata.

- Repository: [Greptime/greptimedb](https://github.com/greptimeteam/greptimedb)
- Tags: deep-dive
- Published: 2026-03-02

---

**The GreptimeDB `index` module enables full-text and vector search by implementing the `IndexCreator` and `IndexApplier` traits for both Tantivy-based text indexing and USearch HNSW vector indexing, storing all indexes as unified Puffin blobs with standardized protobuf metadata.**

The `src/index/src` package in the `greptimeteam/greptimedb` repository serves as the central hub for secondary indexing, supporting advanced query patterns beyond traditional column scans. By abstracting engine-specific implementations behind common interfaces, this module provides full-text and vector search capabilities that integrate seamlessly with the SST pipeline and query optimizer.

## Full-Text Search Implementation

GreptimeDB implements full-text search using **Tantivy**, a Rust full-text search library. The implementation spans creation, storage, and query-time retrieval, with metadata managed through the `FulltextIndexMeta` protobuf structure.

### Configuration and Index Creation

Full-text indexes are configured via `fulltext_index::Config` in [`src/index/src/fulltext_index.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/fulltext_index.rs), which specifies the tokenizer (`Analyzer`) and case-sensitivity settings. During SST construction, the `TantivyFulltextIndexCreator` (defined in [`src/index/src/fulltext_index/create/tantivy.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/fulltext_index/create/tantivy.rs)) manages the indexing process.

The creator receives text values via `push_text`, maintains an optional null bitmap using `RoaringBitmap`, and serializes the Tantivy index directory along with a `FulltextIndexMeta` protobuf. The final blob layout follows the standard index format: the Tantivy payload, followed by the protobuf metadata, preceded by a 4-byte little-endian size field.

```rust
use greptime_db::index::fulltext_index::{
    Config, Analyzer, create::TantivyFulltextIndexCreator,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Configure tokenization
let ft_cfg = Config {
    analyzer: Analyzer::English,
    case_sensitive: false,
};

// Index text during SST write
let mut creator = TantivyFulltextIndexCreator::new(ft_cfg).unwrap();
creator.push_text("the quick brown fox").await?;
creator.push_text("jumps over the lazy dog").await?;
creator.finish(&mut puffin_writer, "mytable/fulltext", Default::default()).await?;

```

### Query Execution and Row Mapping

At query time, `TantivyFulltextIndexSearcher` in [`src/index/src/fulltext_index/search/tantivy.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/fulltext_index/search/tantivy.rs) opens the on-disk index and registers the tokenizer from the stored configuration. The `search` method constructs a Tantivy `QueryParser`, executes the query, and collects matching document IDs into a `BTreeSet<RowId>`.

For multi-segment SSTs, the searcher translates Tantivy document IDs into logical row offsets using the reserved `ROWID_FIELD_NAME` field, ensuring accurate mapping even with deleted or NULL rows.

### SQL Integration via Scan Hints

The query optimizer in [`src/query/src/optimizer/scan_hint/fulltext.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/query/src/optimizer/scan_hint/fulltext.rs) detects `MATCH(text_column, 'term')` expressions and injects a `FulltextScanHint`. This hint triggers the full-text index path instead of brute-force scanning, mirroring the optimization strategy used for inverted indexes on exact string predicates.

## Vector Search Implementation

Vector similarity search uses **USearch** HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor (ANN) queries, with a pluggable `VectorIndexEngine` trait that supports future engines like VSAG.

### Configuration and HNSW Parameters

Vector indexes are defined by `VectorIndexConfig` in [`src/index/src/vector.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/vector.rs), specifying:

- **Engine**: `Usearch` (or future VSAG)
- **Dimensions**: Vector cardinality (e.g., 128, 768)
- **Metric**: Distance function (`L2sq`, `Cosine`, `InnerProduct`)
- **HNSW hyperparameters**: `connectivity` (graph degree), `expansion_add` (build quality), `expansion_search` (query recall)

```rust
use greptime_db::index::vector::{VectorIndexConfig, VectorDistanceMetric};

let vec_cfg = VectorIndexConfig {
    engine: greptime_db::store_api::storage::VectorIndexEngineType::Usearch,
    dim: 128,
    distance_metric: VectorDistanceMetric::Cosine,
    connectivity: 16,
    expansion_add: 128,
    expansion_search: 64,
};

```

### Index Creation and Engine Abstraction

The `HnswVectorIndexCreator` in [`src/index/src/vector/create.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/vector/create.rs) implements the generic `IndexCreator` trait. It instantiates a concrete engine via `engine::create_engine`, producing a `UsearchEngine` (defined in [`src/index/src/vector/engine/usearch.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/vector/engine/usearch.rs)) that wraps the USearch library.

Each non-NULL vector receives a monotonically increasing HNSW key, while NULL rows are tracked in a `RoaringBitmap`. The `build_blob` method serializes the bitmap, the USearch graph bytes, and a `VectorIndexMeta` protobuf containing dimension and metric statistics.

### ANN Query Processing

Query-time operations are handled by `HnswVectorIndexApplier` in [`src/index/src/vector/apply.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/vector/apply.rs). The applier deserializes the null bitmap and loads the USearch engine via `engine::load_engine`. A query vector is passed to `engine.search`, which returns the nearest HNSW keys and distances. The `map_keys_to_row_offsets` function translates each key back to the original row offset using binary search over the bitmap to account for NULL positions.

This ensures returned row offsets correspond to the original table rows, even when the indexed subset excludes NULL values.

### Planner Optimization

The optimizer in [`src/query/src/optimizer/scan_hint/vector_search.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/query/src/optimizer/scan_hint/vector_search.rs) identifies `ORDER BY vec_*_distance(col, <literal>) LIMIT k` patterns and rewrites the execution plan to use `VectorIndexScan`. This scan node invokes `HnswVectorIndexApplier::search`, then applies standard projection and visibility filtering on the returned row offsets.

## Common Architectural Patterns

Both index types share several design elements that ensure consistency across the storage engine.

### Unified Puffin Blob Format

All secondary indexes use an identical on-disk layout consisting of:

1. An optional null bitmap (`RoaringBitmap`)
2. The engine-specific payload (Tantivy files or USearch graph)
3. A protobuf metadata block (`FulltextIndexMeta` or `VectorIndexMeta`)
4. A 4-byte little-endian meta-size prefix

This uniformity allows the `puffin_manager` to treat indexes generically, using `IndexApplier::from_blob` as a factory method for both full-text and vector indexes.

### Pluggable Engine Traits

The module defines `IndexCreator` and `IndexApplier` traits in [`src/index/src/lib.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/lib.rs), decoupling storage logic from search algorithms. For vectors, the `VectorIndexEngine` trait abstracts `add`, `search`, and serialization operations, enabling USearch to be replaced with VSAG or other ANN libraries without modifying the creator or applier logic.

### NULL Handling and Offset Mapping

Both implementations maintain bitmaps of NULL rows. During reads, these bitmaps translate engine-specific identifiers (Tantivy doc IDs or HNSW keys) into logical row offsets. This mapping guarantees correct results when rows are sparse, deleted, or contain NULL values in indexed columns.

## Code Examples

### Full-Text Index Creation and Search

```rust
use greptime_db::index::fulltext_index::{
    Config, Analyzer, create::TantivyFulltextIndexCreator,
    search::TantivyFulltextIndexSearcher,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Build configuration
let ft_cfg = Config {
    analyzer: Analyzer::English,
    case_sensitive: false,
};

// Create index during SST write
let mut creator = TantivyFulltextIndexCreator::new(ft_cfg.clone()).unwrap();
creator.push_text("the quick brown fox").await?;
creator.push_text("jumps over the lazy dog").await?;
creator.finish(&mut puffin_writer, "mytable/fulltext", Default::default()).await?;

// Search at query time
let searcher = TantivyFulltextIndexSearcher::new("/path/to/puffin/blob", ft_cfg).unwrap();
let matching_rows = searcher.search("quick brown").await?; // Returns BTreeSet<RowId>

```

### Vector Index Creation and ANN Search

```rust
use greptime_db::index::vector::{
    VectorIndexConfig, VectorDistanceMetric,
    create::HnswVectorIndexCreator, apply::HnswVectorIndexApplier,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Define configuration
let vec_cfg = VectorIndexConfig {
    engine: greptime_db::store_api::storage::VectorIndexEngineType::Usearch,
    dim: 128,
    distance_metric: VectorDistanceMetric::Cosine,
    connectivity: 16,
    expansion_add: 128,
    expansion_search: 64,
};

// Build index
let mut creator = HnswVectorIndexCreator::new(vec_cfg.clone()).unwrap();
creator.push_vector(&[0.1_f32; 128])?;
creator.push_null()?;
creator.push_vector(&[0.9_f32; 128])?;
creator.finish(&mut puffin_writer, "mytable/vec_index", Default::default()).await?;

// Load and search
let blob = /* read blob bytes from Puffin */;
let applier = HnswVectorIndexApplier::from_blob(&blob).unwrap();
let query = vec![0.2_f32; 128];
let result = applier.search(&query, 5).unwrap();
// result.row_offsets and result.distances contain the k-NN results

```

## Summary

- The `src/index/src` module provides full-text capabilities via **Tantivy** and vector search via **USearch HNSW**, unified under common `IndexCreator` and `IndexApplier` traits defined in [`src/index/src/lib.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/lib.rs).
- Both index types use a standardized **Puffin blob format** consisting of an optional null bitmap, engine payload, and protobuf metadata with a 4-byte size prefix.
- **Row-offset mapping** via bitmaps ensures correct result alignment even with NULL or deleted rows.
- **Pluggable engines** allow future vector implementations (VSAG) or full-text backends without changing the core indexing pipeline.
- Query optimization relies on **scan hints** (`FulltextScanHint`, `VectorSearchHint`) to route eligible SQL predicates through index paths rather than brute-force evaluation.

## Frequently Asked Questions

### How does GreptimeDB handle NULL values in full-text and vector indexes?

Both index types maintain a `RoaringBitmap` tracking NULL rows within the indexed column. During queries, the appliers use this bitmap to translate engine-specific keys (Tantivy doc IDs or HNSW keys) into logical row offsets, ensuring that NULL rows are excluded from results and subsequent row offsets remain correctly aligned with the original table.

### What vector similarity metrics are supported by the index module?

According to the source code in [`src/index/src/vector.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/index/src/vector.rs), the module supports three distance metrics via `VectorDistanceMetric`: **L2sq** (squared Euclidean), **Cosine** (angular distance), and **InnerProduct** (dot product). These metrics are passed to the USearch engine during both index construction and query execution.

### Can the full-text or vector search engines be replaced with custom implementations?

Yes. The module abstracts engines behind traits: `FulltextIndexCreator`/`FulltextIndexSearcher` for text, and `VectorIndexEngine` for vectors. The vector implementation already supports pluggable engines (current default is USearch, with VSAG planned), and the trait-based design allows alternative backends without modifying the SST pipeline or query planner.

### How does the query planner decide to use a vector index instead of brute-force scanning?

The optimizer in [`src/query/src/optimizer/scan_hint/vector_search.rs`](https://github.com/greptimeteam/greptimedb/blob/main/src/query/src/optimizer/scan_hint/vector_search.rs) detects SQL patterns matching `ORDER BY vec_*_distance(column, literal) LIMIT k`. When found, it injects a `VectorSearchHint` that rewrites the plan to use `VectorIndexScan`, which invokes `HnswVectorIndexApplier::search`. If the pattern does not match or the index is unavailable, the system falls back to standard table scans with post-filtering.