How the GreptimeDB Index Module Provides Full-Text and Vector Search Capabilities

The GreptimeDB index module enables full-text and vector search by implementing the IndexCreator and IndexApplier traits for both Tantivy-based text indexing and USearch HNSW vector indexing, storing all indexes as unified Puffin blobs with standardized protobuf metadata.

The src/index/src package in the greptimeteam/greptimedb repository serves as the central hub for secondary indexing, supporting advanced query patterns beyond traditional column scans. By abstracting engine-specific implementations behind common interfaces, this module provides full-text and vector search capabilities that integrate seamlessly with the SST pipeline and query optimizer.

Full-Text Search Implementation

GreptimeDB implements full-text search using Tantivy, a Rust full-text search library. The implementation spans creation, storage, and query-time retrieval, with metadata managed through the FulltextIndexMeta protobuf structure.

Configuration and Index Creation

Full-text indexes are configured via fulltext_index::Config in src/index/src/fulltext_index.rs, which specifies the tokenizer (Analyzer) and case-sensitivity settings. During SST construction, the TantivyFulltextIndexCreator (defined in src/index/src/fulltext_index/create/tantivy.rs) manages the indexing process.

The creator receives text values via push_text, maintains an optional null bitmap using RoaringBitmap, and serializes the Tantivy index directory along with a FulltextIndexMeta protobuf. The final blob layout follows the standard index format: the Tantivy payload, followed by the protobuf metadata, preceded by a 4-byte little-endian size field.

use greptime_db::index::fulltext_index::{
    Config, Analyzer, create::TantivyFulltextIndexCreator,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Configure tokenization
let ft_cfg = Config {
    analyzer: Analyzer::English,
    case_sensitive: false,
};

// Index text during SST write
let mut creator = TantivyFulltextIndexCreator::new(ft_cfg).unwrap();
creator.push_text("the quick brown fox").await?;
creator.push_text("jumps over the lazy dog").await?;
creator.finish(&mut puffin_writer, "mytable/fulltext", Default::default()).await?;

Query Execution and Row Mapping

At query time, TantivyFulltextIndexSearcher in src/index/src/fulltext_index/search/tantivy.rs opens the on-disk index and registers the tokenizer from the stored configuration. The search method constructs a Tantivy QueryParser, executes the query, and collects matching document IDs into a BTreeSet<RowId>.

For multi-segment SSTs, the searcher translates Tantivy document IDs into logical row offsets using the reserved ROWID_FIELD_NAME field, ensuring accurate mapping even with deleted or NULL rows.

SQL Integration via Scan Hints

The query optimizer in src/query/src/optimizer/scan_hint/fulltext.rs detects MATCH(text_column, 'term') expressions and injects a FulltextScanHint. This hint triggers the full-text index path instead of brute-force scanning, mirroring the optimization strategy used for inverted indexes on exact string predicates.

Vector Search Implementation

Vector similarity search uses USearch HNSW (Hierarchical Navigable Small World) for approximate nearest neighbor (ANN) queries, with a pluggable VectorIndexEngine trait that supports future engines like VSAG.

Configuration and HNSW Parameters

Vector indexes are defined by VectorIndexConfig in src/index/src/vector.rs, specifying:

• Engine: Usearch (or future VSAG)
• Dimensions: Vector cardinality (e.g., 128, 768)
• Metric: Distance function (L2sq, Cosine, InnerProduct)
• HNSW hyperparameters: connectivity (graph degree), expansion_add (build quality), expansion_search (query recall)

use greptime_db::index::vector::{VectorIndexConfig, VectorDistanceMetric};

let vec_cfg = VectorIndexConfig {
    engine: greptime_db::store_api::storage::VectorIndexEngineType::Usearch,
    dim: 128,
    distance_metric: VectorDistanceMetric::Cosine,
    connectivity: 16,
    expansion_add: 128,
    expansion_search: 64,
};

Index Creation and Engine Abstraction

The HnswVectorIndexCreator in src/index/src/vector/create.rs implements the generic IndexCreator trait. It instantiates a concrete engine via engine::create_engine, producing a UsearchEngine (defined in src/index/src/vector/engine/usearch.rs) that wraps the USearch library.

Each non-NULL vector receives a monotonically increasing HNSW key, while NULL rows are tracked in a RoaringBitmap. The build_blob method serializes the bitmap, the USearch graph bytes, and a VectorIndexMeta protobuf containing dimension and metric statistics.

ANN Query Processing

Query-time operations are handled by HnswVectorIndexApplier in src/index/src/vector/apply.rs. The applier deserializes the null bitmap and loads the USearch engine via engine::load_engine. A query vector is passed to engine.search, which returns the nearest HNSW keys and distances. The map_keys_to_row_offsets function translates each key back to the original row offset using binary search over the bitmap to account for NULL positions.

This ensures returned row offsets correspond to the original table rows, even when the indexed subset excludes NULL values.

Planner Optimization

The optimizer in src/query/src/optimizer/scan_hint/vector_search.rs identifies ORDER BY vec_*_distance(col, <literal>) LIMIT k patterns and rewrites the execution plan to use VectorIndexScan. This scan node invokes HnswVectorIndexApplier::search, then applies standard projection and visibility filtering on the returned row offsets.

Common Architectural Patterns

Both index types share several design elements that ensure consistency across the storage engine.

Unified Puffin Blob Format

All secondary indexes use an identical on-disk layout consisting of:

1. An optional null bitmap (RoaringBitmap)
2. The engine-specific payload (Tantivy files or USearch graph)
3. A protobuf metadata block (FulltextIndexMeta or VectorIndexMeta)
4. A 4-byte little-endian meta-size prefix

This uniformity allows the puffin_manager to treat indexes generically, using IndexApplier::from_blob as a factory method for both full-text and vector indexes.

Pluggable Engine Traits

The module defines IndexCreator and IndexApplier traits in src/index/src/lib.rs, decoupling storage logic from search algorithms. For vectors, the VectorIndexEngine trait abstracts add, search, and serialization operations, enabling USearch to be replaced with VSAG or other ANN libraries without modifying the creator or applier logic.

NULL Handling and Offset Mapping

Both implementations maintain bitmaps of NULL rows. During reads, these bitmaps translate engine-specific identifiers (Tantivy doc IDs or HNSW keys) into logical row offsets. This mapping guarantees correct results when rows are sparse, deleted, or contain NULL values in indexed columns.

Code Examples

Full-Text Index Creation and Search

use greptime_db::index::fulltext_index::{
    Config, Analyzer, create::TantivyFulltextIndexCreator,
    search::TantivyFulltextIndexSearcher,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Build configuration
let ft_cfg = Config {
    analyzer: Analyzer::English,
    case_sensitive: false,
};

// Create index during SST write
let mut creator = TantivyFulltextIndexCreator::new(ft_cfg.clone()).unwrap();
creator.push_text("the quick brown fox").await?;
creator.push_text("jumps over the lazy dog").await?;
creator.finish(&mut puffin_writer, "mytable/fulltext", Default::default()).await?;

// Search at query time
let searcher = TantivyFulltextIndexSearcher::new("/path/to/puffin/blob", ft_cfg).unwrap();
let matching_rows = searcher.search("quick brown").await?; // Returns BTreeSet<RowId>

Vector Index Creation and ANN Search

use greptime_db::index::vector::{
    VectorIndexConfig, VectorDistanceMetric,
    create::HnswVectorIndexCreator, apply::HnswVectorIndexApplier,
};
use greptime_db::puffin::puffin_manager::PuffinWriter;

// Define configuration
let vec_cfg = VectorIndexConfig {
    engine: greptime_db::store_api::storage::VectorIndexEngineType::Usearch,
    dim: 128,
    distance_metric: VectorDistanceMetric::Cosine,
    connectivity: 16,
    expansion_add: 128,
    expansion_search: 64,
};

// Build index
let mut creator = HnswVectorIndexCreator::new(vec_cfg.clone()).unwrap();
creator.push_vector(&[0.1_f32; 128])?;
creator.push_null()?;
creator.push_vector(&[0.9_f32; 128])?;
creator.finish(&mut puffin_writer, "mytable/vec_index", Default::default()).await?;

// Load and search
let blob = /* read blob bytes from Puffin */;
let applier = HnswVectorIndexApplier::from_blob(&blob).unwrap();
let query = vec![0.2_f32; 128];
let result = applier.search(&query, 5).unwrap();
// result.row_offsets and result.distances contain the k-NN results

Summary

• The src/index/src module provides full-text capabilities via Tantivy and vector search via USearch HNSW, unified under common IndexCreator and IndexApplier traits defined in src/index/src/lib.rs.
• Both index types use a standardized Puffin blob format consisting of an optional null bitmap, engine payload, and protobuf metadata with a 4-byte size prefix.
• Row-offset mapping via bitmaps ensures correct result alignment even with NULL or deleted rows.
• Pluggable engines allow future vector implementations (VSAG) or full-text backends without changing the core indexing pipeline.
• Query optimization relies on scan hints (FulltextScanHint, VectorSearchHint) to route eligible SQL predicates through index paths rather than brute-force evaluation.

Frequently Asked Questions

How does GreptimeDB handle NULL values in full-text and vector indexes?

Both index types maintain a RoaringBitmap tracking NULL rows within the indexed column. During queries, the appliers use this bitmap to translate engine-specific keys (Tantivy doc IDs or HNSW keys) into logical row offsets, ensuring that NULL rows are excluded from results and subsequent row offsets remain correctly aligned with the original table.

What vector similarity metrics are supported by the index module?

According to the source code in src/index/src/vector.rs, the module supports three distance metrics via VectorDistanceMetric: L2sq (squared Euclidean), Cosine (angular distance), and InnerProduct (dot product). These metrics are passed to the USearch engine during both index construction and query execution.

Can the full-text or vector search engines be replaced with custom implementations?

Yes. The module abstracts engines behind traits: FulltextIndexCreator/FulltextIndexSearcher for text, and VectorIndexEngine for vectors. The vector implementation already supports pluggable engines (current default is USearch, with VSAG planned), and the trait-based design allows alternative backends without modifying the SST pipeline or query planner.

How does the query planner decide to use a vector index instead of brute-force scanning?

The optimizer in src/query/src/optimizer/scan_hint/vector_search.rs detects SQL patterns matching ORDER BY vec_*_distance(column, literal) LIMIT k. When found, it injects a VectorSearchHint that rewrites the plan to use VectorIndexScan, which invokes HnswVectorIndexApplier::search. If the pattern does not match or the index is unavailable, the system falls back to standard table scans with post-filtering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}