Use Cases for VelesDB's ColumnStore Feature: High-Throughput Metadata Filtering and Hybrid Search

VelesDB's ColumnStore enables 50+ million items per second metadata filtering through typed column storage and bitmap indexes, making it ideal for real-time recommendation engines and hybrid vector-relational queries.

VelesDB pairs a graph-vector engine with a lightweight column-store designed for structured, frequently-filtered metadata. The ColumnStore feature in cyberlife-coder/velesdb stores typed columns (INT, FLOAT, STRING, BOOL) in contiguous vectors, enabling cache-friendly sequential access and bitmap-based filtering that outperforms JSON parsing by nearly 3x throughput.

High-Throughput Metadata Filtering

The ColumnStore eliminates expensive JSON deserialization by running filter operations directly on typed columns. According to the source code in crates/velesdb-core/src/column_store/mod.rs, the module achieves 50+ million items per second compared to approximately 19 million per second with JSON parsing【mod.rs†L6-L10】.

Columns are stored in a HashMap<String, TypedColumn> where each TypedColumn maintains a dense Vec<Option<T>>【mod.rs†L52-L55】. This contiguous memory layout enables SIMD-friendly operations and reduces cache misses during sequential scans.

Supported filter operations include:

• Equality filters (Eq, In) for exact matching
• Range filters (Gt, Lt, Range) for numeric comparisons
• Bitmap compositions for complex AND/OR logic without materializing intermediate vectors

Fast Primary-Key Lookups

For point queries and updates, the ColumnStore provides O(1) row retrieval through a dedicated primary-key index. The primary_index: HashMap<i64, usize> maps primary key values directly to row offsets【mod.rs†L60-L63】.

To enable this functionality, create the store with ColumnStore::with_primary_key, which validates that the primary key column is of type Int and builds the index during initialization【mod.rs†L90-L106】. This design supports instant lookups for single-row updates and deletes without full table scans.

Efficient Batch Upserts and Deletes

Bulk operations leverage bitmap-based deleted-row tracking using RoaringBitmap for O(1) containment checks and cheap vacuuming【mod.rs†L66-L68】. When deleting rows, the delete_by_pk method updates both the internal FxHashSet and the deletion bitmap simultaneously【mod.rs†L97-L111】.

Batch APIs in batch.rs enable atomic multi-row operations. The bitmap structure allows the system to mark rows as deleted without immediate memory reallocation, deferring physical removal to background vacuum processes.

Push-Down Filtering in Cross-Model Queries

When executing VelesQL queries that mix vector similarity with column conditions, the query planner automatically splits WHERE clauses between engines. Graph filters route to the vector engine, while qualified column-store predicates push down to the ColumnStore module【docs/reference/VELESQL_JOIN.md†L31-L44】.

The planner's internal PushdownAnalysis inspects SelectColumns and routes column predicates to the column-store module's filter_* functions. For example, a query filtering prices.amount < 500 executes directly against the typed column data rather than loading and parsing JSON attributes.

Low-Latency Joins Between Graph and Relational Data

CROSS-STORE JOIN operations combine graph node sets with column-store tables to reduce data movement between engines. The ColumnStore supplies filtered rows (such as price ranges) that join against vector search results, dramatically reducing memory pressure【VELESQL_JOIN.md†L11-L18】.

The execution flow follows four stages: parse → filter analysis → execute graph query → execute column-store query → batch-adaptive join【VELESQL_JOIN.md†L59-L74】. This architecture enables hybrid search workloads where vector similarity identifies candidate nodes and column-store filters apply business logic constraints.

TTL and Expiration Management

Per-row expiry timestamps stored in row_expiry: HashMap<usize, u64> enable automatic pruning of stale metadata without full table scans【mod.rs†L69-L70】. The vacuum process iterates over non-deleted rows using the bitmap for fast skipping, checking each row's expiration timestamp against the current time【vacuum.rs†L188-L193】.

This capability suits time-series datasets and session management where metadata must automatically expire while preserving graph relationships.

Complex Analytical Filters

Bitmap-based filter APIs (filter_eq_int_bitmap, filter_range_int_bitmap) return RoaringBitmap instances that support efficient set-based logical operations【filter.rs†L200-L262】. These functions enable:

• Pre-filtering large datasets before materializing row IDs
• Boolean composition of multiple conditions without intermediate allocations
• Vectorized operations on compressed bitmap representations

Practical Code Examples

The following Rust snippets demonstrate common ColumnStore interactions using the core API:

// Create a column store with a primary key and schema
let mut store = ColumnStore::with_primary_key(
    &[("id", ColumnType::Int), ("price", ColumnType::Int), ("category", ColumnType::String)],
    "id",
).unwrap(); // crates/velesdb-core/src/column_store/mod.rs

// Insert rows with automatic primary-key indexing
store.insert_row(&[
    ("id", ColumnValue::Int(1)),
    ("price", ColumnValue::Int(199)),
    ("category", ColumnValue::String(store.string_table_mut().get_or_insert("books"))),
]).unwrap();

// Fast bitmap filter for logical composition
let cheap_items = store.filter_eq_int_bitmap("price", 199);
// crates/velesdb-core/src/column_store/filter.rs

// O(1) primary key lookup
if let Some(idx) = store.get_row_idx_by_pk(1) {
    let price_json = store.get_value_as_json("price", idx).unwrap();
    println!("Row 1 price = {}", price_json);
}

// Delete with bitmap tracking
store.delete_by_pk(2); // crates/velesdb-core/src/column_store/mod.rs

For VelesQL cross-store joins, the planner automatically pushes column predicates to the ColumnStore:

let query = r#"
    MATCH (p:Product)
    JOIN prices ON prices.product_id = p.id
    WHERE vector NEAR $vec
      AND prices.amount < 100          -- pushed to ColumnStore
    RETURN p.name, prices.amount
"#;
// See docs/reference/VELESQL_JOIN.md for push-down rules

Summary

• High-throughput filtering: 50M+ items/sec performance via typed column vectors and bitmap indexes in mod.rs
• Instant lookups: O(1) primary-key access through HashMap<i64, usize> indexing
• Efficient mutations: RoaringBitmap tracking for batch deletes and vacuum operations
• Query optimization: Automatic predicate push-down for hybrid vector-relational queries
• Temporal data: Built-in TTL support with row_expiry HashMap and bitmap-accelerated vacuuming
• Analytical power: Bitmap filter APIs enabling complex boolean logic without materialization

Frequently Asked Questions

What data types does VelesDB ColumnStore support?

The ColumnStore supports INT, FLOAT, STRING, and BOOL types stored in dense vectors. Each column maintains a Vec<Option<T>> structure allowing NULL values while preserving cache-friendly sequential access patterns【mod.rs†L52-L55】.

How does ColumnStore achieve faster filtering than JSON parsing?

By storing typed data in contiguous memory vectors rather than serialized JSON strings, filter operations execute directly on primitive types without deserialization overhead. The source code documents 50+ million items per second throughput versus approximately 19 million per second for JSON parsing【mod.rs†L6-L10】.

Can ColumnStore handle primary keys other than integers?

Currently, the primary-key index requires an Int type column. The with_primary_key constructor validates the schema and returns an error if the specified column is not an integer type【mod.rs†L90-L106】. The index uses HashMap<i64, usize> for O(1) lookups【mod.rs†L60-L63】.

How does the bitmap filtering work for complex queries?

Functions like filter_eq_int_bitmap and filter_range_int_bitmap return RoaringBitmap instances representing matching row sets. These bitmaps support efficient AND/OR operations before materializing final row IDs, eliminating intermediate vector allocations during complex analytical queries【filter.rs†L200-L262】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}