How to Implement Cross-Directory Relational Queries in OpenViking

Cross-directory relational queries in OpenViking leverage the DirIndex Trie structure and the make_path_field_copy method to retrieve records matching hierarchical path fields and their sub-paths up to a configurable depth using the -d=N parameter.

OpenViking stores hierarchical resources such as files and URLs as path fields, enabling efficient cross-directory relational queries through its specialized directory index. This feature allows developers to query records within a directory prefix and optionally include descendants up to a specific depth. According to the volcengine/OpenViking source code, the implementation relies on the DirIndex class and bitmap merging logic to deliver sub-millisecond query performance even on multi-million-record collections.

DirIndex Architecture and Path Field Indexing

The core of cross-directory queries is the DirIndex class defined in src/index/detail/scalar/bitmap_holder/dir_index.h and implemented in src/index/detail/scalar/bitmap_holder/dir_index.cpp. This component maintains a Trie (prefix tree) of all path keys where each TrieNode stores a path segment and an is_leaf_ flag marking concrete resource keys.

When indexing, DirIndex::add_key(key) splits the path into segments and walks the tree, creating nodes as needed and marking the terminal node as a leaf. For querying, DirIndex::get_merged_bitmap(prefix, depth, out_set) locates the prefix node and recursively collects all leaf paths up to the specified depth, returning bitmap identifiers that represent matching records.

Bitmap Merging via make_path_field_copy

The FieldBitmapGroupSet::make_path_field_copy function in src/index/detail/scalar/bitmap_holder/bitmap_field_group.cpp orchestrates cross-directory bitmap generation. This method accepts a field name, a list of path prefixes, and a depth parameter. It resolves the field's bitmap group (which must be a DirIndex group), calls DirIndex::get_merged_bitmap for each prefix, and merges the resulting bitmaps using FastUnion into a single result set for filter evaluation.

Configuring Query Depth with the para Parameter

Cross-directory queries support configurable traversal depth through the para field in the filter JSON. The parser parse_dir_semantic_para in src/index/detail/scalar/filter/filter_ops.cpp extracts depth from strings formatted as -d=N, where N is an integer between -1 and 50.

When para is set to -d=2, the query includes the prefix path and descendants up to two levels deep. If para is omitted or set to -d=-1, the query traverses all descendants (unbounded depth). Values outside the [-1, 50] range are automatically clamped by the parser.

Implementing Cross-Directory Queries

JSON Filter DSL Syntax

To execute a cross-directory query, construct a filter object targeting a path field with the must operator. The conds array specifies the path prefixes, and the optional para field controls depth.

{
  "op": "and",
  "conds": [
    {
      "op": "must",
      "field": "path",
      "conds": ["/project/docs"],
      "para": "-d=1"
    },
    {
      "op": "range",
      "field": "price",
      "gte": 10,
      "lt": 100
    }
  ]
}

This query returns records where the path field matches /project/docs or its immediate children (depth 1), combined with a price range filter.

Python Client Implementation

The OpenViking Python client serializes these filter dictionaries and transmits them to the search API.

from openviking import OpenVikingClient

client = OpenVikingClient()

filter_expr = {
    "op": "and",
    "conds": [
        {
            "op": "must",
            "field": "path",
            "conds": ["/project/docs"],
            "para": "-d=1"
        },
        {
            "op": "range",
            "field": "price",
            "gte": 10,
            "lt": 100
        }
    ]
}

results = client.search(
    collection="my_collection",
    query_vector=[0.12, 0.58, 0.33],
    filter=filter_expr,
    k=20
)

for hit in results.hits:
    print(hit.id, hit.payload["path"], hit.payload["price"])

The client handles the JSON serialization, while the engine processes the path field condition through the FilterOpBase hierarchy in src/index/detail/scalar/filter/filter_ops.h and filter_ops.cpp.

C++ Engine Integration (Advanced)

For developers extending the OpenViking engine directly, the filter operations are implemented in the FilterOpBase hierarchy. The PathFieldOp (accessed via make_filter_op_by_opname("must")) interprets the filter JSON and delegates to make_path_field_copy.

#include "filter_ops.h"
#include "bitmap_field_group.h"

using namespace vectordb;

// Construct the path condition
FilterOpBasePtr path_cond = make_filter_op_by_opname("must");
JsonDoc json; // Populate with field, conds, and para
path_cond->load_json_doc(json);

// Generate bitmap
BitmapPtr result = field_group_set_ptr->make_path_field_copy(
    "path",
    {"/project/docs"},
    1  // depth parsed from para
);

The resulting BitmapPtr integrates with LogicOpBase for AND/OR composition with other scalar or vector filters.

Summary

• DirIndex maintains a Trie of path keys in src/index/detail/scalar/bitmap_holder/dir_index.cpp, enabling efficient prefix-based retrieval.
• make_path_field_copy in src/index/detail/scalar/bitmap_holder/bitmap_field_group.cpp merges bitmaps for hierarchical queries using FastUnion.
• Depth control is managed via the para parameter (-d=N) parsed by parse_dir_semantic_para in src/index/detail/scalar/filter/filter_ops.cpp, supporting unbounded (-1) or limited (0-50) depth.
• Filter integration occurs through the FilterOpBase and LogicOpBase classes, allowing path conditions to combine with numeric ranges and vector similarity search.
• Performance remains sub-millisecond for multi-million-record datasets due to the compact Trie structure and bitmap operations.

Frequently Asked Questions

What is the maximum depth supported for cross-directory relational queries?

The OpenViking engine clamps depth values to the range [-1, 50] in parse_dir_semantic_para within src/index/detail/scalar/filter/filter_ops.cpp. A value of -1 indicates unbounded depth (all descendants), while values 0 through 50 restrict traversal to that many levels below the specified prefix.

How does the DirIndex structure improve query performance?

The Trie (prefix tree) structure in DirIndex stores path segments rather than full strings, enabling O(L) lookup time where L is the path length. During get_merged_bitmap, the engine traverses only the relevant branch and collects pre-computed bitmaps for leaf nodes, avoiding full collection scans.

Can cross-directory filters be combined with vector similarity search?

Yes. Cross-directory relational queries generate standard bitmaps through make_path_field_copy that integrate with the LogicOpBase system. These bitmaps can be ANDed or ORed with vector search results, numeric ranges, or tag filters in the same query expression.

What happens if the para parameter is omitted from a path field query?

If the para field is omitted, the depth defaults to -1 (unbounded), meaning the query retrieves all records matching the specified prefix and any number of sub-directory levels. This behavior is handled in the FilterOpBase implementation when no depth override is provided.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}