Profiling and Debugging Tools in zvec: A Complete Guide to Performance Analysis

The zvec vector database provides built-in stage-level profiling via zvec::Profiler, RAII-based timing helpers, pluggable logging macros, multi-threaded benchmarking utilities, and code coverage scripts to diagnose performance bottlenecks throughout the query lifecycle.

The Alibaba zvec repository ships with a comprehensive, self-contained toolkit for profiling and debugging vector search operations. These integrated utilities allow developers to measure query execution latency at granular stages, monitor index-level operations, and validate performance optimizations without relying on external instrumentation.

Stage-Level Profiling with zvec::Profiler

The primary profiling interface in zvec is the zvec::Profiler class, defined in src/db/common/profiler.h. This lightweight collector records named execution stages and aggregates their timings in seconds.

Core Profiler API

The profiler exposes three main methods for manual instrumentation: open_stage(), close_stage(), and add(). You instantiate a profiler and bracket code sections to capture precise timing data.

#include "zvec/db/common/profiler.h"
#include <iostream>
#include <memory>

auto profiler = std::make_shared<zvec::Profiler>(true);  // true enables the profiler
profiler->open_stage("parse query");
// ... query parsing logic ...
profiler->close_stage();

profiler->open_stage("vector search");
// ... search execution ...
profiler->close_stage();

std::cout << profiler->display();  // outputs a formatted timing table

The display() method renders a human-readable table showing each stage's duration, making it easy to identify bottlenecks in complex query pipelines.

RAII Timing with ScopedLatency

For exception-safe profiling, zvec provides the ScopedLatency helper class (lines 176-186 in src/db/common/profiler.h). This RAII wrapper automatically records timing when the object goes out of scope, eliminating the risk of missing close_stage() calls during early returns or exceptions.

{
    zvec::ScopedLatency timer(profiler.get(), "automatic_stage");
    // ... code to time ...
}  // timing automatically recorded here

SQL Engine Integration

The SQL engine implementation in src/db/sqlengine/sqlengine_impl.cc (lines 125-188) demonstrates production-grade profiler usage. The engine automatically opens specific stages during query processing: "analyze stage", "message_to_sqlinfo", and "plan stage". This integration provides out-of-the-box visibility into query lifecycle performance without requiring manual instrumentation of the engine internals.

Index-Level Performance Tracking

Beyond the high-level query profiler, zvec exposes a separate profiler struct for low-level index operations via IndexContext.

The IndexContext Profiler

Defined in src/include/zvec/core/framework/index_context.h (lines 29-52), zvec::core::Profiler is a lightweight map-based accumulator used by index implementations to record per-operation metrics. Index developers can inject timing data directly into the search context:

context->profiler().add("search", elapsed_seconds);

This collected data can be printed using display(), as referenced in the recall testing tool at tools/core/recall.cc (line 679), enabling fine-grained analysis of index-specific performance characteristics like graph traversal or quantization overhead.

Logging Framework for Debug Output

zvec implements a flexible, macro-based logging system to aid debugging without impacting release performance.

Macro-Based Logging

The logging interface resides in src/include/zvec/ailego/logger/logger.h (lines 39-66) and provides severity-level macros: LOG_DEBUG, LOG_INFO, LOG_WARN, LOG_ERROR, and LOG_FATAL. The underlying zvec::ailego::Logger class is pluggable; developers can register custom log sinks via FACTORY_REGISTER_LOGGER macros (lines 22-30) to route output to files, consoles, or remote aggregation systems.

Runtime Log Level Control

Debug output can be enabled dynamically without recompilation by setting the global log level through the broker:

zvec::ailego::LoggerBroker::SetLevel(zvec::ailego::Logger::LEVEL_DEBUG);

This activates verbose LOG_DEBUG statements scattered throughout the codebase, such as diagnostic output for use_key_info_map_ variables, providing immediate insight into internal state transitions during query execution.

Benchmarking and Testing Utilities

zvec includes standalone tools for reproducible performance testing and quality assurance.

Multi-Threaded Benchmark Driver

The tools/core/bench.cc executable provides a configurable, multi-threaded benchmark harness for vector search and index construction. It accepts YAML configuration files specifying index types, query datasets, thread counts, and batch sizes. The template class Bench<T> utilizes BenchResult (defined in tools/core/bench_result.h) to record per-batch latency statistics and generate summarized performance reports.

cd tools/core
./bench CONFIG.yaml  // executes benchmark and prints latency percentiles

Code Coverage Analysis

For debugging test quality, the repository includes scripts/gcov.sh, which wraps lcov and genhtml to generate HTML coverage reports. The script filters out test and third-party sources to focus on core library coverage.

# After building with -fprofile-arcs -ftest-coverage

./scripts/gcov.sh -p zvec  // generates html/coverage.html

Profiling Workflow Integration

To standardize performance investigations, zvec provides a GitHub issue template specifically for profiling reports (.github/ISSUE_TEMPLATE/profiling.yml). This template prompts contributors to document hardware specifications, profiling goals, and attach relevant benchmark outputs or timing logs, ensuring consistent context for performance-related bug reports.

End-to-End Profiling Example

The following example demonstrates wiring the profiler with the SQL engine, enabling debug logging, and retrieving a comprehensive timing report:

#include "zvec/db/common/profiler.h"
#include "zvec/db/sqlengine/sqlengine.h"
#include "zvec/ailego/logger/logger.h"
#include <iostream>
#include <memory>

int main() {
    // Enable verbose debug output
    zvec::ailego::LoggerBroker::SetLevel(zvec::ailego::Logger::LEVEL_DEBUG);
    
    // Initialize and enable profiler
    auto prof = std::make_shared<zvec::Profiler>(true);
    prof->start();  // begin root timing stage
    
    // Execute query with automatic stage instrumentation
    auto engine = zvec::sqlengine::SQLEngine::create(prof);
    engine->run("SELECT * FROM vectors WHERE id > 100");
    
    prof->stop();  // finalize root stage
    std::cout << prof->display();  // print timing breakdown
    
    return 0;
}

Running this produces output similar to:

================================================================
analyze stage: 0.012345 s
message_to_sqlinfo: 0.003210 s
plan stage: 0.001234 s
================================================================

Summary

• Stage-Level Profiler: Use zvec::Profiler in src/db/common/profiler.h with open_stage()/close_stage() to measure query pipeline segments, or employ ScopedLatency for automatic RAII-based timing.
• Index Context Tracking: Leverage zvec::core::Profiler via IndexContext in src/include/zvec/core/framework/index_context.h for granular index operation metrics.
• Debug Logging: Control verbose output dynamically using LOG_DEBUG macros and LoggerBroker::SetLevel() without recompiling.
• Benchmarking: Run standardized performance tests using tools/core/bench.cc with YAML configurations to measure latency percentiles under concurrent load.
• Coverage Analysis: Generate HTML coverage reports via scripts/gcov.sh to ensure test suite comprehensiveness.

Frequently Asked Questions

How do I enable profiling in a production zvec deployment?

Enable the profiler by passing true to the zvec::Profiler constructor and calling start() before query execution. The profiler adds minimal overhead since it only records high-level stage boundaries. For production systems, consider sampling or conditional compilation to disable profiling in hot paths, though the implementation in src/db/common/profiler.h is designed to be lightweight enough for continuous monitoring.

Can I add custom timing stages to my index implementation?

Yes. Inside your index code, access the profiler through the IndexContext pointer: context->profiler().add("my_custom_stage", elapsed_time). This integrates with the same display mechanism used by the SQL engine profiler, allowing your custom index timings to appear alongside standard query stage metrics.

What is the difference between zvec::Profiler and zvec::core::Profiler?

zvec::Profiler (defined in src/db/common/profiler.h) is the high-level, stage-based profiler used by the SQL engine to track query lifecycle phases. zvec::core::Profiler (defined in src/include/zvec/core/framework/index_context.h) is a simpler map-based struct designed for index developers to accumulate arbitrary timing key-value pairs during low-level vector operations like distance calculations or graph traversals.

How do I generate a code coverage report for my zvec modifications?

First, compile the project with GCC flags -fprofile-arcs -ftest-coverage. Then execute your test suite and run ./scripts/gcov.sh -p zvec. This generates an HTML report in html/coverage.html excluding third-party and test files, allowing you to identify untested code paths in your changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}