How the Claude-Obsidian Query Workflow Works: BM25 and Neural Reranking Explained

Claude-Obsidian executes a read-only retrieval pipeline that combines sparse BM25 indexing with optional neural reranking to answer vault-scoped questions without modifying any files.

The query workflow in Claude-Obsidian enables semantic search across your Obsidian vault using a hybrid retrieval architecture. This open-source tool implements a deterministic, multi-stage pipeline that leverages classical information retrieval techniques alongside modern embedding models. Understanding how the query workflow operates is essential for customizing retrieval behavior and debugging indexing issues.

Vault Resolution and Index Loading

The pipeline begins by establishing the vault context and loading the inverted index. In scripts/retrieve.py, the configure_vault() function first attempts to locate the target vault using claude_obsidian.paths.resolve_vault_root. If the caller provides --vault <path>, that path is used directly; otherwise, the tool resolves the nearest vault to prevent accidental queries against the plugin directory itself.

Once the vault is identified, retrieve.py imports the sibling module bm25-index.py via import_sibling("bm25_index", "bm25-index.py") and loads the BM25 index from .vault-meta/bm25/index.json using bm25.load_index(). If the index is missing or corrupted, the pipeline aborts immediately with exit code 10, forcing the caller to fall back to the legacy hot-file query path.

BM25 Retrieval and Scoring

With the index loaded, the bm25.query(query, top_k=BM25_TOP) function tokenizes the input using bm25-index.py.tokenize and scores each document chunk using the Okapi BM25 formula with parameters k1=1.5 and b=0.75. By default, the system retrieves the top 20 candidate chunks, returning entries structured as {chunk_id, score, path}.

This sparse retrieval stage operates entirely within scripts/bm25-index.py, which implements the inverted index structure and term frequency calculations. The BM25 stage provides fast, lexically grounded candidate selection without requiring GPU resources or external API calls.

Chunk Validation and Integrity Checks

Once BM25 identifies candidate chunks, retrieve.py performs rigorous validation to ensure data integrity. For each candidate, the script resolves the chunk file in .vault-meta/chunks/ and executes chunk_is_current() to verify that:

• The chunk's body hash matches the current content
• The source page's hash remains current

Stale or unreadable chunks are silently discarded during this phase. This validation loop appears in scripts/retrieve.py lines 94-108, ensuring that retrieval results always reflect the actual vault state and not orphaned index entries.

Optional Neural Reranking with Ollama

If the --no-rerank flag is not provided, the pipeline loads scripts/rerank.py as a sibling helper module. The rerank.rerank(query, candidates, …) function computes cosine similarity between the query embedding and each candidate's pre-computed embedding using a local Ollama model.

If the Ollama model cannot be loaded or encounters runtime errors, the reranker gracefully degrades to a no-op, preserving the original BM25 ordering. This defensive design ensures that network issues or missing local models never cause total pipeline failure. The rerank import and conditional execution logic resides in scripts/retrieve.py lines 48-55 and 29-34.

Deduplication and Final Output Assembly

Following the ranking phase—whether BM25-only or BM25-plus-rerank—the pipeline deduplicates candidates that belong to the same source page, keeping only the highest-scoring entry per page. In scripts/retrieve.py lines 38-48, the final list is sorted by rerank score (or BM25 score when reranking is skipped).

The output assembly stage constructs a JSON object containing:

• The original query string
• The chosen strategy identifier (bm25 or bm25_rerank)
• The requested top_k count
• Selected candidates with chunk_id, page_path, absolute_path, scores, and a 200-character text snippet

When --explain is passed, diagnostic metadata such as BM25 hit counts and stale-candidate counts are appended. The final JSON is printed to stdout via print(json.dumps(...)) in scripts/retrieve.py lines 48-66.

Command-Line Usage Examples

Execute a standard query with BM25 and neural reranking, returning the top 5 results:

python3 scripts/retrieve.py "What is the role of the LLM in Claude-Obsidian?" \
  --top 5

Run a BM25-only query with verbose diagnostics and no semantic reranking:

python3 scripts/retrieve.py "Explain vault-scoped query workflow" \
  --top 10 --no-rerank --explain

Specify a custom vault path explicitly:

python3 scripts/retrieve.py "How does chunk validation work?" \
  --vault /absolute/path/to/vault \
  --top 3

Python API Integration

You can import the retrieval logic directly into Python applications to avoid subprocess overhead:

from scripts.retrieve import main as retrieve

# Emulate a CLI call with custom parameters

exit_code = retrieve([
    "--vault", "/path/to/vault",
    "How does the query workflow work?",
    "--top", "3",
    "--no-rerank"
])
assert exit_code == 0

For direct BM25 score inspection without the full pipeline:

from scripts.bm25_index import query as bm25_query

hits = bm25_query("query workflow", top_k=5)
for hit in hits:
    print(f"Chunk: {hit['chunk_id']}, Score: {hit['score']:.4f}")

Access vault resolution utilities programmatically:

from claude_obsidian.paths import resolve_vault_root

vault_path = resolve_vault_root(starting_path="/some/directory")
print(f"Resolved vault: {vault_path}")

Summary

• Read-only guarantee: The query workflow never mutates vault contents, rebuilds indices, or writes log files to disk.
• Hybrid retrieval: Combines sparse BM25 retrieval (k1=1.5, b=0.75) with optional neural reranking via Ollama embeddings.
• Defensive error handling: Missing indices trigger exit code 10; reranker failures gracefully fall back to BM25 ordering.
• Integrity validation: All candidates pass chunk_is_current() checks to ensure hashes match source files.
• Flexible interfaces: Supports both CLI arguments (--vault, --top, --no-rerank, --explain) and direct Python module imports.

Frequently Asked Questions

Is the Claude-Obsidian query workflow truly read-only?

Yes. According to the source code in scripts/retrieve.py and claude_obsidian/transaction.py, the pipeline uses safety helpers like _safe_vault_path and _atomic_vault_write to guarantee it never modifies vault files. The workflow only reads from .vault-meta/bm25/index.json and chunk files in .vault-meta/chunks/.

What happens if the BM25 index is missing or corrupted?

The pipeline aborts immediately with exit code 10. This occurs in scripts/retrieve.py when bm25.load_index() fails to locate .vault-meta/bm25/index.json or encounters corruption. The caller must then regenerate the index using the legacy hot-file query path or rebuild the BM25 index separately.

Can I disable the neural reranking stage?

Yes. Pass the --no-rerank flag to scripts/retrieve.py. When this flag is present, the pipeline skips importing scripts/rerank.py and sorts results purely by BM25 scores. This is useful for offline environments without Ollama or when you need faster retrieval without embedding computation overhead.

How does the vault resolution logic prevent querying the wrong directory?

The configure_vault() function in scripts/retrieve.py uses claude_obsidian.paths.resolve_vault_root to locate the nearest vault boundary. If no --vault argument is provided, it traverses upward from the current working directory to find the vault root, ensuring the query never executes against the plugin installation directory or system folders outside the intended vault scope.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}