# How the Claude-Obsidian Query Workflow Works: BM25 and Neural Reranking Explained

> Discover the Claude-Obsidian query workflow. Learn how BM25 indexing and neural reranking retrieve answers from your vault without file modification.

- Repository: [Agrici.Daniel/claude-obsidian](https://github.com/AgriciDaniel/claude-obsidian)
- Tags: deep-dive
- Published: 2026-08-28

---

**Claude-Obsidian executes a read-only retrieval pipeline that combines sparse BM25 indexing with optional neural reranking to answer vault-scoped questions without modifying any files.**

The query workflow in Claude-Obsidian enables semantic search across your Obsidian vault using a hybrid retrieval architecture. This open-source tool implements a deterministic, multi-stage pipeline that leverages classical information retrieval techniques alongside modern embedding models. Understanding how the query workflow operates is essential for customizing retrieval behavior and debugging indexing issues.

## Vault Resolution and Index Loading

The pipeline begins by establishing the vault context and loading the inverted index. In [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py), the `configure_vault()` function first attempts to locate the target vault using `claude_obsidian.paths.resolve_vault_root`. If the caller provides `--vault <path>`, that path is used directly; otherwise, the tool resolves the nearest vault to prevent accidental queries against the plugin directory itself.

Once the vault is identified, [`retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/retrieve.py) imports the sibling module [`bm25-index.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/bm25-index.py) via `import_sibling("bm25_index", "bm25-index.py")` and loads the BM25 index from [`.vault-meta/bm25/index.json`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/.vault-meta/bm25/index.json) using `bm25.load_index()`. If the index is missing or corrupted, the pipeline aborts immediately with **exit code 10**, forcing the caller to fall back to the legacy hot-file query path.

## BM25 Retrieval and Scoring

With the index loaded, the `bm25.query(query, top_k=BM25_TOP)` function tokenizes the input using `bm25-index.py.tokenize` and scores each document chunk using the **Okapi BM25 formula** with parameters `k1=1.5` and `b=0.75`. By default, the system retrieves the top 20 candidate chunks, returning entries structured as `{chunk_id, score, path}`.

This sparse retrieval stage operates entirely within [`scripts/bm25-index.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/bm25-index.py), which implements the inverted index structure and term frequency calculations. The BM25 stage provides fast, lexically grounded candidate selection without requiring GPU resources or external API calls.

## Chunk Validation and Integrity Checks

Once BM25 identifies candidate chunks, [`retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/retrieve.py) performs rigorous validation to ensure data integrity. For each candidate, the script resolves the chunk file in `.vault-meta/chunks/` and executes `chunk_is_current()` to verify that:

- The chunk's body hash matches the current content
- The source page's hash remains current

Stale or unreadable chunks are silently discarded during this phase. This validation loop appears in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) lines 94-108, ensuring that retrieval results always reflect the actual vault state and not orphaned index entries.

## Optional Neural Reranking with Ollama

If the `--no-rerank` flag is **not** provided, the pipeline loads [`scripts/rerank.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/rerank.py) as a sibling helper module. The `rerank.rerank(query, candidates, …)` function computes cosine similarity between the query embedding and each candidate's pre-computed embedding using a local Ollama model.

If the Ollama model cannot be loaded or encounters runtime errors, the reranker gracefully degrades to a no-op, preserving the original BM25 ordering. This defensive design ensures that network issues or missing local models never cause total pipeline failure. The rerank import and conditional execution logic resides in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) lines 48-55 and 29-34.

## Deduplication and Final Output Assembly

Following the ranking phase—whether BM25-only or BM25-plus-rerank—the pipeline deduplicates candidates that belong to the same source page, keeping only the highest-scoring entry per page. In [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) lines 38-48, the final list is sorted by rerank score (or BM25 score when reranking is skipped).

The output assembly stage constructs a JSON object containing:

- The original query string
- The chosen strategy identifier (`bm25` or `bm25_rerank`)
- The requested `top_k` count
- Selected candidates with `chunk_id`, `page_path`, `absolute_path`, scores, and a 200-character text snippet

When `--explain` is passed, diagnostic metadata such as BM25 hit counts and stale-candidate counts are appended. The final JSON is printed to `stdout` via `print(json.dumps(...))` in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) lines 48-66.

## Command-Line Usage Examples

Execute a standard query with BM25 and neural reranking, returning the top 5 results:

```bash
python3 scripts/retrieve.py "What is the role of the LLM in Claude-Obsidian?" \
  --top 5

```

Run a BM25-only query with verbose diagnostics and no semantic reranking:

```bash
python3 scripts/retrieve.py "Explain vault-scoped query workflow" \
  --top 10 --no-rerank --explain

```

Specify a custom vault path explicitly:

```bash
python3 scripts/retrieve.py "How does chunk validation work?" \
  --vault /absolute/path/to/vault \
  --top 3

```

## Python API Integration

You can import the retrieval logic directly into Python applications to avoid subprocess overhead:

```python
from scripts.retrieve import main as retrieve

# Emulate a CLI call with custom parameters

exit_code = retrieve([
    "--vault", "/path/to/vault",
    "How does the query workflow work?",
    "--top", "3",
    "--no-rerank"
])
assert exit_code == 0

```

For direct BM25 score inspection without the full pipeline:

```python
from scripts.bm25_index import query as bm25_query

hits = bm25_query("query workflow", top_k=5)
for hit in hits:
    print(f"Chunk: {hit['chunk_id']}, Score: {hit['score']:.4f}")

```

Access vault resolution utilities programmatically:

```python
from claude_obsidian.paths import resolve_vault_root

vault_path = resolve_vault_root(starting_path="/some/directory")
print(f"Resolved vault: {vault_path}")

```

## Summary

- **Read-only guarantee**: The query workflow never mutates vault contents, rebuilds indices, or writes log files to disk.
- **Hybrid retrieval**: Combines sparse BM25 retrieval (`k1=1.5`, `b=0.75`) with optional neural reranking via Ollama embeddings.
- **Defensive error handling**: Missing indices trigger exit code 10; reranker failures gracefully fall back to BM25 ordering.
- **Integrity validation**: All candidates pass `chunk_is_current()` checks to ensure hashes match source files.
- **Flexible interfaces**: Supports both CLI arguments (`--vault`, `--top`, `--no-rerank`, `--explain`) and direct Python module imports.

## Frequently Asked Questions

### Is the Claude-Obsidian query workflow truly read-only?

Yes. According to the source code in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) and [`claude_obsidian/transaction.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/claude_obsidian/transaction.py), the pipeline uses safety helpers like `_safe_vault_path` and `_atomic_vault_write` to guarantee it never modifies vault files. The workflow only reads from [`.vault-meta/bm25/index.json`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/.vault-meta/bm25/index.json) and chunk files in `.vault-meta/chunks/`.

### What happens if the BM25 index is missing or corrupted?

The pipeline aborts immediately with **exit code 10**. This occurs in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) when `bm25.load_index()` fails to locate [`.vault-meta/bm25/index.json`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/.vault-meta/bm25/index.json) or encounters corruption. The caller must then regenerate the index using the legacy hot-file query path or rebuild the BM25 index separately.

### Can I disable the neural reranking stage?

Yes. Pass the `--no-rerank` flag to [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py). When this flag is present, the pipeline skips importing [`scripts/rerank.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/rerank.py) and sorts results purely by BM25 scores. This is useful for offline environments without Ollama or when you need faster retrieval without embedding computation overhead.

### How does the vault resolution logic prevent querying the wrong directory?

The `configure_vault()` function in [`scripts/retrieve.py`](https://github.com/AgriciDaniel/claude-obsidian/blob/main/scripts/retrieve.py) uses `claude_obsidian.paths.resolve_vault_root` to locate the nearest vault boundary. If no `--vault` argument is provided, it traverses upward from the current working directory to find the vault root, ensuring the query never executes against the plugin installation directory or system folders outside the intended vault scope.