# How to Use BM25 Retrieval for Session History Search in DeepSeek-Reasonix

> Learn to use BM25 retrieval for session history search in DeepSeek-Reasonix. Rank past messages and retrieve full transcript context using the pure-Go BM25 engine.

- Repository: [YHH/DeepSeek-Reasonix](https://github.com/esengine/DeepSeek-Reasonix)
- Tags: how-to-guide
- Published: 2026-06-24

---

**Invoke the read-only `history` tool with `operation: "search"` to rank past session messages using the pure-Go BM25 engine, then use `operation: "around"` to retrieve full transcript context from specific hits.**

DeepSeek-Reasonix implements a self-contained **BM25 retrieval** system for searching agent session history without external dependencies like SQLite, CGO, or vector databases. The system stores every agent session as a line-delimited JSON file (one message per line) and exposes retrieval functionality through a read-only `history` tool that ranks messages by relevance using the classic BM25 scoring formula.

## How BM25 Retrieval Works in DeepSeek-Reasonix

The retrieval architecture isolates search functionality from the prompt-generation pipeline, keeping the system prompt immutable while fetching historical context on demand. This design maintains a prefix cache-first approach for fast agent reasoning.

### Core Architecture and File Structure

The BM25 engine lives entirely within the repository's Go source code. The primary implementation resides in [`internal/retrieval/bm25.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/retrieval/bm25.go), which provides tokenization, scoring, and snippet generation without external libraries. The [`internal/history/search.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/search.go) file contains the `Searcher` implementation that extracts documents from session files based on requested message `Kind`s (user text, assistant text, tool input, or tool error).

The tool interface defined in [`internal/history/tool.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/tool.go) exposes two operations:

- **`search`**: Ranks saved messages using BM25 scoring and returns relevance-ranked snippets
- **`around`**: Returns surrounding transcript context given a specific `session_path` and `message_index`

### Tokenization and Scoring Pipeline

The retrieval process follows a five-stage pipeline implemented in the source code:

1. **Query Processing**: The `retrieval.QueryTerms` function tokenizes search queries by lower-casing Latin words and splitting CJK (Chinese, Japanese, Korean) characters into single-rune terms.

2. **Document Indexing**: For each session file, the `Searcher` extracts documents matching the requested `Kind`s, storing term-frequency maps (`Counts`) and document lengths.

3. **BM25 Scoring**: The `retrieval.BM25Score` function computes relevance using the standard parameters **k1 = 1.2** and **b = 0.75**, balancing term frequency and document length normalization.

4. **Relative Score Trimming**: The `KeepTopRelativeScore` function filters results, dropping hits that fall below **0.15 × topScore** to ensure only highly relevant matches are returned.

5. **Snippet Generation**: The `retrieval.MakeSnippet` function creates compact excerpts centered on query terms, limited to **240 runes** for concise display.

## Using the History Tool for Session Search

The `history` tool accepts JSON payloads specifying the operation, query parameters, and result limits. All operations are read-only and scoped to configured session directories for security.

### Searching Session History

To perform a BM25 search across session files, construct a payload targeting the `search` operation:

```json
{
  "operation": "search",
  "query": "failed to connect to database",
  "scope": "project",
  "kind": ["user_text", "assistant_text", "tool_input", "tool_error"],
  "limit": 5
}

```

The `scope` parameter accepts `"project"` (current session directory only) or `"global"` (all accessible sessions). The `kind` array filters which message types to index—include `tool_error` to capture exception traces or `user_text` for natural language queries.

The tool returns a formatted string containing numbered hits with BM25 scores, session IDs, message indices, and 240-rune snippets:

```

History search results for "failed to connect to database":

1. score=0.842 source=stdout session_id=20230615-abc123 message_index=42 kind=user_text
   session_path: /home/user/.reasonix/sessions/20230615-abc123.jsonl
   snippet: ...the error "failed to connect to database" occurred when ...

2. score=0.761 source=stdout session_id=20230610-def456 message_index=17 kind=tool_error
   session_path: /home/user/.reasonix/sessions/20230610-def456.jsonl
   snippet: ...tool error: failed to connect to database (timeout)...

```

### Retrieving Surrounding Context

After identifying a relevant hit, use the `around` operation to fetch the full conversation context. This operation accepts `session_path` and `message_index` from a search result and returns the surrounding transcript:

```json
{
  "operation": "around",
  "session_path": "/home/user/.reasonix/sessions/20230615-abc123.jsonl",
  "message_index": 42,
  "before": 3,
  "after": 3
}

```

The `around` operation includes security safeguards implemented in [`internal/history/tool.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/tool.go) that validate the `session_path` falls under the configured session or archive roots, preventing arbitrary file system access. The default context window retrieves ±3 messages, customizable via the `before` and `after` parameters.

## Programmatic Integration in Go

For internal development or custom tooling, instantiate the history tool directly using the Go API:

```go
import (
    "context"
    "encoding/json"
    "fmt"
    "reasonix/internal/history"
)

func searchSessionHistory() {
    // Configure tool options with session directories
    tool := history.NewTool(history.Options{
        SessionDir:       "/home/user/.reasonix/sessions",
        GlobalSessionDir: "/home/user/.reasonix",
        ArchiveDir:       "/home/user/.reasonix/.archive",
    })

    // Build search request
    args := map[string]any{
        "operation": "search",
        "query":     "authentication token",
        "scope":     "global",
        "kind":      []string{"user_text", "assistant_text"},
        "limit":     3,
    }
    raw, _ := json.Marshal(args)

    // Execute the tool
    out, err := tool.Execute(context.Background(), raw)
    if err != nil {
        panic(err)
    }
    fmt.Println(out)
}

```

This code path mirrors the controller execution when the LLM invokes the tool, ensuring consistent behavior between programmatic access and agent-driven queries.

## Summary

- **BM25 retrieval** in DeepSeek-Reasonix uses a pure-Go implementation with parameters k1=1.2 and b=0.75, requiring no external databases or vector stores.
- The **`search`** operation tokenizes queries (handling both Latin and CJK text), scores documents, and returns ranked snippets with relevance scores above the 0.15 relative threshold.
- The **`around`** operation retrieves conversational context from specific hits with security-validated path constraints.
- Key source files include [`internal/retrieval/bm25.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/retrieval/bm25.go) for the scoring engine, [`internal/history/search.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/search.go) for document extraction, and [`internal/history/tool.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/tool.go) for the public API.

## Frequently Asked Questions

### What makes BM25 suitable for session history search?

BM25 provides probabilistic relevance scoring that accounts for term frequency saturation and document length normalization, making it ideal for conversational data where message lengths vary significantly. Unlike vector similarity search, BM25 requires no embedding model or GPU acceleration, allowing DeepSeek-Reasonix to run entirely with Go standard libraries.

### How does the tokenizer handle multilingual queries?

The `retrieval.QueryTerms` function in [`internal/retrieval/bm25.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/retrieval/bm25.go) applies different strategies based on character type: Latin text is converted to lowercase, while CJK characters are split into individual runes. This ensures that searches for Chinese or Japanese terms match precisely without requiring language-specific stemmers.

### Why are some search results filtered out despite having scores?

After initial ranking, the system applies `KeepTopRelativeScore` to discard results falling below 15% of the top score. This pruning prevents low-relevance matches from cluttering results while preserving the most significant hits, ensuring the agent receives only meaningful historical context.

### Can the `around` operation access any file on the system?

No. The `around` implementation in [`internal/history/tool.go`](https://github.com/esengine/DeepSeek-Reasonix/blob/main/internal/history/tool.go) validates that requested `session_path` values reside within the configured `SessionDir`, `GlobalSessionDir`, or `ArchiveDir` roots. This sandboxing prevents path traversal attacks and ensures the tool remains read-only and scope-limited.