How to Use BM25 Retrieval for Session History Search in DeepSeek-Reasonix

Invoke the read-only history tool with operation: "search" to rank past session messages using the pure-Go BM25 engine, then use operation: "around" to retrieve full transcript context from specific hits.

DeepSeek-Reasonix implements a self-contained BM25 retrieval system for searching agent session history without external dependencies like SQLite, CGO, or vector databases. The system stores every agent session as a line-delimited JSON file (one message per line) and exposes retrieval functionality through a read-only history tool that ranks messages by relevance using the classic BM25 scoring formula.

How BM25 Retrieval Works in DeepSeek-Reasonix

The retrieval architecture isolates search functionality from the prompt-generation pipeline, keeping the system prompt immutable while fetching historical context on demand. This design maintains a prefix cache-first approach for fast agent reasoning.

Core Architecture and File Structure

The BM25 engine lives entirely within the repository's Go source code. The primary implementation resides in internal/retrieval/bm25.go, which provides tokenization, scoring, and snippet generation without external libraries. The internal/history/search.go file contains the Searcher implementation that extracts documents from session files based on requested message Kinds (user text, assistant text, tool input, or tool error).

The tool interface defined in internal/history/tool.go exposes two operations:

• search: Ranks saved messages using BM25 scoring and returns relevance-ranked snippets
• around: Returns surrounding transcript context given a specific session_path and message_index

Tokenization and Scoring Pipeline

The retrieval process follows a five-stage pipeline implemented in the source code:

1. Query Processing: The retrieval.QueryTerms function tokenizes search queries by lower-casing Latin words and splitting CJK (Chinese, Japanese, Korean) characters into single-rune terms.

2. Document Indexing: For each session file, the Searcher extracts documents matching the requested Kinds, storing term-frequency maps (Counts) and document lengths.

3. BM25 Scoring: The retrieval.BM25Score function computes relevance using the standard parameters k1 = 1.2 and b = 0.75, balancing term frequency and document length normalization.

4. Relative Score Trimming: The KeepTopRelativeScore function filters results, dropping hits that fall below 0.15 × topScore to ensure only highly relevant matches are returned.

5. Snippet Generation: The retrieval.MakeSnippet function creates compact excerpts centered on query terms, limited to 240 runes for concise display.

Using the History Tool for Session Search

The history tool accepts JSON payloads specifying the operation, query parameters, and result limits. All operations are read-only and scoped to configured session directories for security.

Searching Session History

To perform a BM25 search across session files, construct a payload targeting the search operation:

{
  "operation": "search",
  "query": "failed to connect to database",
  "scope": "project",
  "kind": ["user_text", "assistant_text", "tool_input", "tool_error"],
  "limit": 5
}

The scope parameter accepts "project" (current session directory only) or "global" (all accessible sessions). The kind array filters which message types to index—include tool_error to capture exception traces or user_text for natural language queries.

The tool returns a formatted string containing numbered hits with BM25 scores, session IDs, message indices, and 240-rune snippets:

History search results for "failed to connect to database":

1. score=0.842 source=stdout session_id=20230615-abc123 message_index=42 kind=user_text
   session_path: /home/user/.reasonix/sessions/20230615-abc123.jsonl
   snippet: ...the error "failed to connect to database" occurred when ...

2. score=0.761 source=stdout session_id=20230610-def456 message_index=17 kind=tool_error
   session_path: /home/user/.reasonix/sessions/20230610-def456.jsonl
   snippet: ...tool error: failed to connect to database (timeout)...

Retrieving Surrounding Context

After identifying a relevant hit, use the around operation to fetch the full conversation context. This operation accepts session_path and message_index from a search result and returns the surrounding transcript:

{
  "operation": "around",
  "session_path": "/home/user/.reasonix/sessions/20230615-abc123.jsonl",
  "message_index": 42,
  "before": 3,
  "after": 3
}

The around operation includes security safeguards implemented in internal/history/tool.go that validate the session_path falls under the configured session or archive roots, preventing arbitrary file system access. The default context window retrieves ±3 messages, customizable via the before and after parameters.

Programmatic Integration in Go

For internal development or custom tooling, instantiate the history tool directly using the Go API:

import (
    "context"
    "encoding/json"
    "fmt"
    "reasonix/internal/history"
)

func searchSessionHistory() {
    // Configure tool options with session directories
    tool := history.NewTool(history.Options{
        SessionDir:       "/home/user/.reasonix/sessions",
        GlobalSessionDir: "/home/user/.reasonix",
        ArchiveDir:       "/home/user/.reasonix/.archive",
    })

    // Build search request
    args := map[string]any{
        "operation": "search",
        "query":     "authentication token",
        "scope":     "global",
        "kind":      []string{"user_text", "assistant_text"},
        "limit":     3,
    }
    raw, _ := json.Marshal(args)

    // Execute the tool
    out, err := tool.Execute(context.Background(), raw)
    if err != nil {
        panic(err)
    }
    fmt.Println(out)
}

This code path mirrors the controller execution when the LLM invokes the tool, ensuring consistent behavior between programmatic access and agent-driven queries.

Summary

• BM25 retrieval in DeepSeek-Reasonix uses a pure-Go implementation with parameters k1=1.2 and b=0.75, requiring no external databases or vector stores.
• The search operation tokenizes queries (handling both Latin and CJK text), scores documents, and returns ranked snippets with relevance scores above the 0.15 relative threshold.
• The around operation retrieves conversational context from specific hits with security-validated path constraints.
• Key source files include internal/retrieval/bm25.go for the scoring engine, internal/history/search.go for document extraction, and internal/history/tool.go for the public API.

Frequently Asked Questions

What makes BM25 suitable for session history search?

BM25 provides probabilistic relevance scoring that accounts for term frequency saturation and document length normalization, making it ideal for conversational data where message lengths vary significantly. Unlike vector similarity search, BM25 requires no embedding model or GPU acceleration, allowing DeepSeek-Reasonix to run entirely with Go standard libraries.

How does the tokenizer handle multilingual queries?

The retrieval.QueryTerms function in internal/retrieval/bm25.go applies different strategies based on character type: Latin text is converted to lowercase, while CJK characters are split into individual runes. This ensures that searches for Chinese or Japanese terms match precisely without requiring language-specific stemmers.

Why are some search results filtered out despite having scores?

After initial ranking, the system applies KeepTopRelativeScore to discard results falling below 15% of the top score. This pruning prevents low-relevance matches from cluttering results while preserving the most significant hits, ensuring the agent receives only meaningful historical context.

Can the around operation access any file on the system?

No. The around implementation in internal/history/tool.go validates that requested session_path values reside within the configured SessionDir, GlobalSessionDir, or ArchiveDir roots. This sandboxing prevents path traversal attacks and ensures the tool remains read-only and scope-limited.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}