How BM25 Search Retrieves Past Sessions with Scope="project" vs "global" in DeepSeek-Reasonix

The scope parameter in DeepSeek-Reasonix limits BM25 search to the current project directory when set to "project", or expands retrieval to include global sessions and archived files when set to "global", as implemented in the sources method of internal/history/search.go.

The DeepSeek-Reasonix repository implements a BM25 retrieval system for accessing conversation history through the history tool. When querying past sessions, the scope parameter determines whether the search remains confined to the active project or spans across all historical interactions on the machine. Understanding how scope="project" differs from scope="global" is essential for optimizing retrieval performance and relevance.

How Scope Filtering Works in the Searcher

The history tool's search implementation resides in internal/history/search.go. The Searcher struct determines which files to index through its private sources method, which branches based on the scope parameter.

Project Scope Implementation

When scope="project" is specified (or left empty), the searcher only examines the current session directory. The sources method calls appendSessionSources exclusively on s.sessionDir:

func (s *Searcher) sources(scope string) ([]sourceFile, error) {
    var out []sourceFile
    seen := map[string]bool{}
    // always add the current session directory (project‑scoped)
    out = appendSessionSources(out, seen, s.sessionDir, scopeProject)

    // …only when the caller asked for global scope
    if scope == scopeGlobal {
        out = appendSessionSources(out, seen, s.globalSessionDir, scopeGlobal)
        out = appendFiles(out, seen, listJSONL(s.archiveDir, "archive", nil)...)
    }
    // sort newest‑first …
    return out, nil
}

This limits the BM25 search to JSONL files within the active project directory and its subagents/ subdirectory.

Global Scope Implementation

Setting scope="global" triggers additional calls to include the user-wide session store and archived files. After appending project sources, the code executes:

if scope == scopeGlobal {
    out = appendSessionSources(out, seen, s.globalSessionDir, scopeGlobal)
    out = appendFiles(out, seen, listJSONL(s.archiveDir, "archive", nil)...)
}

This expands the corpus to include s.globalSessionDir (shared across projects) and every *.jsonl file in s.archiveDir, allowing the model to surface results from any past interaction.

Scope Validation and Normalization

Before processing, the normalizeScope helper in internal/history/search.go validates the input:

func normalizeScope(scope string) (string, error) {
    switch strings.TrimSpace(scope) {
    case "", scopeProject:
        return scopeProject, nil
    case scopeGlobal:
        return scopeGlobal, nil
    default:
        return "", fmt.Errorf("scope must be %q or %q", scopeProject, scopeGlobal)
    }
}

An empty string defaults to project scope, ensuring backward compatibility and preventing invalid scope values from reaching the search logic.

Practical Implementation Examples

Searching Within Project Scope

To restrict BM25 retrieval to the current project:

s := history.NewSearcher(history.Options{
    SessionDir:       "/home/user/.reasonix/sessions/2024-06-22-abc",
    GlobalSessionDir: "/home/user/.reasonix/sessions",
    ArchiveDir:       "/home/user/.reasonix/sessions/.archive",
})

req := history.SearchRequest{
    Query: "failed to compile",
    Scope: "project",
    Limit: 5,
}
hits, _ := s.Search(context.Background(), req)

This configuration only indexes files under the specified SessionDir and its subdirectories.

Searching Across All Sessions

To include global and archived sessions:

req.Scope = "global"
hits, _ = s.Search(context.Background(), req)

Now the BM25 algorithm scores documents from the global session directory and archive folder alongside the current project files.

Command Line Usage

From the terminal:

reasonix history search \
  --query "permission denied" \
  --scope global \
  --limit 10

The CLI constructs a SearchRequest with the specified scope and invokes the same underlying Searcher according to the esengine/DeepSeek-Reasonix source code.

Source Code Architecture

The BM25 retrieval system spans several key files:

• internal/history/search.go: Contains the Searcher struct, sources method, and normalizeScope helper that implement scope filtering.
• internal/history/tool.go: Exposes the history tool to the controller via JSON-RPC.
• internal/retrieval/bm25.go: Implements the BM25 scoring algorithm used by the searcher.
• internal/history/strip.go: Provides message preprocessing before indexing.

Summary

• scope="project" limits BM25 search to the active session directory and its subagents.
• scope="global" expands the search to include the global session directory and archived files.
• The sources method in internal/history/search.go enforces these boundaries by conditionally appending file sources.
• normalizeScope ensures empty strings default to project scope while explicitly validating allowed values.

Frequently Asked Questions

What happens if I omit the scope parameter entirely?

The normalizeScope function treats empty strings as equivalent to "project", so the search defaults to project-scoped retrieval. This ensures that BM25 searches remain localized unless explicitly broadened to global scope.

Does global scope include archived sessions?

Yes. When scope="global" is set, the sources method calls listJSONL on the archive directory, including every *.jsonl file stored in s.archiveDir alongside active global sessions.

Where is the scope logic implemented in the codebase?

The scope filtering logic resides in internal/history/search.go within the sources method (lines 94-102) and the normalizeScope helper function (lines 63-71), as implemented in the esengine/DeepSeek-Reasonix repository.

Can I search only archived sessions without global active sessions?

No. The current implementation bundles archives with the global scope. When scope="global" is selected, the search includes both the global session directory and the archive directory; there is no standalone "archive-only" scope option.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}