How DesktopCommanderMCP Implements Streaming Search Results with Pagination in startSearch

startSearch implements streaming results with pagination by spawning a ripgrep process that incrementally parses JSON output into a session-specific buffer, returning a session ID immediately while readSearchResults provides offset-based access to the accumulating results.

The SearchManager class in wonderwhy-er/DesktopCommanderMCP delivers high-performance file searching through a streaming results with pagination architecture. By leveraging Node.js child processes and ripgrep's JSON output mode, the system balances immediate responsiveness with complete result set availability.

The Architecture Behind Streaming Search

In src/search-manager.ts, the startSearch method (lines 59-120) orchestrates a non-blocking search operation by wrapping ripgrep in a Node.js child process. Rather than waiting for the entire search to complete, the method immediately returns a session identifier while the process continues executing in the background.

Spawning the Ripgrep Process

The implementation begins by calling buildRipgrepArgs (lines 333-410) to construct a ripgrep command with JSON output formatting. The method then spawns the process and wires up event handlers via setupProcessHandlers (lines 813-886), which establishes the streaming pipeline between the process stdout and the session storage.

Incremental Output Parsing

As ripgrep writes JSON lines to stdout, the process.on('data') handler appends chunks to a session-wide buffer. The processBufferedOutput routine splits this buffer on newlines, passing complete lines to parseLine for deserialization into SearchResult objects. These objects are immediately pushed into the session's results array, making them available for pagination before the search completes.

The First-Chunk Response Pattern

To provide immediate feedback while maintaining the streaming architecture, startSearch creates a first-chunk promise that resolves when the initial data event arrives or after a 40ms timeout. This allows the method to return the current session state—including any results parsed up to that moment—while the ripgrep process continues executing. The returned payload includes the sessionId, initial results array, and metadata indicating whether the search is complete.

Pagination with readSearchResults

Once a session is established, clients retrieve additional pages via readSearchResults (lines 222-286), which implements offset-based pagination against the in-memory results array.

Offset-Based Slicing

The method signature readSearchResults(sessionId, offset, length) accepts a starting offset and page length, returning a slice of the stored results along with metadata including totalResults, hasMoreResults, and isComplete. This design allows efficient random access to any portion of the result set without re-executing the search.

Tail Mode with Negative Offsets

For monitoring recent matches, the API supports negative offset values that trigger "tail" behavior. When passed a negative offset, readSearchResults calculates the position from the end of the results array, returning the last N entries regardless of total result count. This enables real-time log monitoring use cases where only the most recent matches are relevant.

Session State and Lifecycle Management

Each search session maintains a comprehensive state object tracking lastReadTime, totalMatches, totalContextLines, and boolean flags for isComplete and wasIncomplete. The close handler finalizes the session when the ripgrep process exits, recording whether the search terminated due to permission errors. A periodic cleanup interval removes stale completed sessions to prevent memory leaks.

Code Examples

// Initiate a streaming search
const { sessionId, results } = await searchManager.startSearch({
  rootPath: '/my/project',
  pattern: 'TODO',
  searchType: 'content',
  ignoreCase: true,
  contextLines: 2,
  maxResults: 100
});

console.log('Initial batch:', results);

// Retrieve page 2 (items 100-199)
const page2 = searchManager.readSearchResults(sessionId, 100, 100);
console.log('Page 2:', page2.results);

// Tail mode: get last 10 results
const recent = searchManager.readSearchResults(sessionId, -10);
console.log('Recent matches:', recent.results);

Summary

• startSearch launches ripgrep as a background process and returns immediately with a session ID and initial results.
• Streaming occurs via stdout event handlers that parse JSON lines incrementally into the session's results array.
• First-chunk resolution provides sub-100ms response times through a promise that resolves on the first data event or 40ms timeout.
• readSearchResults supports standard pagination with offset/length parameters and tail-mode with negative offsets.
• Session cleanup automatically removes completed sessions based on inactivity timestamps and process exit status.

Frequently Asked Questions

How does startSearch handle large result sets without blocking?

The method uses a streaming architecture where ripgrep runs in a separate process, with stdout data processed incrementally. This keeps the event loop unblocked while results accumulate in memory, allowing the initial response to return within milliseconds while hundreds of thousands of matches continue processing in the background.

What happens if I request an offset beyond the current results count?

readSearchResults returns the available results up to the requested length, along with hasMoreResults: true and isComplete: false flags. Clients should poll the method until isComplete becomes true or totalResults stabilizes, indicating the ripgrep process has finished writing to the buffer.

Can I stream results in real-time without using pagination?

While startSearch returns the initial batch immediately, the session object in src/search-manager.ts exposes the underlying results array that grows as new data arrives. For true real-time streaming, you would need to poll readSearchResults with appropriate offsets or access the session object directly, though the latter is not recommended as it bypasses the tail-mode calculation and boundary checking.

Why does the first-chunk timeout default to 40ms?

The 40ms timeout balances immediate responsiveness with result completeness. According to the source code, this duration ensures the method returns promptly even for searches with few matches, while still capturing the initial burst of results from ripgrep's highly optimized regex engine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}