Efficient Code Searching with earendil pi grep and find tools

The earendil pi grep and find tools provide high-performance code search by wrapping native binaries (ripgrep and fd) with intelligent truncation, streaming JSON parsing, and pluggable filesystem operations, enabling agents to search massive repositories while enforcing strict output limits.

The coding-agent package in the earendil-works/pi repository ships two purpose-built utilities designed for AI agent workflows. These earendil pi grep and find tools combine the raw speed of ripgrep and fd with architectural safety rails—including automatic binary management, size truncation, and type-safe parameter validation—to deliver sub-second search across large codebases without overwhelming the interactive UI.

Tool Architecture and Capabilities

Both tools follow a consistent architectural pattern while serving distinct search purposes:

• grep – Built on ripgrep (rg), performs regex or literal text search across file contents with support for case-insensitivity, glob filtering, and context lines. Respects .gitignore and returns file-relative paths with line numbers. Hard limits: 100 matches (configurable) and output size capped at ≈50 KB, with individual lines truncated to 500 characters.

• find – Built on fd (with fallback), executes file-system glob searches returning matching paths. Respects .gitignore automatically. Hard limits: 1,000 results (configurable) and ≈50 KB output size.

Core Implementation Details

Each tool exports a factory function conforming to the ToolDefinition interface: createGrepToolDefinition in [packages/coding-agent/src/core/tools/grep.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/grep.ts) and createFindToolDefinition in [packages/coding-agent/src/core/tools/find.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/find.ts).

Parameter validation uses typebox (Type.Object) to enforce strong type-checking for JSON-RPC calls, ensuring agents pass valid pattern, limit, ignoreCase, and path parameters.

Pluggable operations are defined via GrepOperations and FindOperations interfaces. By default, implementations delegate to Node’s fs/promises (readFile, stat) and external binaries, but callers can inject custom logic—for example, remote SSH filesystem access—without modifying core search logic.

Binary discovery guarantees cross-platform functionality. The ensureTool("rg", true) and ensureTool("fd", true) functions download the correct platform-specific binaries on-demand if missing, caching them for subsequent calls.

Execution Flow and Truncation Logic

The search process follows a strict pipeline to balance performance with safety:

1. Path resolution – resolveToCwd converts relative paths to absolute against the working directory.
2. Validation – ops.isDirectory or ops.exists verifies the target path before spawning processes.
3. Argument building – Constructs native binary flags (e.g., ["--json", "--line-number", "--ignore-case"] for rg).
4. Streaming execution – Spawns the child process, streaming JSON output (ripgrep) or plain lines (fd) while parsing in real-time.
5. Limit enforcement – Once the configured match count is reached, the tool invokes stopChild(true) to kill the process immediately, preventing unnecessary CPU use.
6. Post-processing – Reads files for context lines via readFile, applies truncateLine (500 char limit), and runs truncateHead to enforce the byte and line caps defined in [packages/coding-agent/src/core/tools/truncate.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/truncate.ts).
7. Rendering – renderCall and renderResult format human-readable output using the TUI’s Text component, appending warnings like “100 matches limit reached” when truncation occurs.

Practical Code Examples

Searching Code with the Grep Tool

Invoke createGrepTool with a working directory, then execute with regex patterns and ripgrep-compatible options:

import { createGrepTool } from "@earendil-works/pi-coding-agent";

const cwd = process.cwd();
const grep = createGrepTool(cwd);

(async () => {
  // Case-insensitive search for "TODO" with 2 context lines, capped at 50 matches
  const result = await grep.execute("0", {
    pattern: "TODO",
    ignoreCase: true,
    context: 2,
    limit: 50,
  });

  console.log(result.content[0].text);
})();

• Set literal: true to treat pattern as a fixed string instead of regex.
• ignoreCase, context, and limit map directly to ripgrep flags (--ignore-case, -C, --max-count).

Locating Files with the Find Tool

Use createFindTool for glob-based file discovery that respects .gitignore:

import { createFindTool } from "@earendil-works/pi-coding-agent";

const find = createFindTool(process.cwd());

(async () => {
  // Find all TypeScript test files under packages/, limit to 200 results
  const { content } = await find.execute("0", {
    pattern: "packages/**/*.test.ts",
    limit: 200,
  });

  console.log("Matching files:\n" + content[0].text);
})();

• Patterns follow standard glob syntax; the tool automatically adds --full-path when the pattern contains / to ensure hierarchical matching.

Custom Filesystem Operations for Remote Workflows

Override operations to support non-local filesystems such as SSH or remote APIs:

import { createGrepTool, type GrepOperations } from "@earendil-works/pi-coding-agent";

const remoteOps: GrepOperations = {
  isDirectory: async (p) => true,
  readFile: async (p) => {
    // Replace with SSH fetch logic
    return await fetchRemoteFile(p);
  },
};

const grep = createGrepTool(process.cwd(), { operations: remoteOps });
await grep.execute("0", { pattern: "ERROR", path: "/remote/logs" });

The same FindOperations interface applies to the find tool for custom file listing logic.

Key Source Files

• [packages/coding-agent/src/core/tools/grep.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/grep.ts) – Core grep implementation, argument building, and JSON streaming parser.
• [packages/coding-agent/src/core/tools/find.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/find.ts) – Find tool implementation and fd execution logic.
• [packages/coding-agent/src/core/tools/truncate.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/truncate.ts) – Shared truncation utilities for head/tail and line-level limits.
• [packages/coding-agent/src/core/tools/render-utils.ts](https://github.com/earendil-works/pi/blob/main/packages/coding-agent/src/core/tools/render-utils.ts) – Safe output rendering helpers.

Summary

• earendil pi grep and find tools wrap ripgrep and fd to deliver high-performance search with built-in safety limits.
• Hard caps prevent output overflow: 100 matches (grep) / 1,000 files (find) and approximately 50 KB total output size.
• Pluggable architecture via GrepOperations and FindOperations supports remote filesystems and custom I/O without core code changes.
• Automatic binary management ensures rg and fd are available on any platform via ensureTool.
• Streaming truncation stops searches early when limits are reached, preserving agent context window space.

Frequently Asked Questions

How do the earendil pi grep and find tools handle very large repositories?

The tools use streaming JSON parsing for ripgrep output and line-by-line reading for fd, coupled with aggressive limit enforcement. Once the match cap (100 for grep, 1,000 for find) is reached, the child process is killed immediately via stopChild(true). Post-processing applies truncateLine (500 character limit) and truncateHead (50 KB limit) to ensure the agent receives only essential data.

What is the difference between the grep tool and the find tool?

grep searches file contents using ripgrep, supporting regex patterns, case-insensitivity, and context lines, making it ideal for finding specific code snippets. find searches filenames and paths using fd with glob syntax, returning file locations without opening content. Grep caps results at 100 matches, while find allows up to 1,000 file paths.

Can I use these tools with remote or virtual filesystems?

Yes. Both tools accept custom operations objects implementing GrepOperations or FindOperations interfaces. You can inject custom readFile, isDirectory, and exists implementations—such as SSH-based or cloud storage adapters—without modifying the core search logic in grep.ts or find.ts.

How are the ripgrep and fd binaries managed on different platforms?

The ensureTool function handles platform detection and on-demand binary downloads. When createGrepTool or createFindTool initializes, it calls ensureTool("rg", true) or ensureTool("fd", true), which downloads the appropriate executable if not present in the cache, ensuring the earendil pi grep and find tools work consistently across macOS, Linux, and Windows environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}