internals

How the pi-natives Rust Addon Enables In-Process Grep and Shell Commands

May 21, 2026 can1357/oh-my-pi ↗

The pi-natives addon is a Rust-based N-API module that embeds high-performance grep searching and shell execution directly into the Node.js process, eliminating the overhead of external binaries through parallel regex engines and async process management.

The pi-natives package in the can1357/oh-my-pi repository bridges JavaScript and systems programming by compiling Rust crates into a native Node-API addon. This architecture enables in-process grep and shell commands that run inside the same memory space as the JavaScript runtime, providing fine-grained cancellation, streaming output, and significantly faster file operations than traditional child-process spawning.

Architecture of the Native Addon

The addon is built with napi-rs and exposes Rust functionality through the #[napi] macro system. All entry points are defined in [crates/pi-natives/src/lib.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs), which re-exports the version sentinel (__piNativesV15_2_1) and public functions from feature-specific modules.

The codebase organizes functionality into three core domains:

In-Process Grep Implementation

The grep functionality provides a pure-Rust alternative to spawning ripgrep binaries, embedding the search engine directly via the grep crate ecosystem.

Pattern Compilation and Sanitization

When JavaScript calls piNatives.grep(), the grep_sync function deserializes GrepOptions into a GrepConfig struct. The build_matcher function sanitizes regex patterns by escaping braces and parentheses, then constructs a RegexMatcher from the grep-regex crate. This matcher compiles the pattern once and reuses it across all files in the search scope.

Parallel File Walking and Searching

File discovery utilizes fs_cache::build_walker to construct an ignore::WalkBuilder that respects .gitignore rules, hidden-file filters, and a shared filesystem cache for repeated searches. The run_parallel_search function distributes work across threads using rayon::prelude::*, processing files in parallel.

For each file, the system uses memory-mapped I/O for files larger than 128KB. The MatchCollector struct implements grep_searcher::Sink to receive matches, collect context lines, and enforce limits or byte offsets without loading entire files into memory.

Result Aggregation and Cancellation

The aggregate_parallel_results function transforms internal CollectedMatch structs into GrepMatch and GrepResult objects, supporting three output modes: content (full lines), count (match totals), and filesWithMatches (path lists only).

Cancellation is cooperative: a CancelToken from [task.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/task.rs) is checked every 128 files and passed to the underlying Searcher. If the token signals timeout or abort, the search returns immediately with a Timeout error, preventing runaway operations from blocking the event loop.

In-Process Shell Command Execution

The shell module provides persistent session management and streaming output without relying on Node.js child_process.

Persistent CoreShell Sessions

Each Shell N-API object maintains an Arc<CoreShell> instance from the pi-shell crate (built on brush-core). This persistent shell retains environment variables and working directory state across multiple commands, unlike one-off process spawns. The run and execute_shell entry points in [shell.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs) manage this lifecycle.

Async Streaming via Threadsafe Functions

Command execution uses tokio::process::Command for async process handling. The bridge_chunks function creates an unbounded mpsc channel; stdout and stderr streams are captured by Tokio tasks and forwarded to JavaScript through a ThreadsafeFunction<String>. This mechanism allows real-time output streaming without blocking the main thread or buffering entire outputs in memory.

Cancellation and Output Minimization

Each command receives a CancelToken that drives both timeout enforcement (via timeoutMs) and explicit abort signals. If the token fires, the underlying Tokio process is killed and the promise resolves with cancelled: true.

When MinimizerOptions are provided, the output buffer passes through brush_core::minimizer before returning to JavaScript. This feature can rewrite large outputs into concise artifact references, reducing memory pressure and IPC overhead for verbose commands.

Practical Usage Examples

Searching Codebases with Native Grep

import * as piNatives from "pi-natives";

// Search for TODO markers in TypeScript files
const result = await piNatives.grep({
  pattern: "TODO",
  path: ".",
  glob: "**/*.ts",
  ignoreCase: true,
  maxCount: 10,
  mode: "content"
});

console.log(`Found ${result.totalMatches} matches`);
result.matches.forEach(m => {
  console.log(`${m.path}:${m.lineNumber} → ${m.line}`);
});

This invokes grep_sync in [crates/pi-natives/src/grep.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/grep.rs), executing the full parallel search pipeline described above.

Executing Shell Commands with Streaming Output

import * as piNatives from "pi-natives";

const shell = new piNatives.Shell({
  sessionEnv: { NODE_ENV: "production" }
});

const onChunk = (err, chunk) => {
  if (err) console.error(err);
  else process.stdout.write(chunk);
};

const result = await shell.run(
  {
    command: "git log --oneline -5",
    cwd: "/path/to/repo",
    timeoutMs: 5000
  },
  onChunk
);

console.log("Exit code:", result.exitCode);
if (result.minimized) {
  console.log("Minimized output:", result.minimized.text);
}

This interfaces with Shell::run in [crates/pi-natives/src/shell.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs), utilizing the CoreShell persistent session and Tokio-based streaming architecture.

Summary

Frequently Asked Questions

How does pi-natives achieve better performance than spawning ripgrep or bash processes?

By staying in-process, the addon avoids the overhead of process creation, IPC serialization, and context switching. The grep implementation uses memory-mapped I/O for large files and rayon for CPU-bound parallel searching, while the shell uses tokio for efficient async I/O without blocking the JavaScript thread. According to the can1357/oh-my-pi source code, this eliminates the "spawn tax" associated with child_process while providing finer control over cancellation and resource limits through the CancelToken primitive.

Can I cancel a long-running grep or shell command mid-execution?

Yes. Both the grep_sync function and Shell::run method accept a cancellation mechanism. The grep engine checks a CancelToken every 128 files and respects it during the search phase, while shell commands use the same token to kill the underlying tokio::process::Child. You can trigger cancellation via timeout (by setting timeoutMs) or by explicitly aborting the operation from JavaScript.

What Rust crates power the underlying functionality?

The grep feature relies on the grep-matcher, grep-regex, grep-searcher, globset, ignore, and rayon crates for regex compilation and parallel filesystem walking. The shell feature uses brush-core (via pi-shell) for POSIX-compliant shell parsing and tokio for async process management. All bindings are generated using napi-rs, which provides the #[napi] procedural macros visible throughout [lib.rs](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs).

How does the minimizer feature work in shell commands?

When MinimizerOptions are passed to shell.run(), the output buffer is processed by brush_core::minimizer before being returned to JavaScript. This Rust-side optimization can truncate verbose outputs or replace them with structured artifact references, reducing the amount of data that must cross the N-API boundary. The minimized result is included in the ShellRunResult under the minimized field, allowing the JavaScript caller to handle large outputs efficiently.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →