# How the pi-natives Rust Addon Enables In-Process Grep and Shell Commands

> Discover how the pi-natives Rust addon brings in-process grep and shell commands to Node.js. Experience lightning-fast execution and reduced overhead by embedding Rust's power directly into your application. Learn more today.

- Repository: [Can Bölük/oh-my-pi](https://github.com/can1357/oh-my-pi)
- Tags: internals
- Published: 2026-05-21

---

**The pi-natives addon is a Rust-based N-API module that embeds high-performance grep searching and shell execution directly into the Node.js process, eliminating the overhead of external binaries through parallel regex engines and async process management.**

The `pi-natives` package in the `can1357/oh-my-pi` repository bridges JavaScript and systems programming by compiling Rust crates into a native Node-API addon. This architecture enables **in-process grep and shell commands** that run inside the same memory space as the JavaScript runtime, providing fine-grained cancellation, streaming output, and significantly faster file operations than traditional child-process spawning.

## Architecture of the Native Addon

The addon is built with `napi-rs` and exposes Rust functionality through the `#[napi]` macro system. All entry points are defined in [[`crates/pi-natives/src/lib.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs), which re-exports the version sentinel (`__piNativesV15_2_1`) and public functions from feature-specific modules.

The codebase organizes functionality into three core domains:

- **Grep Engine** – Located in [[`crates/pi-natives/src/grep.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/grep.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/grep.rs), leveraging the `grep-matcher`, `grep-regex`, and `grep-searcher` crates alongside `rayon` for parallelism.
- **Shell Runtime** – Implemented in [[`crates/pi-natives/src/shell.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs), wrapping `brush-core` (via `pi-shell`) and `tokio` for persistent command sessions.
- **Shared Infrastructure** – [[`crates/pi-natives/src/task.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/task.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/task.rs) provides the `CancelToken` primitive used by both features for timeout and abort semantics.

## In-Process Grep Implementation

The grep functionality provides a pure-Rust alternative to spawning `ripgrep` binaries, embedding the search engine directly via the `grep` crate ecosystem.

### Pattern Compilation and Sanitization

When JavaScript calls `piNatives.grep()`, the `grep_sync` function deserializes `GrepOptions` into a `GrepConfig` struct. The `build_matcher` function sanitizes regex patterns by escaping braces and parentheses, then constructs a `RegexMatcher` from the `grep-regex` crate. This matcher compiles the pattern once and reuses it across all files in the search scope.

### Parallel File Walking and Searching

File discovery utilizes `fs_cache::build_walker` to construct an `ignore::WalkBuilder` that respects `.gitignore` rules, hidden-file filters, and a shared filesystem cache for repeated searches. The `run_parallel_search` function distributes work across threads using `rayon::prelude::*`, processing files in parallel.

For each file, the system uses memory-mapped I/O for files larger than 128KB. The `MatchCollector` struct implements `grep_searcher::Sink` to receive matches, collect context lines, and enforce limits or byte offsets without loading entire files into memory.

### Result Aggregation and Cancellation

The `aggregate_parallel_results` function transforms internal `CollectedMatch` structs into `GrepMatch` and `GrepResult` objects, supporting three output modes: `content` (full lines), `count` (match totals), and `filesWithMatches` (path lists only).

Cancellation is cooperative: a `CancelToken` from [[`task.rs`](https://github.com/can1357/oh-my-pi/blob/main/task.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/task.rs) is checked every 128 files and passed to the underlying `Searcher`. If the token signals timeout or abort, the search returns immediately with a `Timeout` error, preventing runaway operations from blocking the event loop.

## In-Process Shell Command Execution

The shell module provides persistent session management and streaming output without relying on Node.js `child_process`.

### Persistent CoreShell Sessions

Each `Shell` N-API object maintains an `Arc<CoreShell>` instance from the `pi-shell` crate (built on `brush-core`). This persistent shell retains environment variables and working directory state across multiple commands, unlike one-off process spawns. The `run` and `execute_shell` entry points in [[`shell.rs`](https://github.com/can1357/oh-my-pi/blob/main/shell.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs) manage this lifecycle.

### Async Streaming via Threadsafe Functions

Command execution uses `tokio::process::Command` for async process handling. The `bridge_chunks` function creates an unbounded `mpsc` channel; stdout and stderr streams are captured by Tokio tasks and forwarded to JavaScript through a `ThreadsafeFunction<String>`. This mechanism allows real-time output streaming without blocking the main thread or buffering entire outputs in memory.

### Cancellation and Output Minimization

Each command receives a `CancelToken` that drives both timeout enforcement (via `timeoutMs`) and explicit abort signals. If the token fires, the underlying Tokio process is killed and the promise resolves with `cancelled: true`.

When `MinimizerOptions` are provided, the output buffer passes through `brush_core::minimizer` before returning to JavaScript. This feature can rewrite large outputs into concise artifact references, reducing memory pressure and IPC overhead for verbose commands.

## Practical Usage Examples

### Searching Codebases with Native Grep

```javascript
import * as piNatives from "pi-natives";

// Search for TODO markers in TypeScript files
const result = await piNatives.grep({
  pattern: "TODO",
  path: ".",
  glob: "**/*.ts",
  ignoreCase: true,
  maxCount: 10,
  mode: "content"
});

console.log(`Found ${result.totalMatches} matches`);
result.matches.forEach(m => {
  console.log(`${m.path}:${m.lineNumber} → ${m.line}`);
});

```

This invokes `grep_sync` in [[`crates/pi-natives/src/grep.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/grep.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/grep.rs), executing the full parallel search pipeline described above.

### Executing Shell Commands with Streaming Output

```javascript
import * as piNatives from "pi-natives";

const shell = new piNatives.Shell({
  sessionEnv: { NODE_ENV: "production" }
});

const onChunk = (err, chunk) => {
  if (err) console.error(err);
  else process.stdout.write(chunk);
};

const result = await shell.run(
  {
    command: "git log --oneline -5",
    cwd: "/path/to/repo",
    timeoutMs: 5000
  },
  onChunk
);

console.log("Exit code:", result.exitCode);
if (result.minimized) {
  console.log("Minimized output:", result.minimized.text);
}

```

This interfaces with `Shell::run` in [[`crates/pi-natives/src/shell.rs`](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/shell.rs), utilizing the `CoreShell` persistent session and Tokio-based streaming architecture.

## Summary

- **pi-natives** is a Rust N-API addon that embeds `grep-searcher` and `brush-core` directly into the Node.js process, eliminating external binary dependencies.
- **Grep operations** use `rayon` for parallel file walking, `RegexMatcher` for pattern compilation, and `CancelToken` for cooperative cancellation every 128 files.
- **Shell execution** maintains persistent `Arc<CoreShell>` sessions, streams output via `ThreadsafeFunction` callbacks, and uses `tokio::process::Command` for async I/O.
- Both features share infrastructure from [[`task.rs`](https://github.com/can1357/oh-my-pi/blob/main/task.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/task.rs) (cancellation), [[`fs_cache.rs`](https://github.com/can1357/oh-my-pi/blob/main/fs_cache.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/fs_cache.rs) (filesystem caching), and [[`glob.rs`](https://github.com/can1357/oh-my-pi/blob/main/glob.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/glob.rs) (pattern matching).
- The implementation lives entirely in [`crates/pi-natives/src/`](https://github.com/can1357/oh-my-pi/tree/main/crates/pi-natives/src), with entry points exported from [[`lib.rs`](https://github.com/can1357/oh-my-pi/blob/main/lib.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs).

## Frequently Asked Questions

### How does pi-natives achieve better performance than spawning ripgrep or bash processes?

By staying in-process, the addon avoids the overhead of process creation, IPC serialization, and context switching. The grep implementation uses memory-mapped I/O for large files and `rayon` for CPU-bound parallel searching, while the shell uses `tokio` for efficient async I/O without blocking the JavaScript thread. According to the `can1357/oh-my-pi` source code, this eliminates the "spawn tax" associated with `child_process` while providing finer control over cancellation and resource limits through the `CancelToken` primitive.

### Can I cancel a long-running grep or shell command mid-execution?

Yes. Both the `grep_sync` function and `Shell::run` method accept a cancellation mechanism. The grep engine checks a `CancelToken` every 128 files and respects it during the search phase, while shell commands use the same token to kill the underlying `tokio::process::Child`. You can trigger cancellation via timeout (by setting `timeoutMs`) or by explicitly aborting the operation from JavaScript.

### What Rust crates power the underlying functionality?

The grep feature relies on the `grep-matcher`, `grep-regex`, `grep-searcher`, `globset`, `ignore`, and `rayon` crates for regex compilation and parallel filesystem walking. The shell feature uses `brush-core` (via `pi-shell`) for POSIX-compliant shell parsing and `tokio` for async process management. All bindings are generated using `napi-rs`, which provides the `#[napi]` procedural macros visible throughout [[`lib.rs`](https://github.com/can1357/oh-my-pi/blob/main/lib.rs)](https://github.com/can1357/oh-my-pi/blob/main/crates/pi-natives/src/lib.rs).

### How does the minimizer feature work in shell commands?

When `MinimizerOptions` are passed to `shell.run()`, the output buffer is processed by `brush_core::minimizer` before being returned to JavaScript. This Rust-side optimization can truncate verbose outputs or replace them with structured artifact references, reducing the amount of data that must cross the N-API boundary. The minimized result is included in the `ShellRunResult` under the `minimized` field, allowing the JavaScript caller to handle large outputs efficiently.