How to Filter Bash Directory Listings with Pi Web: A Full-Stack Guide

Pi Web provides a server-side /api/file-index endpoint that performs fuzzy filtering on directory listings using the same scoring ladder as the pi TUI.

Filtering directory listings efficiently requires balancing filesystem I/O costs with responsive query performance. Pi Web solves this by separating raw file enumeration from fuzzy matching, caching full listings, and applying a multi-tier scoring algorithm. Below is a complete walkthrough of how this works, drawn directly from the agegr/pi-web source code.

The File-Index API Architecture

Pi Web's filtering system operates in three distinct phases: directory scanning, caching, and fuzzy matching. Each phase uses specific functions from the codebase to ensure security and performance.

Phase 1: Directory Enumeration

When a request hits GET /api/file-index, Pi Web first validates the cwd parameter against an allow-list in [lib/file-access.ts](https://github.com/agegr/pi-web/blob/main/lib/file-access.ts). Then it gathers files using one of two strategies.

Git-optimized listing: For Git repositories, Pi Web executes git ls-files combined with git status to capture tracked, modified, and untracked files while respecting .gitignore:

// app/api/file-index/route.ts
let gitResult = runCommand("git status --porcelain .", cwd);
let prefixedWithRootPath = ...;
let result = runCommand("git ls-files --cached --modified --others --deduplicate", cwd);

Fallback filesystem walk: When Git is unavailable, Pi Web performs a breadth-first directory traversal with configurable ignore patterns:

// app/api/file-index/route.ts
for await (const entry of opendir(posix, hangingPromiseGuard)) {
  if (entry.isDirectory()) {
    if (IGNORE_DIRS.includes(entry.name)) continue;
    dirs.push(secondPathJoin([base, entry.name]));
  } else {
    files.push(pathJoin(posix, [base, entry.name]));
  }
}

Both methods truncate hard limits—200,000 entries for Git listings and 50,000 for walked directories—preventing memory exhaustion on massive codebases.

Phase 2: Response Caching

Full directory listings are expensive to generate. Pi Web caches them on globalThis.__piFileIndexCache for 10 seconds using a cache key that includes the normalized cwd path:

// app/api/file-index/route.ts
const cacheKey = cwd;
let cached = globalThis.__piFileIndexCache?.get(cacheKey);
if (!cached) {
  cached = { ...(gitResult ?? fallBack(cwd)), timestamp: Date.now() };
  if (!globalThis.__piFileIndexCache) globalThis.__piFileIndexCache = new Map();
  globalThis.__piFileIndexCache.set(cacheKey, cached);
}

This cache-aside pattern eliminates redundant git or filesystem calls when multiple filter requests arrive for the same directory within the TTL window.

Phase 3: Fuzzy Filtering with filterFileEntries

When a query parameter q is present, Pi Web transforms the raw file list into structured FileIndexEntry objects and applies the scoring algorithm defined in [lib/file-fuzzy.ts](https://github.com/agegr/pi-web/blob/main/lib/file-fuzzy.ts).

The filterFileEntries function handles the heavy lifting:

// lib/file-fuzzy.ts
export function filterFileEntries(
  entries: FileIndexEntry[],
  query: string,
  limit: number = AT_RESULT_LIMIT,
): FileIndexEntry[] {
  const lowerQuery = query.toLowerCase();
  if (!lowerQuery) return entries.slice(0, limit);

  const scored: Array<{ entry: FileIndexEntry; score: number }> = [];
  for (const entry of entries) {
    const score = scoreEntry(entry, lowerQuery);
    if (score > 0) scored.push({ entry, score });
  }

  scored.sort((a, b) =>
    b.score - a.score
    || pathDepth(a.entry.path) - pathDepth(b.entry.path)
    || a.entry.path.localeCompare(b.entry.path)
  );

  return scored.slice(0, limit).map(s => s.entry);
}

The Scoring Ladder

Each entry receives a score based on match quality, implemented in scoreEntry:

Match Type	Score	Description
Exact match	100	Filename equals query exactly
Prefix match	80	Filename starts with query
Substring match	50	Query appears anywhere in filename
Path substring	30	Query appears in parent directory path
Fuzzy match	10	All characters of query appear in order

Directories receive a +10 bonus to surface folder navigation options even with weak matches:

// lib/file-fuzzy.ts
if (result == null) {
  const fuzzy = fuzzyMatch(entry, lowerQuery);
  if (fuzzy.hit) {
    score = AT_SCORE_FUZZY_MATCH + (entry.isDir ? 10 : 0) - fuzzy.errors;
  }
}

Using the API: Practical Examples

Bash/Curl: Quick Directory Filtering

Query the API directly from your shell to find files matching "util" in the current project:

curl -s "http://localhost:30141/api/file-index?cwd=$(pwd)&q=util" | jq '.matches'

Response format:

{
  "matches": [
    { "path": "src/utils/helpers.ts", "isDir": false },
    { "path": "src/components/utility", "isDir": true }
  ]
}

JavaScript/TypeScript: Programmatic Access

Integrate the filter into build tools or custom scripts:

async function findFiles(cwd: string, query: string) {
  const params = new URLSearchParams({ cwd, q: query });
  const response = await fetch(`/api/file-index?${params}`);
  const data = await response.json();
  return data.matches as Array<{ path: string; isDir: boolean }>;
}

// Usage
const matches = await findFiles('/home/user/project', 'config');
console.log(matches.map(m => m.path));

Unfiltered Listing: Bulk Directory Index

Omit the q parameter to retrieve the first 5,000 entries for client-side filtering:

curl -s "http://localhost:30141/api/file-index?cwd=$(pwd)" | jq

The response includes a hardTruncated flag indicating whether the full listing exceeds the safety limit.

Key Implementation Files

File	Responsibility
[app/api/file-index/route.ts](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts)	HTTP handler, caching logic, Git/walk selection
[lib/file-fuzzy.ts](https://github.com/agegr/pi-web/blob/main/lib/file-fuzzy.ts)	buildEntriesFromFiles, filterFileEntries, scoring ladder
[lib/file-access.ts](https://github.com/agegr/pi-web/blob/main/lib/file-access.ts)	Allow-list validation for secure path access
[lib/file-paths.ts](https://github.com/agegr/pi-web/blob/main/lib/file-paths.ts)	Cross-platform path normalization utilities

Summary

• Git-first enumeration in [app/api/file-index/route.ts](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts) optimizes for version-controlled projects.
• 10-second response caching on globalThis.__piFileIndexCache eliminates redundant filesystem scans.
• Five-tier scoring ladder in [lib/file-fuzzy.ts](https://github.com/agegr/pi-web/blob/main/lib/file-fuzzy.ts) ranks exact matches highest, with directory bonuses for navigation.
• Hard limits (200K Git, 50K walk) protect against resource exhaustion on large repositories.
• Optional query parameter switches between bulk listing mode (5,000 entries) and filtered match mode (top 20 scored results).

Frequently Asked Questions

How does Pi Web handle very large directories?

Pi Web enforces hard truncation limits: 200,000 entries when using git ls-files and 50,000 entries for filesystem walks. The hardTruncated boolean in responses indicates whether these limits were hit, signaling that client-side filtering may miss matches.

What matching algorithm does the fuzzy filter use?

The filterFileEntries function implements a custom scoring ladder with exact, prefix, substring, path-substring, and fuzzy tiers. The fuzzy tier uses character-ordered matching with error penalties, adapted from the same logic powering the pi TUI experience.

Why does Pi Web cache directory listings instead of filtering on every request?

The 10-second cache TTL balances freshness with performance. Since full directory enumeration requires spawning Git processes or walking the filesystem, caching prevents redundant work when multiple autocomplete requests arrive for the same working directory.

Can I use this API outside the Pi Web interface?

Yes. The /api/file-index endpoint accepts standard HTTP GET requests with cwd and optional q query parameters. It returns JSON suitable for shell scripts, editor plugins, or custom developer tools running against a local Pi Web server.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}