How the pi-web file-index API Route Builds and Serves File Indices

The file-index API route in pi-web validates directory permissions against an allow-list, caches file listings for 10 seconds using a global cache, generates indices via git ls-files when available or a breadth-first filesystem walk otherwise, and returns either truncated plain listings or fuzzy-search filtered results based on query parameters.

The file-index endpoint is a critical component of the pi-web file management interface, providing fast, searchable directory listings to the frontend. Located at app/api/file-index/route.ts, this Next.js API route handles both full index generation and client-side search queries while enforcing strict security boundaries identical to the /api/files endpoint.

Request Validation and Authorization

Before touching the filesystem, the handler extracts the cwd query parameter and validates that it is an absolute path. It then checks permissions using getAllowedFileRoots and isFilePathAllowed imported from lib/file-access.ts (lines 24‑28).

If the path is not in the allow-list, the route returns a 403 Forbidden response. After authorization, it verifies the path exists and is a directory using fs.statSync. Any validation failure results in an immediate JSON error response with the appropriate HTTP status code (400, 403, or 404) (lines 17‑23).

Global Caching Strategy

To survive Next.js hot-reloads, the route stores cached indices on globalThis.__piFileIndexCache. The cache implementation enforces two hard limits:

• Cache TTL: Entries expire after 10 seconds (CACHE_TTL_MS)
• Max Entries: The cache retains at most 20 indices (CACHE_MAX_ENTRIES)

When a request arrives, getIndexCache checks for a valid cached entry (lines 49‑60). On a cache miss, the route generates a fresh listing and stores it before serving the response.

Index Generation: Git-Aware Fast Path vs. Fallback Walk

The route optimizes for Git repositories while providing a robust fallback for standard directories.

Git Repository Detection

If the cwd is inside a Git worktree, the route spawns git ls-files --cached --others --exclude-standard -z via execFileAsync. The NUL-delimited output is split into an array of relative paths.

To prevent memory exhaustion on massive repositories, the collection stops at GIT_HARD_CAP = 200 000 entries. When truncation occurs, the response includes a hardTruncated flag (lines 61‑73).

Filesystem Breadth-First Search

For non-Git directories, the route performs a breadth-first walk limited to depth 8. It skips common ignore patterns such as node_modules, .git, and .pyc files. The walk halts once WALK_HARD_CAP = 50 000 entries are collected, again flagging truncation in the response (lines 79‑106).

Serving Responses: Full Listings vs. Search Queries

The route behaves differently depending on the presence of the q query parameter.

Plain Index Without Query Parameters

When no search query is provided, the route returns the first MAX_FILES = 5 000 entries from the cached list. The JSON payload includes a truncated boolean indicating whether the underlying generation exceeded either the Git or walk hard caps (lines 57‑62).

curl "https://your-pi-web.local/api/file-index?cwd=/home/user/project"

{
  "files": ["src/index.ts", "src/lib/utils.ts", "README.md"],
  "truncated": false
}

Fuzzy Search With Query Parameter

When the q parameter is present, the cached file list is lazily converted into FileIndexEntry objects via buildEntriesFromFiles. The filterFileEntries function—implemented in lib/file-fuzzy.ts—ranks entries against the query string and returns matches containing path and isDir properties (lines 52‑55).

curl "https://your-pi-web.local/api/file-index?cwd=/home/user/project&q=config"

{
  "matches": [
    {"path": "config/webpack.config.js", "isDir": false},
    {"path": "src/config/default.ts", "isDir": false},
    {"path": "config", "isDir": true}
  ]
}

Client-Side Integration

You can consume the endpoint from a React component using SWR for automatic caching and revalidation:

import useSWR from "swr";

function useFileIndex(cwd: string, query?: string) {
  const url = new URL("/api/file-index", window.location.origin);
  url.searchParams.set("cwd", cwd);
  if (query) url.searchParams.set("q", query);
  return useSWR(url.toString(), (url) => fetch(url).then(r => r.json()));
}

Error Handling and Security Boundaries

Any filesystem or validation error triggers a JSON error response with a corresponding HTTP status. The route handles permission denials (403), missing directories (404), and internal server errors (500) consistently (lines 30‑38, lines 62‑64).

Summary

• Security first: The route enforces access control via isFilePathAllowed from lib/file-access.ts before accessing any paths.
• Short-lived cache: A 10-second TTL cache on globalThis.__piFileIndexCache minimizes redundant Git or filesystem operations while surviving Next.js hot-reloads.
• Git optimization: Repositories use git ls-files for instant listings capped at 200,000 entries, falling back to manual traversal only when necessary.
• Resource limits: Non-Git directories trigger a breadth-first walk limited to depth 8 and 50,000 entries to prevent blocking the event loop.
• Client caps: API responses return a maximum of 5,000 files unless filtered by the fuzzy search engine implemented in lib/file-fuzzy.ts.

Frequently Asked Questions

What is the file-index API route in pi-web used for?

The file-index API route provides a fast, searchable index of files under a specified working directory, enabling the frontend to display file trees and perform fuzzy searches without reloading the entire page. It serves as the backend for file picker interfaces and quick-open dialogs.

How does pi-web cache file index results?

The route stores generated indices on globalThis.__piFileIndexCache with a 10-second TTL and a maximum of 20 entries, ensuring cache persistence across Next.js hot-reloads while preventing memory bloat. This global cache survives module reloads during development.

Why does the file-index route use git ls-files instead of walking the directory?

When inside a Git repository, git ls-files provides significantly faster access to the tracked file list while automatically respecting .gitignore rules and exclude patterns. The route falls back to manual filesystem traversal only when Git is unavailable or the directory is not a repository.

What are the hard caps on file index generation?

The Git path caps at 200,000 entries (GIT_HARD_CAP), the filesystem walk caps at 50,000 entries (WALK_HARD_CAP), and standard API responses return a maximum of 5,000 files (MAX_FILES) unless using the search query parameter, which filters the full cached list before returning results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}