# How the pi-web file-index API Route Builds and Serves File Indices

> Discover how the pi-web file-index API route builds and serves file indices. Learn about directory validation, caching, index generation with git ls-files or filesystem walk, and filtered results.

- Repository: [Alex Yang/pi-web](https://github.com/agegr/pi-web)
- Tags: how-to-guide
- Published: 2026-07-24

---

**The file-index API route in pi-web validates directory permissions against an allow-list, caches file listings for 10 seconds using a global cache, generates indices via `git ls-files` when available or a breadth-first filesystem walk otherwise, and returns either truncated plain listings or fuzzy-search filtered results based on query parameters.**

The file-index endpoint is a critical component of the pi-web file management interface, providing fast, searchable directory listings to the frontend. Located at [`app/api/file-index/route.ts`](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts), this Next.js API route handles both full index generation and client-side search queries while enforcing strict security boundaries identical to the `/api/files` endpoint.

## Request Validation and Authorization

Before touching the filesystem, the handler extracts the `cwd` query parameter and validates that it is an absolute path. It then checks permissions using `getAllowedFileRoots` and `isFilePathAllowed` imported from [`lib/file-access.ts`](https://github.com/agegr/pi-web/blob/main/lib/file-access.ts) ([lines 24‑28](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L24-L28)). 

If the path is not in the allow-list, the route returns a 403 Forbidden response. After authorization, it verifies the path exists and is a directory using `fs.statSync`. Any validation failure results in an immediate JSON error response with the appropriate HTTP status code (400, 403, or 404) ([lines 17‑23](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L17-L23)).

## Global Caching Strategy

To survive Next.js hot-reloads, the route stores cached indices on `globalThis.__piFileIndexCache`. The cache implementation enforces two hard limits:

- **Cache TTL:** Entries expire after **10 seconds** (`CACHE_TTL_MS`)
- **Max Entries:** The cache retains at most **20** indices (`CACHE_MAX_ENTRIES`)

When a request arrives, `getIndexCache` checks for a valid cached entry ([lines 49‑60](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L49-L60)). On a cache miss, the route generates a fresh listing and stores it before serving the response.

## Index Generation: Git-Aware Fast Path vs. Fallback Walk

The route optimizes for Git repositories while providing a robust fallback for standard directories.

### Git Repository Detection

If the `cwd` is inside a Git worktree, the route spawns `git ls-files --cached --others --exclude-standard -z` via `execFileAsync`. The NUL-delimited output is split into an array of relative paths. 

To prevent memory exhaustion on massive repositories, the collection stops at **`GIT_HARD_CAP = 200 000`** entries. When truncation occurs, the response includes a `hardTruncated` flag ([lines 61‑73](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L61-L73)).

### Filesystem Breadth-First Search

For non-Git directories, the route performs a breadth-first walk limited to **depth 8**. It skips common ignore patterns such as `node_modules`, `.git`, and `.pyc` files. The walk halts once **`WALK_HARD_CAP = 50 000`** entries are collected, again flagging truncation in the response ([lines 79‑106](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L79-L106)).

## Serving Responses: Full Listings vs. Search Queries

The route behaves differently depending on the presence of the `q` query parameter.

### Plain Index Without Query Parameters

When no search query is provided, the route returns the first **`MAX_FILES = 5 000`** entries from the cached list. The JSON payload includes a `truncated` boolean indicating whether the underlying generation exceeded either the Git or walk hard caps ([lines 57‑62](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L57-L62)).

```bash
curl "https://your-pi-web.local/api/file-index?cwd=/home/user/project"

```

```json
{
  "files": ["src/index.ts", "src/lib/utils.ts", "README.md"],
  "truncated": false
}

```

### Fuzzy Search With Query Parameter

When the `q` parameter is present, the cached file list is lazily converted into `FileIndexEntry` objects via `buildEntriesFromFiles`. The `filterFileEntries` function—implemented in [`lib/file-fuzzy.ts`](https://github.com/agegr/pi-web/blob/main/lib/file-fuzzy.ts)—ranks entries against the query string and returns matches containing `path` and `isDir` properties ([lines 52‑55](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L52-L55)).

```bash
curl "https://your-pi-web.local/api/file-index?cwd=/home/user/project&q=config"

```

```json
{
  "matches": [
    {"path": "config/webpack.config.js", "isDir": false},
    {"path": "src/config/default.ts", "isDir": false},
    {"path": "config", "isDir": true}
  ]
}

```

### Client-Side Integration

You can consume the endpoint from a React component using SWR for automatic caching and revalidation:

```tsx
import useSWR from "swr";

function useFileIndex(cwd: string, query?: string) {
  const url = new URL("/api/file-index", window.location.origin);
  url.searchParams.set("cwd", cwd);
  if (query) url.searchParams.set("q", query);
  return useSWR(url.toString(), (url) => fetch(url).then(r => r.json()));
}

```

## Error Handling and Security Boundaries

Any filesystem or validation error triggers a JSON error response with a corresponding HTTP status. The route handles permission denials (403), missing directories (404), and internal server errors (500) consistently ([lines 30‑38](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L30-L38), [lines 62‑64](https://github.com/agegr/pi-web/blob/main/app/api/file-index/route.ts#L62-L64)).

## Summary

- **Security first:** The route enforces access control via `isFilePathAllowed` from [`lib/file-access.ts`](https://github.com/agegr/pi-web/blob/main/lib/file-access.ts) before accessing any paths.
- **Short-lived cache:** A 10-second TTL cache on `globalThis.__piFileIndexCache` minimizes redundant Git or filesystem operations while surviving Next.js hot-reloads.
- **Git optimization:** Repositories use `git ls-files` for instant listings capped at 200,000 entries, falling back to manual traversal only when necessary.
- **Resource limits:** Non-Git directories trigger a breadth-first walk limited to depth 8 and 50,000 entries to prevent blocking the event loop.
- **Client caps:** API responses return a maximum of 5,000 files unless filtered by the fuzzy search engine implemented in [`lib/file-fuzzy.ts`](https://github.com/agegr/pi-web/blob/main/lib/file-fuzzy.ts).

## Frequently Asked Questions

### What is the file-index API route in pi-web used for?

The file-index API route provides a fast, searchable index of files under a specified working directory, enabling the frontend to display file trees and perform fuzzy searches without reloading the entire page. It serves as the backend for file picker interfaces and quick-open dialogs.

### How does pi-web cache file index results?

The route stores generated indices on `globalThis.__piFileIndexCache` with a 10-second TTL and a maximum of 20 entries, ensuring cache persistence across Next.js hot-reloads while preventing memory bloat. This global cache survives module reloads during development.

### Why does the file-index route use git ls-files instead of walking the directory?

When inside a Git repository, `git ls-files` provides significantly faster access to the tracked file list while automatically respecting `.gitignore` rules and exclude patterns. The route falls back to manual filesystem traversal only when Git is unavailable or the directory is not a repository.

### What are the hard caps on file index generation?

The Git path caps at 200,000 entries (`GIT_HARD_CAP`), the filesystem walk caps at 50,000 entries (`WALK_HARD_CAP`), and standard API responses return a maximum of 5,000 files (`MAX_FILES`) unless using the search query parameter, which filters the full cached list before returning results.