# How to Parse Data Using the Hivemind Graph Module: A Complete Guide

> Learn how to parse data with Hivemind's graph module. This guide covers using Tree-Sitter, building JSON snapshots, and querying symbols and relationships via a VFS interface. Optimize your code analysis workflow.

- Repository: [Activeloop/hivemind](https://github.com/activeloopai/hivemind)
- Tags: how-to-guide
- Published: 2026-06-11

---

**The Hivemind graph module parses source code using Tree-Sitter, builds a JSON snapshot of nodes and edges, and exposes it through a virtual filesystem (VFS) interface for querying symbols and relationships.**

The **activeloopai/hivemind** repository provides a code-understanding engine that transforms source files into queryable graph data. When you parse data using the graph module, it creates a structured representation of symbols and their relationships that can be accessed programmatically or through virtual filesystem calls. This architecture enables AI agents to understand codebase structure without loading the entire repository into context.

## How the Graph Module Parses Source Code

The parsing pipeline relies on language-specific Tree-Sitter grammars and a chunked processing strategy to handle large files efficiently.

### Language-Specific Extractors

Each supported language has a dedicated extractor in `src/graph/extract/*.ts` (for example, [`typescript.ts`](https://github.com/activeloopai/hivemind/blob/main/typescript.ts), [`python.ts`](https://github.com/activeloopai/hivemind/blob/main/python.ts), or [`rust.ts`](https://github.com/activeloopai/hivemind/blob/main/rust.ts)). These files export a `parseWithChunks` helper that instantiates a Tree-Sitter `Parser` and walks the syntax tree to emit **GraphNode** and **GraphEdge** objects. The parser operates as a singleton per grammar, managed by the shared utilities in [`src/graph/extract/shared.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/extract/shared.ts).

### Chunked Parsing Strategy

To avoid memory issues with large source files, the module processes code in **16 KB chunks**. This approach prevents the O(N²) blow-up that can occur with the native Tree-Sitter parser on very large inputs. The `parseWithChunks` function in [`src/graph/extract/shared.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/extract/shared.ts) handles the chunking logic, feeding each segment to the parser and aggregating the resulting nodes and edges.

### Snapshot Assembly

After extraction, [`src/graph/snapshot.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/snapshot.ts) collects all `GraphNode` and `GraphEdge` objects from every language extractor. It adds metadata including `observation` timestamps and `graph.commit_sha`, then writes the resulting JSON document to `<repo-root>/snapshots/<sha>.json`. The [`src/graph/last-build.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/last-build.ts) and [`src/graph/build-lock.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/build-lock.ts) modules record the latest build SHA, enabling the VFS to locate the correct snapshot later.

## Querying the Graph via the VFS Interface

Once built, the snapshot is exposed through a read-only virtual filesystem mounted at `~/.deeplake/memory/graph/`. The [`src/graph/vfs-handler.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/vfs-handler.ts) file implements the `handleGraphVfs` dispatcher, which loads snapshots via `loadSnapshotOrError` and routes requests to appropriate renderers.

### Available VFS Endpoints

The dispatcher supports several query patterns. Each endpoint returns a formatted string or a fallback object with `kind: "no-graph"` if the snapshot is missing.

- **[`index.md`](https://github.com/activeloopai/hivemind/blob/main/index.md)** – Returns a high-level summary of the current snapshot.
- **`find/<pattern>`** – Performs case-insensitive substring searches on node IDs and labels, storing a handle map for subsequent `show` commands.
- **`show/<key>`** – Resolves a handle (digit) or pattern to a single node, displaying its details and one-hop neighbors.
- **`query/<pattern>`** – Combines find and show operations, returning the top 5 matches.
- **`neighborhood/<file>`** – Lists symbols defined in a specific file and their cross-file neighbors.
- **`layers`**, **`tour`**, **`path/<from>/<to>`** – Provide architectural visualizations, dependency-ordered walkthroughs, or shortest-path lookups.

### Neighborhood Rendering

The `neighborhood/<file>` endpoint delegates to `renderNeighborhood` in [`src/graph/render/neighborhood.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/render/neighborhood.ts). This pure function takes a `GraphSnapshot` and returns a human-readable view of symbols and their relationships. All renderers in `src/graph/render/*` follow this pattern, operating on the snapshot without throwing exceptions.

## Programmatic Usage Examples

You can interact with the graph module directly from Node.js code without using the CLI.

### Direct VFS Invocation from Node

Call `handleGraphVfs` to query the graph programmatically:

```typescript
import { handleGraphVfs } from "hivemind/src/graph/vfs-handler.js";

const cwd = "/path/to/your/project";

// Query the neighborhood of a specific file
const result = handleGraphVfs("neighborhood/src/graph/render/neighborhood.ts", cwd);

if (result.kind === "ok") {
  console.log(result.body);          // Formatted list of symbols + cross-file neighbors
} else {
  console.error(result.message);    // "no-graph" or "not-found"
}

```

This function loads the latest snapshot and delegates to the appropriate renderer.

### Symbol Search and Resolution

Use the find endpoint to locate symbols, then resolve specific matches:

```typescript
import { handleGraphVfs } from "hivemind/src/graph/vfs-handler.js";

const cwd = "/path/to/repo";
const findResult = handleGraphVfs("find/GraphNode", cwd);

if (findResult.kind === "ok") {
  console.log(findResult.body);   // List of matching node IDs with handles [1], [2]...
  
  // Retrieve the first match using its handle
  const show = handleGraphVfs("show/1", cwd);
  console.log(show.body);
}

```

### Single-File Parsing

To parse a single file without building the entire graph, use the language-specific extractors directly:

```typescript
import { parseWithChunks } from "hivemind/src/graph/extract/typescript.js";
import { readFileSync } from "node:fs";

const source = readFileSync("src/graph/vfs-handler.ts", "utf8");
const result = parseWithChunks(source, "src/graph/vfs-handler.ts");

console.log(`Parsed ${result.nodes.length} symbols, ${result.links.length} edges`);

```

All language extractors share the same signature (`parseWithChunks(source, relativePath)`) and return an object containing `nodes`, `links`, and `parse_errors`.

## Summary

- The **Hivemind graph module** uses Tree-Sitter parsers in `src/graph/extract/*.ts` to process source code in 16 KB chunks, avoiding memory issues with large files.
- Parsed data is assembled into a JSON snapshot by [`src/graph/snapshot.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/snapshot.ts) and stored under `<repo-root>/snapshots/<sha>.json`.
- The **VFS interface** in [`src/graph/vfs-handler.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/vfs-handler.ts) exposes this data through a virtual filesystem at `~/.deeplake/memory/graph/`, supporting endpoints like `find/`, `show/`, and `neighborhood/`.
- All renderers are pure functions that operate on the `GraphSnapshot` type, returning formatted strings or safe fallback messages.
- You can parse data programmatically using `handleGraphVfs` for queries or `parseWithChunks` for single-file processing.

## Frequently Asked Questions

### How does the graph module handle large files without running out of memory?

The module implements a chunked parsing strategy in [`src/graph/extract/shared.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/extract/shared.ts) that feeds source code to Tree-Sitter in **16 KB increments**. This prevents the O(N²) memory blow-up that can occur when parsing very large files, allowing the system to process files of any size efficiently.

### What query patterns does the VFS support for exploring code relationships?

The VFS dispatcher in [`src/graph/vfs-handler.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/vfs-handler.ts) supports multiple endpoints including `find/<pattern>` for substring searches, `show/<key>` for detailed node views, `neighborhood/<file>` for cross-file symbol relationships, and `path/<from>/<to>` for shortest-path analysis. Each endpoint returns human-readable text suitable for AI agent consumption.

### Can I parse a single source file without building the entire repository graph?

Yes. Each language extractor in `src/graph/extract/*.ts` exports a `parseWithChunks` function that accepts a source string and relative path, returning a `GraphSnapshot` object containing `nodes` and `links`. This allows you to extract symbols from individual files without triggering the full snapshot build process.

### Where are graph snapshots stored and how does the VFS locate them?

Snapshots are written as JSON files to `<repo-root>/snapshots/<sha>.json` by the snapshot assembler. The [`src/graph/last-build.ts`](https://github.com/activeloopai/hivemind/blob/main/src/graph/last-build.ts) module records the most recent commit SHA, and the VFS uses this record to locate and load the correct snapshot via `loadSnapshotOrError` when processing queries.