How to Parse Data Using the Hivemind Graph Module: A Complete Guide

The Hivemind graph module parses source code using Tree-Sitter, builds a JSON snapshot of nodes and edges, and exposes it through a virtual filesystem (VFS) interface for querying symbols and relationships.

The activeloopai/hivemind repository provides a code-understanding engine that transforms source files into queryable graph data. When you parse data using the graph module, it creates a structured representation of symbols and their relationships that can be accessed programmatically or through virtual filesystem calls. This architecture enables AI agents to understand codebase structure without loading the entire repository into context.

How the Graph Module Parses Source Code

The parsing pipeline relies on language-specific Tree-Sitter grammars and a chunked processing strategy to handle large files efficiently.

Language-Specific Extractors

Each supported language has a dedicated extractor in src/graph/extract/*.ts (for example, typescript.ts, python.ts, or rust.ts). These files export a parseWithChunks helper that instantiates a Tree-Sitter Parser and walks the syntax tree to emit GraphNode and GraphEdge objects. The parser operates as a singleton per grammar, managed by the shared utilities in src/graph/extract/shared.ts.

Chunked Parsing Strategy

To avoid memory issues with large source files, the module processes code in 16 KB chunks. This approach prevents the O(N²) blow-up that can occur with the native Tree-Sitter parser on very large inputs. The parseWithChunks function in src/graph/extract/shared.ts handles the chunking logic, feeding each segment to the parser and aggregating the resulting nodes and edges.

Snapshot Assembly

After extraction, src/graph/snapshot.ts collects all GraphNode and GraphEdge objects from every language extractor. It adds metadata including observation timestamps and graph.commit_sha, then writes the resulting JSON document to <repo-root>/snapshots/<sha>.json. The src/graph/last-build.ts and src/graph/build-lock.ts modules record the latest build SHA, enabling the VFS to locate the correct snapshot later.

Querying the Graph via the VFS Interface

Once built, the snapshot is exposed through a read-only virtual filesystem mounted at ~/.deeplake/memory/graph/. The src/graph/vfs-handler.ts file implements the handleGraphVfs dispatcher, which loads snapshots via loadSnapshotOrError and routes requests to appropriate renderers.

Available VFS Endpoints

The dispatcher supports several query patterns. Each endpoint returns a formatted string or a fallback object with kind: "no-graph" if the snapshot is missing.

• index.md – Returns a high-level summary of the current snapshot.
• find/<pattern> – Performs case-insensitive substring searches on node IDs and labels, storing a handle map for subsequent show commands.
• show/<key> – Resolves a handle (digit) or pattern to a single node, displaying its details and one-hop neighbors.
• query/<pattern> – Combines find and show operations, returning the top 5 matches.
• neighborhood/<file> – Lists symbols defined in a specific file and their cross-file neighbors.
• layers, tour, path/<from>/<to> – Provide architectural visualizations, dependency-ordered walkthroughs, or shortest-path lookups.

Neighborhood Rendering

The neighborhood/<file> endpoint delegates to renderNeighborhood in src/graph/render/neighborhood.ts. This pure function takes a GraphSnapshot and returns a human-readable view of symbols and their relationships. All renderers in src/graph/render/* follow this pattern, operating on the snapshot without throwing exceptions.

Programmatic Usage Examples

You can interact with the graph module directly from Node.js code without using the CLI.

Direct VFS Invocation from Node

Call handleGraphVfs to query the graph programmatically:

import { handleGraphVfs } from "hivemind/src/graph/vfs-handler.js";

const cwd = "/path/to/your/project";

// Query the neighborhood of a specific file
const result = handleGraphVfs("neighborhood/src/graph/render/neighborhood.ts", cwd);

if (result.kind === "ok") {
  console.log(result.body);          // Formatted list of symbols + cross-file neighbors
} else {
  console.error(result.message);    // "no-graph" or "not-found"
}

This function loads the latest snapshot and delegates to the appropriate renderer.

Symbol Search and Resolution

Use the find endpoint to locate symbols, then resolve specific matches:

import { handleGraphVfs } from "hivemind/src/graph/vfs-handler.js";

const cwd = "/path/to/repo";
const findResult = handleGraphVfs("find/GraphNode", cwd);

if (findResult.kind === "ok") {
  console.log(findResult.body);   // List of matching node IDs with handles [1], [2]...
  
  // Retrieve the first match using its handle
  const show = handleGraphVfs("show/1", cwd);
  console.log(show.body);
}

Single-File Parsing

To parse a single file without building the entire graph, use the language-specific extractors directly:

import { parseWithChunks } from "hivemind/src/graph/extract/typescript.js";
import { readFileSync } from "node:fs";

const source = readFileSync("src/graph/vfs-handler.ts", "utf8");
const result = parseWithChunks(source, "src/graph/vfs-handler.ts");

console.log(`Parsed ${result.nodes.length} symbols, ${result.links.length} edges`);

All language extractors share the same signature (parseWithChunks(source, relativePath)) and return an object containing nodes, links, and parse_errors.

Summary

• The Hivemind graph module uses Tree-Sitter parsers in src/graph/extract/*.ts to process source code in 16 KB chunks, avoiding memory issues with large files.
• Parsed data is assembled into a JSON snapshot by src/graph/snapshot.ts and stored under <repo-root>/snapshots/<sha>.json.
• The VFS interface in src/graph/vfs-handler.ts exposes this data through a virtual filesystem at ~/.deeplake/memory/graph/, supporting endpoints like find/, show/, and neighborhood/.
• All renderers are pure functions that operate on the GraphSnapshot type, returning formatted strings or safe fallback messages.
• You can parse data programmatically using handleGraphVfs for queries or parseWithChunks for single-file processing.

Frequently Asked Questions

How does the graph module handle large files without running out of memory?

The module implements a chunked parsing strategy in src/graph/extract/shared.ts that feeds source code to Tree-Sitter in 16 KB increments. This prevents the O(N²) memory blow-up that can occur when parsing very large files, allowing the system to process files of any size efficiently.

What query patterns does the VFS support for exploring code relationships?

The VFS dispatcher in src/graph/vfs-handler.ts supports multiple endpoints including find/<pattern> for substring searches, show/<key> for detailed node views, neighborhood/<file> for cross-file symbol relationships, and path/<from>/<to> for shortest-path analysis. Each endpoint returns human-readable text suitable for AI agent consumption.

Can I parse a single source file without building the entire repository graph?

Yes. Each language extractor in src/graph/extract/*.ts exports a parseWithChunks function that accepts a source string and relative path, returning a GraphSnapshot object containing nodes and links. This allows you to extract symbols from individual files without triggering the full snapshot build process.

Where are graph snapshots stored and how does the VFS locate them?

Snapshots are written as JSON files to <repo-root>/snapshots/<sha>.json by the snapshot assembler. The src/graph/last-build.ts module records the most recent commit SHA, and the VFS uses this record to locate and load the correct snapshot via loadSnapshotOrError when processing queries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}