How QMD Handles Large Files During Multi-Get Operations

QMD caps individual file sizes at 10 KB by default during multi-get operations, automatically skipping larger files with a descriptive reason while allowing users to override the limit via CLI flags or MCP parameters.

The tobi/qmd repository implements a multi-get command designed to retrieve multiple documents in a single request without overwhelming system memory or network bandwidth. Understanding how QMD handles large files during these operations is essential for optimizing CLI workflows and MCP (Message-Control-Protocol) integrations.

Default Size Limits and Safety Mechanisms

QMD enforces strict size constraints at the storage layer to prevent multi-get operations from consuming excessive resources.

The 10 KB Default Cap

In src/store.ts at line 51, QMD defines a constant that governs all multi-get size decisions:

const DEFAULT_MULTI_GET_MAX_BYTES = 10 * 1024; // 10 KB

This default applies to both CLI invocations and MCP requests unless explicitly overridden.

How the Size Check Works

When findDocuments processes candidate files, it compares each document's stored body_length against the configured limit. As implemented in src/store.ts lines 85-90:

if (row.body_length > maxBytes) {
  results.push({
    skipped: true,
    skipReason: `File too large (${formatBytes(row.body_length)} > ${formatBytes(maxBytes)}). Use 'qmd get <path>' to retrieve.`,
    doc: { filepath: row.filepath, bodyLength: row.body_length }
  });
  continue;
}

Files exceeding the limit are marked with skipped: true and omitted from the payload, while smaller files proceed with their full content included.

Configuring Multi-Get Limits in QMD

Users can adjust the size threshold through both command-line interfaces and programmatic API calls.

CLI Override with --max-bytes

The CLI entry point in src/qmd.ts exposes a configurable flag at lines 94-96:

.option('--max-bytes <number>', 'Maximum file size in bytes for multi-get (default 10KB)', parseInt)

To retrieve files up to 50 KB:

qmd multi-get notes/**/*.md --max-bytes 51200

When files exceed this custom limit, the CLI prints a warning (lines 22-30 in src/qmd.ts) suggesting the use of qmd get <file> for individual retrieval.

MCP Protocol Configuration

For MCP server integrations, the HTTP endpoint defined in src/mcp.ts accepts a maxBytes parameter at lines 426-436:

app.post('/multi-get', async (req, res) => {
  const { pattern, maxBytes = DEFAULT_MULTI_GET_MAX_BYTES } = req.body;
  const { docs, errors } = await store.findDocuments(db, pattern, { 
    includeBody: true, 
    maxBytes 
  });
  res.json({ results: docs, errors });
});

This allows MCP clients to specify appropriate limits based on their own memory constraints.

Understanding Multi-Get Results and Skip Reasons

QMD provides structured feedback when files are omitted from multi-get responses.

The MultiGetResult Structure

Defined in src/store.ts lines 40-49, the result type distinguishes between retrieved and skipped documents:

interface MultiGetResult {
  skipped: boolean;
  skipReason?: string;
  doc: {
    filepath: string;
    body?: string;
    bodyLength?: number;
  };
}

When skipped is false, the doc.body contains the full file content. When true, skipReason explains the omission.

User Feedback for Skipped Files

The CLI formatter (referenced in src/formatter.ts) processes these results to generate human-readable output. For each skipped file, users see:

Warning: File too large (62KB > 50KB). Use 'qmd get <path>' to retrieve.

This pattern ensures that multi-get operations remain performant while providing clear remediation paths for oversized files.

Practical Examples

Retrieving Files with Increased Limits via CLI

To fetch all Markdown files in a collection while allowing larger documents:

# Allow files up to 50 KB

qmd multi-get projects/**/*.md --max-bytes 51200

Files exceeding 50 KB will be skipped with a warning message directing you to use qmd get <specific-file> instead.

Programmatic Store Access with Custom Limits

When building tools that consume QMD's store directly:

import { createStore } from "./store.js";

const store = createStore("/path/to/index.sqlite");

// Allow files up to 200 KB
const { docs, errors } = store.findDocuments(
  store.getDb(),
  "archives/**/*.pdf",
  { includeBody: true, maxBytes: 200 * 1024 }
);

for (const result of docs) {
  if (result.skipped) {
    console.log(`Skipped: ${result.doc.filepath} – ${result.skipReason}`);
  } else {
    console.log(`Retrieved: ${result.doc.filepath} (${result.doc.body?.length} bytes)`);
  }
}

MCP Server Request with Size Constraints

When interacting with QMD via its MCP HTTP interface:

curl -X POST http://localhost:8181/multi-get \
  -H "Content-Type: application/json" \
  -d '{"pattern":"logs/**/*.md","maxBytes":65536}'

The response will include skipped: true entries for any log files exceeding 64 KB, each with a skipReason explaining the size constraint.

Summary

• QMD enforces a 10 KB default limit per file during multi-get operations to prevent memory and bandwidth exhaustion.
• Size checks occur in src/store.ts where body_length is compared against maxBytes, with oversized files marked as skipped.
• Users can override limits via the --max-bytes CLI flag or the maxBytes parameter in MCP requests.
• Skipped files return structured feedback through the MultiGetResult interface, including human-readable reasons and remediation suggestions.
• Individual retrieval remains available for skipped files using the standard qmd get <path> command.

Frequently Asked Questions

What happens if a file exceeds the multi-get size limit?

When a file exceeds the configured maxBytes limit (default 10 KB), QMD includes it in the results with skipped: true and a skipReason field explaining the size constraint. The file content is omitted from the response, but the CLI prints a warning suggesting the use of qmd get <filepath> to retrieve the full document individually.

How do I increase the file size limit for multi-get operations?

You can increase the limit using the --max-bytes option in the CLI or the maxBytes field in MCP requests. For example, run qmd multi-get 'docs/**/*.md' --max-bytes 51200 to allow files up to 50 KB. When calling the store programmatically, pass maxBytes: 200 * 1024 in the options object to findDocuments.

Where is the size check logic implemented in the QMD source code?

The core size validation logic resides in src/store.ts within the findDocuments function. At lines 85-90, the code compares each document's body_length against the maxBytes parameter. If the file is too large, it constructs a MultiGetResult with skipped: true and an appropriate skipReason before continuing to the next candidate file.

Can I retrieve files that were skipped in a multi-get operation?

Yes, files skipped due to size limits can be retrieved individually using the standard qmd get <filepath> command. The skip reason message explicitly suggests this approach: "Use 'qmd get ' to retrieve." This design allows multi-get operations to remain fast and memory-efficient while still providing access to large files through targeted single-document requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}