ModLens Integration Requirements for Host Applications: A Complete Guide

Host applications must provide a single image input, configure a supported vision provider via CLI flags or ~/.modlens/config.json, and invoke the ModLens CLI as a subprocess to receive JSON-structured evidence that conforms to the schema defined in src/schema.ts.

ModLens is a CLI-driven vision layer from the liustack/modlens repository that converts images into structured JSON evidence for non-vision LLM workflows. When embedding ModLens into host applications such as browser extensions, IDE plugins, or automation scripts, developers must satisfy specific architectural requirements around input handling, provider configuration, and schema validation to ensure reliable visual-to-text processing.

Core ModLens Integration Requirements

Image Input Source Handling

Host applications must supply a single image to ModLens using either the -i <path> flag for local files or a reachable remote URL. According to the source implementation in src/main.ts, ModLens reads the image exactly once during command execution, converting visual content into parseable text evidence before passing results downstream.

Vision Provider Configuration

ModLens supports five vision providers: antigravity-cli, claude-cli, kimi-cli, gemini-api, and openai-compatible. Configuration resolution follows a strict hierarchy: CLI flags take precedence, followed by values in ~/.modlens/config.json, and finally built-in defaults defined in src/config.ts. For API-based providers requiring authentication, credentials are sourced from the configuration file or corresponding environment variables including GEMINI_API_KEY, OPENAI_API_KEY, and ANTHROPIC_API_KEY.

Model Metadata Awareness

When implementing "paste takeover" functionality, host applications must query the provider registry defined in src/providers/index.ts to verify the selected model’s declared inputModalities. Only models declaring image support trigger the visual-parsing path; pure-text models should fall back to native paste flows to prevent accidental image handling for text-only models.

Network Proxy Support

Hosts operating behind HTTP proxies must forward proxy settings via the HTTPS_PROXY or HTTP_PROXY environment variables, or using the --proxy CLI flag. The implementation utilizes the undici library in src/main.ts to ensure consistent fetch behavior for both proxied and direct downloads.

Runtime Permissions and Security

Host environments must grant ModLens read access to local image paths and permission for outbound HTTP requests when processing remote URLs. The security model explicitly ensures no secret values are written to logs, requiring hosts to manage credential exposure carefully.

CLI Invocation and Exit Codes

Integration requires invoking the ModLens CLI (modlens …) as a subprocess and capturing stdout and stderr streams. Exit codes follow standard conventions where 0 indicates success and non-zero values signal errors. For API-based providers, successful execution returns a JSON payload that strictly conforms to the schema defined in src/schema.ts.

Schema Enforcement

All provider outputs must validate against the canonical vision result schema located in src/schema.ts. Host applications can rely on this deterministic JSON structure for downstream processing; any deviation from the schema results in a hard error, ensuring type-safe evidence handling.

Optional Paste Recovery

For hosts requiring session history access, the modlens recover-paste command enables recovery of images from LLM sessions. This integration point requires access to the host's session storage (such as Claude Code or OpenCode SQLite databases) and appropriate filesystem permissions to read those files, as documented in docs/harness-setup.md.

Implementing ModLens Integration: Code Examples

Node.js Host Application

The following example demonstrates spawning ModLens as a child process with environment-based authentication and proxy support:

const { spawn } = require('child_process');
const path = require('path');

// 1️⃣ Ensure provider config exists (e.g., ~/.modlens/config.json)
//    or set env vars for the chosen provider.
process.env.OPENAI_API_KEY = 'sk-…';               // <-- placeholder only

// 2️⃣ Prepare the image input (local file)
const imagePath = path.resolve(__dirname, 'screenshot.png');

// 3️⃣ Invoke the CLI
const modlens = spawn('modlens', ['-i', imagePath, '-p', 'openaiCompat']);

let stdout = '';
let stderr = '';

modlens.stdout.on('data', data => (stdout += data));
modlens.stderr.on('data', data => (stderr += data));

modlens.on('close', code => {
  if (code !== 0) {
    console.error('Mod Lens failed:', stderr);
    return;
  }
  // 4️⃣ Parse the JSON evidence
  const evidence = JSON.parse(stdout);
  console.log('Vision evidence:', evidence);
  // … forward `evidence` to your LLM workflow …
});

Bash Integration for Browser Extensions

For browser extension backends or shell scripts requiring proxy configuration:

# Example: Bash script for a browser extension backend

#   - Uses environment variables for proxy and API key

export HTTPS_PROXY="http://proxy.local:3128"
export GEMINI_API_KEY="…your‑key…"

#   - Run Mod Lens on a remote image URL

modlens -i https://example.com/image.jpg -p gemini-api --json-schema

Key Source Files for Integration

Understanding these core files helps developers debug integration issues and extend functionality:

• src/main.ts: CLI entry point that parses flags and dispatches to providers
• src/providers/index.ts: Provider registry and interface definitions for the five supported providers
• src/schema.ts: Canonical JSON schema for vision output validation
• src/config.ts: Layered configuration loader implementing the CLI → config file → built-in resolution chain
• src/util/extraBody.ts: Utility for merging vendor-specific parameters into API requests
• docs/harness-setup.md: Documentation for host model metadata exposure and paste-takeover logic
• docs/cli.md: Complete CLI reference and flag documentation

Summary

• Hosts must provide single image inputs via local paths (-i) or URLs
• Provider credentials resolve through CLI flags, ~/.modlens/config.json, or environment variables (GEMINI_API_KEY, OPENAI_API_KEY, etc.)
• Model metadata checks via inputModalities prevent incorrect image routing to text-only models
• Proxy support requires HTTPS_PROXY/HTTP_PROXY env vars or the --proxy flag using undici
• Zero-exit codes indicate success with JSON output conforming to src/schema.ts
• Optional paste recovery requires session storage access and the recover-paste subcommand

Frequently Asked Questions

What are the ModLens integration requirements for API authentication?

Host applications must ensure API keys are available through environment variables (GEMINI_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY) or the ~/.modlens/config.json file. ModLens reads these credentials during provider initialization in src/config.ts without logging sensitive values to ensure secure credential handling.

How does ModLens handle proxy configurations in enterprise environments?

Enterprise hosts can configure ModLens using standard HTTPS_PROXY or HTTP_PROXY environment variables, or explicitly via the --proxy CLI flag. The implementation uses the undici library to maintain consistent fetch behavior across proxy and direct connections, ensuring reliable remote image downloads.

What JSON schema does ModLens use for vision evidence output?

All ModLens providers output JSON that validates against the schema defined in src/schema.ts. This schema-enforced structure ensures host applications receive predictable fields for downstream LLM processing, with validation errors thrown for any non-conforming responses to maintain type safety.

Which vision providers can host applications configure in ModLens?

ModLens supports five providers: antigravity-cli, claude-cli, kimi-cli, gemini-api, and openai-compatible. Each provider is registered in src/providers/index.ts and accepts specific vendor parameters through the extraBody utility in src/util/extraBody.ts, allowing flexible API request customization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}