# How ModLens Gives Text-Only LLMs the Ability to Analyze Images

> Discover how ModLens empowers text-only LLMs to analyze images by converting visuals into structured JSON evidence. Learn to read, quote, and reason over visual data.

- Repository: [liustack/modlens](https://github.com/liustack/modlens)
- Tags: how-to-guide
- Published: 2026-08-25

---

**ModLens converts images into structured JSON evidence that text-only LLMs can read, quote, and reason over by using vision-capable providers to process visual inputs and return strictly formatted text output.**

ModLens is an open-source CLI tool that bridges the gap between visual data and text-only large language models. By encapsulating image processing within a provider-agnostic pipeline and normalizing results to a strict JSON schema, ModLens enables any text-based LLM to analyze images without native vision capabilities.

## The Architecture: Converting Images to Structured Text

ModLens operates as an intermediary layer that handles all visual processing internally, exposing only text-based evidence to downstream consumers.

### Input Resolution and Base64 Encoding

The pipeline begins in [`src/main.ts`](https://github.com/liustack/modlens/blob/main/src/main.ts), which parses the `-i`/`--input` flag and determines whether the argument represents a local filesystem path or a remote HTTP URL. This value is handed to [`src/imageInput.ts`](https://github.com/liustack/modlens/blob/main/src/imageInput.ts), which handles the actual byte retrieval.

For local files, the system uses `fs.readFileSync`; for remote URLs, it uses `fetch`. The module determines the MIME type automatically and encodes the raw image data as **base64**. The resulting string attaches to the request payload under the provider-specific image field, formatted as `data:<mime>;base64,<blob>`.

### The Provider Abstraction Layer

All vision backends implement the **`Provider`** contract defined in [`src/providers/index.ts`](https://github.com/liustack/modlens/blob/main/src/providers/index.ts). This interface standardizes how ModLens communicates with diverse vision services.

Five concrete providers exist:
- **Antigravity CLI** (default zero-config subprocess)
- **Claude CLI**
- **Kimi CLI**
- **Gemini API**
- **OpenAI-compatible API**

Each provider implements two critical methods: **`buildInvocation`** (for subprocess providers) or **`execute`** (for in-process API providers) to construct requests, and **`parseOutput`** to handle raw responses. This abstraction allows ModLens to switch between local CLI tools and remote APIs without changing the core analysis logic.

## The Vision Pipeline: From Request to Validated JSON

Once the image is encoded and a provider selected, ModLens constructs a prompt that forces structured output.

### Prompt Construction with Schema Enforcement

The [`src/prompt.ts`](https://github.com/liustack/modlens/blob/main/src/prompt.ts) module assembles the final payload sent to vision models. This prompt includes:
- A task description instructing the model to analyze the image
- The **base64-encoded image** embedded according to provider expectations
- A **JSON schema reference** passed via `--json-schema` (for Antigravity/Claude) or the `responseJsonSchema` field (for Gemini)

The canonical schema lives in [`src/schema.ts`](https://github.com/liustack/modlens/blob/main/src/schema.ts) and defines exact required fields such as `description`, `tags`, and `details`. Providers enforce this schema strictly; any deviation triggers a validation error surfaced directly to the user.

### Execution and Provider Selection

The **analyzer** ([`src/analyzer.ts`](https://github.com/liustack/modlens/blob/main/src/analyzer.ts)) orchestrates the entire flow. It selects the appropriate provider based on CLI flags, user configuration ([`src/config.ts`](https://github.com/liustack/modlens/blob/main/src/config.ts)), and runtime availability checks ([`src/providers/availability.ts`](https://github.com/liustack/modlens/blob/main/src/providers/availability.ts)).

After the provider parses the model's response, the analyzer validates the JSON against [`src/schema.ts`](https://github.com/liustack/modlens/blob/main/src/schema.ts) and stringifies the result. The CLI prints this JSON to STDOUT, meaning any downstream text-only LLM consuming the output receives pure text evidence that it can quote verbatim and reason over without image-processing capabilities.

## Resilience and Rate Limiting

Production usage requires handling API quotas and failures gracefully.

### Cooldown Management for Exhausted Providers

When a provider returns a quota-related error, the **cooldown module** ([`src/cooldown.ts`](https://github.com/liustack/modlens/blob/main/src/cooldown.ts)) records the failure in `~/.modlens/state.json`. Subsequent calls automatically avoid the exhausted provider until the cooldown period expires. This state persistence ensures reliable operation across multiple analysis sessions without manual intervention.

## Practical Usage Examples

Analyze a local screenshot using the default provider:

```bash
modlens -i screenshot.png

```

Process a remote image via the Gemini API (fastest free route):

```bash
modlens -i https://example.com/photo.jpg -p gemini-api

```

Output raw JSON evidence for downstream LLM consumption:

```bash
modlens -i diagram.png --json

```

Programmatic usage within TypeScript projects:

```typescript
import { analyze } from "./analyzer";

const result = await analyze({
  input: "https://example.com/graph.png",
  provider: "anthropic-api",
  json: true,
});

```

## Summary

- **ModLens acts as a vision-to-text bridge** by processing images internally and outputting only structured JSON that text-only LLMs can consume.
- **Provider abstraction** in [`src/providers/index.ts`](https://github.com/liustack/modlens/blob/main/src/providers/index.ts) enables support for five different vision backends (CLI tools and APIs) through a unified interface.
- **Strict schema enforcement** via [`src/schema.ts`](https://github.com/liustack/modlens/blob/main/src/schema.ts) ensures all providers return consistent, predictable JSON structures with fields like `description` and `tags`.
- **Base64 encoding** in [`src/imageInput.ts`](https://github.com/liustack/modlens/blob/main/src/imageInput.ts) handles both local files and remote URLs, making image input source-agnostic.
- **Cooldown tracking** in [`src/cooldown.ts`](https://github.com/liustack/modlens/blob/main/src/cooldown.ts) maintains provider reliability by persisting rate-limit state to `~/.modlens/state.json`.

## Frequently Asked Questions

### How does ModLens handle different image formats?

ModLens automatically detects MIME types during the base64 encoding process in [`src/imageInput.ts`](https://github.com/liustack/modlens/blob/main/src/imageInput.ts). Whether the input is PNG, JPEG, WebP, or other common formats, the system reads the bytes, determines the correct content type, and embeds the data using the standard `data:<mime>;base64,<blob>` format that vision providers expect.

### Can I use ModLens with custom vision models?

Yes, the provider architecture supports custom implementations. By implementing the `Provider` interface in [`src/providers/index.ts`](https://github.com/liustack/modlens/blob/main/src/providers/index.ts)—specifically the `buildInvocation` or `execute` methods for request building and `parseOutput` for response handling—you can integrate any vision-capable service that accepts base64-encoded images and returns JSON.

### What happens when a provider hits a rate limit?

The cooldown system in [`src/cooldown.ts`](https://github.com/liustack/modlens/blob/main/src/cooldown.ts) intercepts quota-related errors and records them in `~/.modlens/state.json`. Subsequent analysis calls automatically skip the exhausted provider and fallback to available alternatives. This state persists across CLI sessions, ensuring automatic recovery without manual configuration changes.

### How does the JSON schema ensure consistent output?

The schema defined in [`src/schema.ts`](https://github.com/liustack/modlens/blob/main/src/schema.ts) specifies mandatory fields and data types that every provider must return. When [`src/analyzer.ts`](https://github.com/liustack/modlens/blob/main/src/analyzer.ts) receives a response, it validates the parsed JSON against this schema. Any missing fields or type mismatches trigger validation errors before the text reaches STDOUT, guaranteeing that downstream text-only LLMs receive well-structured evidence they can reliably parse.