# Requirements for Using Vision Models with NextChat: Complete Setup Guide

> Discover the requirements for using Vision models with NextChat. Get your API key, install Node.js 18, and configure your model name for seamless integration. Start building today!

- Repository: [NextChat/NextChat](https://github.com/ChatGPTNextWeb/NextChat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**To use Vision models with NextChat, you need a valid provider API key, Node.js ≥18, and a model name that matches the built-in vision regex patterns or is explicitly listed in the `VISION_MODELS` environment variable.**

NextChat (the open-source project formerly known as ChatGPT-Next-Web) supports multimodal interactions that combine images with text when you configure a vision-capable model. The application automatically detects vision models using regex-based matching and processes image content before sending requests to providers like OpenAI, Anthropic, or Google.

## Prerequisites for Vision Model Support

### Runtime and API Requirements

Before enabling vision capabilities, ensure your deployment meets these baseline requirements:

- **Node.js ≥18**: The NextChat codebase requires this runtime version for proper async handling and image processing.
- **Provider API key**: You must set the appropriate environment variable for your chosen provider—`OPENAI_API_KEY` for OpenAI, `GOOGLE_API_KEY` for Gemini, `ANTHROPIC_API_KEY` for Claude, or equivalent keys for other supported services.
- **Valid model access**: Your API key must have access to vision-capable endpoints (e.g., GPT-4 Vision, Claude 3, Gemini 1.5 Pro).

### Environment Variables

Several environment variables control vision model behavior in [`app/config/server.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/config/server.ts):

- **`VISION_MODELS`**: Optional comma-separated list of custom model names to treat as vision models (e.g., `VISION_MODELS=gpt-4-vision,my-custom-model`).
- **`DISABLE_GPT4`**: If set to `1` or `true`, GPT-4 models (including vision variants) are hidden from the UI. Ensure this is unset or `0` when using GPT-4 vision.

The server exposes these configurations via the `/api/config` endpoint, which the client accesses through `useAccessStore.getState().visionModels`.

## How NextChat Detects Vision Models

### Built-in Regex Patterns (VISION_MODEL_REGEXES)

NextChat maintains a comprehensive list of regex patterns in [`app/constant.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/constant.ts) (lines 78-96) to identify vision-capable models automatically:

```typescript
export const VISION_MODEL_REGEXES = [
  /vision/,
  /gpt-4o/,
  /gpt-4\.1/,
  /claude.*[34]/,
  /gemini-1\.5/,
  /gemini-exp/,
  /gemini-2\.[05]/,
  /learnlm/,
  /qwen-vl/,
  /qwen2-vl/,
  /gpt-4-turbo(?!.*preview)/,
  /^dall-e-3$/,
  /glm-4v/,
  /vl/i,
  /o3/,
  /o4-mini/,
  /grok-4/i,
  /gpt-5/
];

```

These patterns cover mainstream vision models including OpenAI's GPT-4 Vision and GPT-4o, Anthropic's Claude 3/4 family, Google's Gemini series, Alibaba's Qwen-VL, and others.

### Exclusion Logic (EXCLUDE_VISION_MODEL_REGEXES)

The system also maintains exclusion patterns to override false positives. In [`app/constant.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/constant.ts) (lines 99-100):

```typescript
export const EXCLUDE_VISION_MODEL_REGEXES = [/claude-3-5-haiku-20241022/];

```

Any model matching these exclusion patterns is forced to **not** be treated as a vision model, even if it matches a positive pattern.

### The isVisionModel() Function

The core detection logic resides in [`app/utils.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/utils.ts) (lines 283-293):

```typescript
export function isVisionModel(model: string) {
  const visionModels = useAccessStore.getState().visionModels;
  const envVisionModels = visionModels?.split(",").map((m) => m.trim());
  if (envVisionModels?.includes(model)) {
    return true;
  }
  return (
    !EXCLUDE_VISION_MODEL_REGEXES.some((regex) => regex.test(model)) &&
    VISION_MODEL_REGEXES.some((regex) => regex.test(model))
  );
}

```

This function checks the user-supplied `VISION_MODELS` list first, then falls back to regex matching while respecting exclusions. The behavior is verified by the test suite in [`test/vision-model-checker.test.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/test/vision-model-checker.test.ts) (lines 16-66).

## Configuring Custom Vision Models

### Using the VISION_MODELS Environment Variable

For models that don't match the built-in regex patterns, add them to your `.env.local` or deployment environment:

```bash
VISION_MODELS=gpt-4-vision,claude-3-opus,my-custom-vision

```

When `isVisionModel()` runs, it splits this string by commas and trims whitespace to create an array of valid vision model names. This allows immediate use of new vision models without waiting for code updates or regex additions.

### Server-Side Configuration

In [`app/config/server.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/config/server.ts) (lines 139-146), the `getServerSideConfig()` function reads `process.env.VISION_MODELS` and returns it as `visionModels` in the configuration object:

```typescript
// From app/config/server.ts
visionModels: process.env.VISION_MODELS,

```

The client retrieves this configuration during initialization, making the custom list available to the `isVisionModel()` utility throughout the application lifecycle.

## Using Vision Models in Practice

### Message Payload Structure

When `isVisionModel()` returns true, NextChat triggers `preProcessImageContent()` to handle image preprocessing. For OpenAI-compatible providers in [`app/client/platforms/openai.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/client/platforms/openai.ts) (lines 19-27), the request payload structure becomes:

```typescript
const requestPayload = {
  model: "gpt-4-vision",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Describe this picture." },
        { type: "image_url", image_url: { url: "data:image/png;base64,..." } }
      ]
    }
  ],
  // Vision models automatically receive higher token limits
  max_tokens: Math.max(modelConfig.max_tokens, 4000),
  temperature: modelConfig.temperature,
  stream: true
};

```

The preprocessing converts image URLs to base64 strings or provider-specific formats before the request leaves the client.

### Provider-Specific Handling

Different providers implement vision support through distinct code paths:

- **OpenAI**: Uses standard `image_url` objects with automatic `max_tokens` adjustment to 4000 minimum.
- **Anthropic**: Shares the OpenAI preprocessing path; handles Claude 3/4 vision models with specific streaming restrictions for certain variants.
- **Google**: Processes Gemini 1.5 Pro and Vision models in [`app/client/platforms/google.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/client/platforms/google.ts) with base64 encoding.
- **Alibaba**: Invokes `preProcessImageContentForAlibabaDashScope` for Qwen-VL models (`qwen-vl-plus`, `qwen-vl-max`).

## Summary

- **Detection relies on regex matching** against `VISION_MODEL_REGEXES` in [`app/constant.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/constant.ts), with exclusions defined in `EXCLUDE_VISION_MODEL_REGEXES`.
- **Custom models** require adding their names to the `VISION_MODELS` environment variable, exposed via [`app/config/server.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/config/server.ts).
- **Runtime requirements** include Node.js ≥18 and valid provider API keys (`OPENAI_API_KEY`, `GOOGLE_API_KEY`, etc.).
- **Image preprocessing** occurs automatically when `isVisionModel()` identifies a vision-capable model, converting `image_url` entries to appropriate formats in [`app/client/platforms/openai.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/client/platforms/openai.ts).
- **Testing coverage** exists in [`test/vision-model-checker.test.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/test/vision-model-checker.test.ts) to validate detection logic for new model releases.

## Frequently Asked Questions

### What models are supported as vision models in NextChat?

NextChat automatically recognizes models matching patterns like `gpt-4-vision`, `gpt-4o`, `claude-3/4` variants, `gemini-1.5`, `qwen-vl`, `dall-e-3`, and others defined in `VISION_MODEL_REGEXES` within [`app/constant.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/constant.ts). You can verify specific model support by checking the test file [`test/vision-model-checker.test.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/test/vision-model-checker.test.ts) or adding custom models via the `VISION_MODELS` environment variable.

### How do I add a custom vision model that isn't detected automatically?

Add the exact model name to your environment configuration as `VISION_MODELS=your-model-name`. The server reads this in [`app/config/server.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/config/server.ts) and exposes it to the client, where `isVisionModel()` in [`app/utils.ts`](https://github.com/ChatGPTNextWeb/NextChat/blob/main/app/utils.ts) checks this list before applying regex matching. Separate multiple custom models with commas.

### Why are my images not being processed even though I'm using a vision model?

First, confirm your model name matches a regex in `VISION_MODEL_REGEXES` or appears in `VISION_MODELS`. Check that `DISABLE_GPT4` is not set to `1` if using GPT-4 vision variants. Finally, ensure your messages actually contain `image_url` objects in the content array—NextChat only preprocesses images when this specific structure is present in the user message.

### Does NextChat support vision for all LLM providers?

NextChat implements vision support for major providers including OpenAI, Anthropic, Google (Gemini), and Alibaba (Qwen) through dedicated platform files in `app/client/platforms/`. However, the specific preprocessing logic varies by provider—OpenAI uses standard `image_url` objects, while Google and Alibaba require provider-specific formatting handled by their respective platform implementations.