Requirements for Using Vision Models with NextChat: Complete Setup Guide

To use Vision models with NextChat, you need a valid provider API key, Node.js ≥18, and a model name that matches the built-in vision regex patterns or is explicitly listed in the VISION_MODELS environment variable.

NextChat (the open-source project formerly known as ChatGPT-Next-Web) supports multimodal interactions that combine images with text when you configure a vision-capable model. The application automatically detects vision models using regex-based matching and processes image content before sending requests to providers like OpenAI, Anthropic, or Google.

Prerequisites for Vision Model Support

Runtime and API Requirements

Before enabling vision capabilities, ensure your deployment meets these baseline requirements:

• Node.js ≥18: The NextChat codebase requires this runtime version for proper async handling and image processing.
• Provider API key: You must set the appropriate environment variable for your chosen provider—OPENAI_API_KEY for OpenAI, GOOGLE_API_KEY for Gemini, ANTHROPIC_API_KEY for Claude, or equivalent keys for other supported services.
• Valid model access: Your API key must have access to vision-capable endpoints (e.g., GPT-4 Vision, Claude 3, Gemini 1.5 Pro).

Environment Variables

Several environment variables control vision model behavior in app/config/server.ts:

• VISION_MODELS: Optional comma-separated list of custom model names to treat as vision models (e.g., VISION_MODELS=gpt-4-vision,my-custom-model).
• DISABLE_GPT4: If set to 1 or true, GPT-4 models (including vision variants) are hidden from the UI. Ensure this is unset or 0 when using GPT-4 vision.

The server exposes these configurations via the /api/config endpoint, which the client accesses through useAccessStore.getState().visionModels.

How NextChat Detects Vision Models

Built-in Regex Patterns (VISION_MODEL_REGEXES)

NextChat maintains a comprehensive list of regex patterns in app/constant.ts (lines 78-96) to identify vision-capable models automatically:

export const VISION_MODEL_REGEXES = [
  /vision/,
  /gpt-4o/,
  /gpt-4\.1/,
  /claude.*[34]/,
  /gemini-1\.5/,
  /gemini-exp/,
  /gemini-2\.[05]/,
  /learnlm/,
  /qwen-vl/,
  /qwen2-vl/,
  /gpt-4-turbo(?!.*preview)/,
  /^dall-e-3$/,
  /glm-4v/,
  /vl/i,
  /o3/,
  /o4-mini/,
  /grok-4/i,
  /gpt-5/
];

These patterns cover mainstream vision models including OpenAI's GPT-4 Vision and GPT-4o, Anthropic's Claude 3/4 family, Google's Gemini series, Alibaba's Qwen-VL, and others.

Exclusion Logic (EXCLUDE_VISION_MODEL_REGEXES)

The system also maintains exclusion patterns to override false positives. In app/constant.ts (lines 99-100):

export const EXCLUDE_VISION_MODEL_REGEXES = [/claude-3-5-haiku-20241022/];

Any model matching these exclusion patterns is forced to not be treated as a vision model, even if it matches a positive pattern.

The isVisionModel() Function

The core detection logic resides in app/utils.ts (lines 283-293):

export function isVisionModel(model: string) {
  const visionModels = useAccessStore.getState().visionModels;
  const envVisionModels = visionModels?.split(",").map((m) => m.trim());
  if (envVisionModels?.includes(model)) {
    return true;
  }
  return (
    !EXCLUDE_VISION_MODEL_REGEXES.some((regex) => regex.test(model)) &&
    VISION_MODEL_REGEXES.some((regex) => regex.test(model))
  );
}

This function checks the user-supplied VISION_MODELS list first, then falls back to regex matching while respecting exclusions. The behavior is verified by the test suite in test/vision-model-checker.test.ts (lines 16-66).

Configuring Custom Vision Models

Using the VISION_MODELS Environment Variable

For models that don't match the built-in regex patterns, add them to your .env.local or deployment environment:

VISION_MODELS=gpt-4-vision,claude-3-opus,my-custom-vision

When isVisionModel() runs, it splits this string by commas and trims whitespace to create an array of valid vision model names. This allows immediate use of new vision models without waiting for code updates or regex additions.

Server-Side Configuration

In app/config/server.ts (lines 139-146), the getServerSideConfig() function reads process.env.VISION_MODELS and returns it as visionModels in the configuration object:

// From app/config/server.ts
visionModels: process.env.VISION_MODELS,

The client retrieves this configuration during initialization, making the custom list available to the isVisionModel() utility throughout the application lifecycle.

Using Vision Models in Practice

Message Payload Structure

When isVisionModel() returns true, NextChat triggers preProcessImageContent() to handle image preprocessing. For OpenAI-compatible providers in app/client/platforms/openai.ts (lines 19-27), the request payload structure becomes:

const requestPayload = {
  model: "gpt-4-vision",
  messages: [
    {
      role: "user",
      content: [
        { type: "text", text: "Describe this picture." },
        { type: "image_url", image_url: { url: "data:image/png;base64,..." } }
      ]
    }
  ],
  // Vision models automatically receive higher token limits
  max_tokens: Math.max(modelConfig.max_tokens, 4000),
  temperature: modelConfig.temperature,
  stream: true
};

The preprocessing converts image URLs to base64 strings or provider-specific formats before the request leaves the client.

Provider-Specific Handling

Different providers implement vision support through distinct code paths:

• OpenAI: Uses standard image_url objects with automatic max_tokens adjustment to 4000 minimum.
• Anthropic: Shares the OpenAI preprocessing path; handles Claude 3/4 vision models with specific streaming restrictions for certain variants.
• Google: Processes Gemini 1.5 Pro and Vision models in app/client/platforms/google.ts with base64 encoding.
• Alibaba: Invokes preProcessImageContentForAlibabaDashScope for Qwen-VL models (qwen-vl-plus, qwen-vl-max).

Summary

• Detection relies on regex matching against VISION_MODEL_REGEXES in app/constant.ts, with exclusions defined in EXCLUDE_VISION_MODEL_REGEXES.
• Custom models require adding their names to the VISION_MODELS environment variable, exposed via app/config/server.ts.
• Runtime requirements include Node.js ≥18 and valid provider API keys (OPENAI_API_KEY, GOOGLE_API_KEY, etc.).
• Image preprocessing occurs automatically when isVisionModel() identifies a vision-capable model, converting image_url entries to appropriate formats in app/client/platforms/openai.ts.
• Testing coverage exists in test/vision-model-checker.test.ts to validate detection logic for new model releases.

Frequently Asked Questions

What models are supported as vision models in NextChat?

NextChat automatically recognizes models matching patterns like gpt-4-vision, gpt-4o, claude-3/4 variants, gemini-1.5, qwen-vl, dall-e-3, and others defined in VISION_MODEL_REGEXES within app/constant.ts. You can verify specific model support by checking the test file test/vision-model-checker.test.ts or adding custom models via the VISION_MODELS environment variable.

How do I add a custom vision model that isn't detected automatically?

Add the exact model name to your environment configuration as VISION_MODELS=your-model-name. The server reads this in app/config/server.ts and exposes it to the client, where isVisionModel() in app/utils.ts checks this list before applying regex matching. Separate multiple custom models with commas.

Why are my images not being processed even though I'm using a vision model?

First, confirm your model name matches a regex in VISION_MODEL_REGEXES or appears in VISION_MODELS. Check that DISABLE_GPT4 is not set to 1 if using GPT-4 vision variants. Finally, ensure your messages actually contain image_url objects in the content array—NextChat only preprocesses images when this specific structure is present in the user message.

Does NextChat support vision for all LLM providers?

NextChat implements vision support for major providers including OpenAI, Anthropic, Google (Gemini), and Alibaba (Qwen) through dedicated platform files in app/client/platforms/. However, the specific preprocessing logic varies by provider—OpenAI uses standard image_url objects, while Google and Alibaba require provider-specific formatting handled by their respective platform implementations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}