How Lemon AI Integrates with Ollama and VLLM for Local LLM Support

Lemon AI treats Ollama and VLLM as PRIVATE channel providers, routing requests through a generic OpenAI-compatible client that forwards HTTP calls to local endpoints like http://localhost:11434/v1 without requiring API keys.

The hexdocom/lemonai repository implements a unified provider architecture that enables seamless Lemon AI Ollama VLLM integration for self-hosted language models. By abstracting every LLM behind a standardized provider string and channel selection logic, the system eliminates the complexity of managing different local inference engines separately. This approach allows developers to switch between commercial APIs and local instances without changing application code.

Provider String Architecture and Model Resolution

Every LLM request begins with a provider string formatted as provider#<platform_name>#<model_name>. In src/utils/llm.js, line 27 constructs this identifier:

const model = `provider#${model_info.platform_name}#${model_info.model_name}`;

The createLLMInstance function then passes this string to the factory in src/completion/llm.one.js. Lines 58-66 parse the identifier into three components: channel, service, and model. For platforms that do not require commercial API authentication, the system selects the PRIVATE channel, triggering a configuration-based instantiation rather than a vendor-specific implementation.

PRIVATE Channel Configuration for Local Models

When the channel resolves to PRIVATE, Lemon AI bypasses proprietary SDKs in favor of generic OpenAI-compatible HTTP clients. The resolveServiceConfig function in src/completion/resolveServiceConfig.js (lines 3-9) retrieves platform settings from the JSON data store, returning an object containing api_url and optional api_key fields.

This configuration drives the ConfigLLM class defined in src/completion/llm.config.js (lines 5-12), which extends the base functionality from src/completion/llm.base.js. Because both Ollama and VLLM expose endpoints compatible with OpenAI's /v1/chat/completions schema, the same ConfigLLM implementation handles request serialization and token streaming for both platforms.

Ollama Integration Implementation

Ollama ships with a predefined entry in public/default_data/default_platform.json (lines 168-175), configuring the default api_url as http://localhost:11434/v1 with an empty API key. The UI component in frontend/src/view/setting/model.vue (lines 84-86) detects this platform name to hide the API-key input field, streamlining the setup experience.

When executing a chat completion, src/completion/llm.base.js (lines 75-104) sends a POST request to the configured endpoint using standard OpenAI message formatting. This allows Ollama's native API compatibility layer to handle model inference without custom adapters.

VLLM Integration Setup

Unlike Ollama, VLLM requires manual platform registration but uses the same underlying infrastructure. Users add a platform entry with provider_type: "OpenAI" and specify the VLLM server URL (typically http://localhost:8000/v1) in the api_url field.

The system treats this identically to the Ollama flow: src/completion/llm.one.js routes it through the PRIVATE channel, resolveServiceConfig loads the endpoint configuration, and ConfigLLM manages the HTTP communication. This design means VLLM support requires no additional code changes beyond the platform configuration JSON.

Practical Code Examples

To invoke an Ollama model from server code, use the high-level helper in src/utils/llm.js:

const response = await call(
  "Explain the difference between LLM and AGI",
  conversationId,
  "assistant",
  { temperature: 0.7, stream: true }
);

This internally constructs a provider#Ollama#llama2 identifier and routes the request through the PRIVATE channel pipeline.

For direct instantiation bypassing the factory, import the generic client:

const ConfigLLM = require('@src/completion/llm.config');

const ollama = new ConfigLLM(
  { url: 'http://localhost:11434/v1', model: 'llama2', api_key: '' },
  chunk => process.stdout.write(chunk)
);

ollama.completion('Write a short poem about AI', {}, { temperature: 0.5 })
  .then(console.log)
  .catch(console.error);

To register VLLM via configuration payload:

{
  "name": "VLLM",
  "provider_type": "OpenAI",
  "api_url": "http://localhost:8000/v1",
  "api_key": "",
  "is_enabled": "True"
}

Summary

• Lemon AI uses a provider string format (provider#platform#model) to abstract all LLM integrations in src/utils/llm.js.
• PRIVATE channel selection in src/completion/llm.one.js triggers generic OpenAI-compatible clients for local models.
• The ConfigLLM class in src/completion/llm.config.js handles HTTP requests to Ollama and VLLM endpoints using standard /v1/chat/completions schemas.
• Ollama comes pre-configured in public/default_data/default_platform.json with UI support hiding API-key fields in frontend/src/view/setting/model.vue.
• VLLM integrates through the same code path by registering a custom platform with a local api_url.

Frequently Asked Questions

Does Lemon AI require an API key for Ollama or VLLM?

No. Local LLM integrations use the PRIVATE channel, which expects an empty or optional api_key field. The UI automatically hides the API-key input when Ollama is selected, and the ConfigLLM class does not inject authentication headers when the key is absent.

What file handles the routing decision between commercial and local LLMs?

src/completion/llm.one.js contains the factory logic that parses the provider string and selects the channel. Lines 81-90 specifically check for channel === CHANNEL.PRIVATE to instantiate the generic ConfigLLM instead of vendor-specific implementations.

Can I use the same code to call both Ollama and OpenAI models?

Yes. The call() function in src/utils/llm.js accepts any provider string, whether pointing to OpenAI's commercial API or a local Ollama instance. The architecture ensures that application code remains agnostic to the underlying inference engine, with differences confined to the platform configuration and channel selection logic.

How do I add a custom VLLM server to Lemon AI?

Create a platform entry in your configuration store with provider_type set to "OpenAI" and your server's URL in the api_url field (e.g., http://localhost:8000/v1). The resolveServiceConfig function will load this entry, and llm.one.js will route requests through the PRIVATE channel to your VLLM endpoint using the standard OpenAI HTTP format.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}