# HugeGraph AI LiteLLM Integration: Complete Guide to Supported LLM Providers

> Explore HugeGraph AI's LiteLLM integration. Discover supported LLM providers like OpenAI, Claude, Gemini, Azure OpenAI, and Ollama. Integrate seamlessly with a standard naming convention.

- Repository: [The Apache Software Foundation/incubator-hugegraph-ai](https://github.com/apache/incubator-hugegraph-ai)
- Tags: how-to-guide
- Published: 2026-02-24

---

**HugeGraph AI supports every LLM provider compatible with LiteLLM—including OpenAI, Anthropic Claude, Google Gemini, Azure OpenAI, and local Ollama servers—by using a standard `<provider>/<model>` naming convention without modifying core code.**

The `apache/incubator-hugegraph-ai` repository implements a flexible HugeGraph AI LiteLLM integration that enables seamless switching between commercial cloud APIs and local language models. This integration leverages the LiteLLM library as a unified router, allowing developers to access dozens of providers through a single client interface defined in the source tree.

## How HugeGraph AI LiteLLM Integration Works

The integration centers on the `LiteLLMClient` class defined in [`hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py). This wrapper implements standard methods including `generate()`, `agenerate()`, and streaming capabilities while forwarding all requests to the underlying LiteLLM library.

When you instantiate the client, you specify the model using the format `<provider>/<model-name>`. For example, `anthropic/claude-3-sonnet-20240229` or `openai/gpt-4.1-mini`. The client passes this string directly to LiteLLM, which handles provider-specific API routing, authentication, and request formatting automatically.

## Supported LLM Providers

Because HugeGraph AI delegates provider resolution to LiteLLM, the system supports any provider listed in the LiteLLM documentation. Key categories include:

**Cloud API Providers**

- **OpenAI** (`openai`): GPT-4, GPT-3.5-turbo, and legacy models
- **Anthropic** (`anthropic`): Claude 3 Opus, Sonnet, and Haiku variants
- **Google** (`google`): Gemini Pro and Ultra models
- **Cohere** (`cohere`): Command and Embed models
- **Mistral AI** (`mistralai`): Mistral Large and Medium

**Enterprise and Specialized**

- **Azure OpenAI** (`azure`): GPT models hosted on Azure infrastructure
- **AWS Bedrock** (`bedrock`): Amazon's managed LLM service
- **Together AI** (`togetherai`): Inference for open-source models
- **Groq** (`groq`): High-speed inference for Llama and Mixtral
- **DeepInfra**, **Fireworks**, **SambaNova**, **Perplexity**, **AI21**: Specialized inference providers

**Local and Self-Hosted**

- **Ollama** (`ollama`): Local inference for Llama, Mistral, and other open weights
- **LiteLLM Local** (`litellm/local`): Custom local deployments via compatible APIs

## Configuring LiteLLM in HugeGraph AI

### Programmatic Configuration

Import the `LiteLLMClient` directly from the models module to instantiate specific providers:

```python
from hugegraph_llm.models.llms.litellm import LiteLLMClient

# Example: Anthropic Claude configuration

llm = LiteLLMClient(
    api_key="YOUR_ANTHROPIC_API_KEY",
    model_name="anthropic/claude-3-sonnet-20240229",
    max_tokens=4096,
    temperature=0.7,
)

response = llm.generate(prompt="Explain graph database indexing strategies.")
print(response)

```

To switch providers, change only the `model_name` and `api_key` parameters. For local servers like Ollama, add the `api_base` parameter pointing to your local endpoint.

### UI Configuration

The demo interface in [`hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py) provides a visual configuration panel:

1. Open the **"Set up the LLM"** accordion in the RAG demo UI.
2. Select **`litellm`** from the provider dropdown.
3. Configure the required fields:
   - `api_key`: Authentication token for your chosen provider
   - `api_base` (optional): Custom endpoint URL for Azure, Ollama, or proxy servers
   - `model_name`: Provider prefix and model identifier (e.g., `groq/llama3-70b-8192`)
   - `max_token`: Context window limit for the specific model
4. Click **"Apply configuration"** to save settings.

The UI displays a link to the official LiteLLM providers documentation for reference.

## Key Implementation Files

Understanding the source structure helps with debugging and extending the integration:

- **[`hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py)**: Contains the `LiteLLMClient` class implementing `generate()`, `agenerate()`, and token counting methods.
- **[`hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py)**: Factory class that instantiates chat, extract, and text-to-GQL LLMs based on configuration parameters `chat_llm_type`, `extract_llm_type`, and `text2gql_llm_type`.
- **[`hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py)**: Gradio UI component handling provider selection and credential input fields (lines 55-61).
- **[`hugegraph-llm/README.md`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/README.md)**: Documentation referencing LiteLLM multi-provider capabilities (lines 84-85).

## Summary

- **HugeGraph AI LiteLLM integration** supports any provider compatible with the LiteLLM library through a unified interface.
- Use the **`<provider>/<model>`** naming convention (e.g., `openai/gpt-4`, `anthropic/claude-3-opus-20240229`) to specify models.
- The `LiteLLMClient` class in [`litellm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/litellm.py) handles all provider-specific routing without requiring code modifications.
- Configuration works both programmatically and through the Gradio demo UI.
- Local deployments via **Ollama** or custom endpoints are supported using the `api_base` parameter.

## Frequently Asked Questions

### What is the format for specifying LLM models in HugeGraph AI LiteLLM integration?

HugeGraph AI uses the standard LiteLLM convention of `<provider>/<model>`. For example, use `openai/gpt-4o-mini` for OpenAI's GPT-4o Mini or `google/gemini-pro` for Google's Gemini Pro. The provider prefix tells LiteLLM which API endpoint and authentication headers to use.

### Can I use local LLMs with HugeGraph AI through LiteLLM?

Yes. Local models running on **Ollama** or similar servers are fully supported. Specify `ollama/llama3` as the model name and set the `api_base` parameter to your local endpoint (typically `http://localhost:11434`). This configuration requires no internet connectivity for inference.

### Do I need to modify source code to add a new LLM provider?

No. Because the `LiteLLMClient` dynamically routes requests based on the model name string, any new provider added to the LiteLLM library automatically becomes available in HugeGraph AI. Simply update your configuration with the new provider prefix and valid API credentials.

### Where is the LiteLLM client implementation located in the HugeGraph AI repository?

The core implementation resides in **[`hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py)**. This file defines the `LiteLLMClient` class that wraps LiteLLM's completion methods. The factory pattern used to instantiate this client is located in **[`hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py`](https://github.com/apache/incubator-hugegraph-ai/blob/main/hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py)**.