HugeGraph AI LiteLLM Integration: Complete Guide to Supported LLM Providers

HugeGraph AI supports every LLM provider compatible with LiteLLM—including OpenAI, Anthropic Claude, Google Gemini, Azure OpenAI, and local Ollama servers—by using a standard <provider>/<model> naming convention without modifying core code.

The apache/incubator-hugegraph-ai repository implements a flexible HugeGraph AI LiteLLM integration that enables seamless switching between commercial cloud APIs and local language models. This integration leverages the LiteLLM library as a unified router, allowing developers to access dozens of providers through a single client interface defined in the source tree.

How HugeGraph AI LiteLLM Integration Works

The integration centers on the LiteLLMClient class defined in hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py. This wrapper implements standard methods including generate(), agenerate(), and streaming capabilities while forwarding all requests to the underlying LiteLLM library.

When you instantiate the client, you specify the model using the format <provider>/<model-name>. For example, anthropic/claude-3-sonnet-20240229 or openai/gpt-4.1-mini. The client passes this string directly to LiteLLM, which handles provider-specific API routing, authentication, and request formatting automatically.

Supported LLM Providers

Because HugeGraph AI delegates provider resolution to LiteLLM, the system supports any provider listed in the LiteLLM documentation. Key categories include:

Cloud API Providers

• OpenAI (openai): GPT-4, GPT-3.5-turbo, and legacy models
• Anthropic (anthropic): Claude 3 Opus, Sonnet, and Haiku variants
• Google (google): Gemini Pro and Ultra models
• Cohere (cohere): Command and Embed models
• Mistral AI (mistralai): Mistral Large and Medium

Enterprise and Specialized

• Azure OpenAI (azure): GPT models hosted on Azure infrastructure
• AWS Bedrock (bedrock): Amazon's managed LLM service
• Together AI (togetherai): Inference for open-source models
• Groq (groq): High-speed inference for Llama and Mixtral
• DeepInfra, Fireworks, SambaNova, Perplexity, AI21: Specialized inference providers

Local and Self-Hosted

• Ollama (ollama): Local inference for Llama, Mistral, and other open weights
• LiteLLM Local (litellm/local): Custom local deployments via compatible APIs

Configuring LiteLLM in HugeGraph AI

Programmatic Configuration

Import the LiteLLMClient directly from the models module to instantiate specific providers:

from hugegraph_llm.models.llms.litellm import LiteLLMClient

# Example: Anthropic Claude configuration

llm = LiteLLMClient(
    api_key="YOUR_ANTHROPIC_API_KEY",
    model_name="anthropic/claude-3-sonnet-20240229",
    max_tokens=4096,
    temperature=0.7,
)

response = llm.generate(prompt="Explain graph database indexing strategies.")
print(response)

To switch providers, change only the model_name and api_key parameters. For local servers like Ollama, add the api_base parameter pointing to your local endpoint.

UI Configuration

The demo interface in hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py provides a visual configuration panel:

1. Open the "Set up the LLM" accordion in the RAG demo UI.
2. Select litellm from the provider dropdown.
3. Configure the required fields:
   • api_key: Authentication token for your chosen provider
   • api_base (optional): Custom endpoint URL for Azure, Ollama, or proxy servers
   • model_name: Provider prefix and model identifier (e.g., groq/llama3-70b-8192)
   • max_token: Context window limit for the specific model
4. Click "Apply configuration" to save settings.

The UI displays a link to the official LiteLLM providers documentation for reference.

Key Implementation Files

Understanding the source structure helps with debugging and extending the integration:

• hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py: Contains the LiteLLMClient class implementing generate(), agenerate(), and token counting methods.
• hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py: Factory class that instantiates chat, extract, and text-to-GQL LLMs based on configuration parameters chat_llm_type, extract_llm_type, and text2gql_llm_type.
• hugegraph-llm/src/hugegraph_llm/demo/rag_demo/configs_block.py: Gradio UI component handling provider selection and credential input fields (lines 55-61).
• hugegraph-llm/README.md: Documentation referencing LiteLLM multi-provider capabilities (lines 84-85).

Summary

• HugeGraph AI LiteLLM integration supports any provider compatible with the LiteLLM library through a unified interface.
• Use the <provider>/<model> naming convention (e.g., openai/gpt-4, anthropic/claude-3-opus-20240229) to specify models.
• The LiteLLMClient class in litellm.py handles all provider-specific routing without requiring code modifications.
• Configuration works both programmatically and through the Gradio demo UI.
• Local deployments via Ollama or custom endpoints are supported using the api_base parameter.

Frequently Asked Questions

What is the format for specifying LLM models in HugeGraph AI LiteLLM integration?

HugeGraph AI uses the standard LiteLLM convention of <provider>/<model>. For example, use openai/gpt-4o-mini for OpenAI's GPT-4o Mini or google/gemini-pro for Google's Gemini Pro. The provider prefix tells LiteLLM which API endpoint and authentication headers to use.

Can I use local LLMs with HugeGraph AI through LiteLLM?

Yes. Local models running on Ollama or similar servers are fully supported. Specify ollama/llama3 as the model name and set the api_base parameter to your local endpoint (typically http://localhost:11434). This configuration requires no internet connectivity for inference.

Do I need to modify source code to add a new LLM provider?

No. Because the LiteLLMClient dynamically routes requests based on the model name string, any new provider added to the LiteLLM library automatically becomes available in HugeGraph AI. Simply update your configuration with the new provider prefix and valid API credentials.

Where is the LiteLLM client implementation located in the HugeGraph AI repository?

The core implementation resides in hugegraph-llm/src/hugegraph_llm/models/llms/litellm.py. This file defines the LiteLLMClient class that wraps LiteLLM's completion methods. The factory pattern used to instantiate this client is located in hugegraph-llm/src/hugegraph_llm/models/llms/init_llm.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}