Configuring LLM Providers (Anthropic, Google, Ollama) for Code-Graph-RAG Orchestration

Set environment variables in ORCHESTRATOR_* and CYPHER_* namespaces, then let codebase_rag/config.py validate and instantiate the appropriate provider from anthropic, google, or ollama.

Code‑Graph‑RAG from vitali87/code-graph-rag is a flexible retrieval‑augmented generation system that lets you plug in any LLM following the pydantic‑ai contract. The orchestrator—the model that generates Cypher queries, prompts, and reasoning logic—is driven by a small, declarative configuration layer. This guide explains how to configure Anthropic, Google (GLA or Vertex), and Ollama providers for both orchestration and Cypher generation.

---

Provider Selection via Environment Variables

All provider configuration flows through the Pydantic settings class in codebase_rag/config.py【/codebase_rag/config.py†L166-L173】. The system recognizes these variable namespaces:

Variable	Purpose	Example Values
ORCHESTRATOR_PROVIDER	Provider for the main reasoning LLM	anthropic, google, ollama
ORCHESTRATOR_MODEL	Model identifier	claude-3-5-sonnet, gemini-2.5-pro, llama3.2
ORCHESTRATOR_API_KEY	API key for cloud providers	sk-ant-..., AIza...
ORCHESTRATOR_ENDPOINT	Base URL for local/custom endpoints	http://localhost:11434/v1
ORCHESTRATOR_PROVIDER_TYPE	Google subtype (gla or vertex)	gla, vertex
ORCHESTRATOR_PROJECT_ID / ORCHESTRATOR_REGION	Required for Vertex AI	my-project, us-central1
ORCHESTRATOR_SERVICE_ACCOUNT_FILE	GCP service account path	/path/to/sa.json

A parallel CYPHER_* namespace exists for the Cypher‑generation model. The validator enforces required fields per provider—for example, Anthropic demands an API key while Vertex requires a project ID.

---

Provider Implementations in providers/base.py

The concrete provider classes inherit from the abstract ModelProvider in codebase_rag/providers/base.py. Each implements create_model() to return a pydantic‑ai compatible instance.

AnthropicProvider

AnthropicProvider wraps pydantic_ai.models.anthropic.AnthropicModel and requires api_key. It constructs the model with AnthropicModelSettings【/codebase_rag/providers/base.py†L190-L219】.

# Required: ORCHESTRATOR_API_KEY

ORCHESTRATOR_PROVIDER=anthropic
ORCHESTRATOR_MODEL=claude-3-5-sonnet

GoogleProvider

GoogleProvider supports two sub‑types controlled by GoogleProviderType defined in codebase_rag/constants/providers.py【/codebase_rag/constants/providers.py†L36-L45】:

• gla – Google LLM API via PydanticGoogleProvider
• vertex – Vertex AI via GoogleCloudProvider

Both paths inject GoogleModelSettings【/codebase_rag/providers/base.py†L70-L125】.

# Google LLM API (GLA)

ORCHESTRATOR_PROVIDER=google
ORCHESTRATOR_PROVIDER_TYPE=gla
ORCHESTRATOR_MODEL=gemini-2.5-pro
ORCHESTRATOR_API_KEY=AIza...

# Vertex AI

ORCHESTRATOR_PROVIDER=google
ORCHESTRATOR_PROVIDER_TYPE=vertex
ORCHESTRATOR_MODEL=gemini-2.5-pro
ORCHESTRATOR_PROJECT_ID=my-project
ORCHESTRATOR_REGION=us-central1
ORCHESTRATOR_SERVICE_ACCOUNT_FILE=/path/to/sa.json

OllamaProvider

OllamaProvider is a lightweight local provider. The endpoint defaults to settings.ollama_endpoint (see the property in config.py【/codebase_rag/config.py†L191-L197】) but can be overridden. On initialization, it validates reachability via check_ollama_running【/codebase_rag/providers/base.py†L168-L178】【/codebase_rag/providers/base.py†L354-L357】.

ORCHESTRATOR_PROVIDER=ollama
ORCHESTRATOR_MODEL=llama3.2
ORCHESTRATOR_ENDPOINT=http://localhost:11434/v1

All providers are registered in a mapping so get_provider() resolves strings like "anthropic" or "ollama" to the correct class【/codebase_rag/providers/base.py†L302-L304】.

---

Orchestrator Wiring in main.py

During startup, codebase_rag/main.py loads the active configuration:

from codebase_rag import settings, cs
orchestrator_cfg = settings.active_orchestrator_config

It then materializes the LLM:

model = orchestrator_cfg.provider.create_model(
    orchestrator_cfg.model_id,
    endpoint=orchestrator_cfg.endpoint,
)

The resulting model passes to codebase_rag/services/llm.py, which calls model.generate(...) per query. Provider initialization failures raise LLMGenerationError with the original exception preserved【/codebase_rag/services/llm.py†L182-L190】.

---

Mixing Providers for Cost Optimization

Code‑Graph‑RAG allows different providers for orchestrator and Cypher. A typical hybrid setup pairs a capable cloud model for reasoning with a local Ollama instance for cheap Cypher generation:

ORCHESTRATOR_PROVIDER=google
ORCHESTRATOR_MODEL=gemini-2.5-pro
ORCHESTRATOR_API_KEY=YOUR_GOOGLE_KEY
ORCHESTRATOR_PROVIDER_TYPE=gla

CYPHER_PROVIDER=ollama
CYPHER_MODEL=llama3.2
CYPHER_ENDPOINT=http://localhost:11434/v1

The test suite tests/test_provider_configuration.py validates these mixed configurations【/codebase_rag/tests/test_provider_configuration.py†L17-L38】.

---

Programmatic Configuration

Bypass environment variables by constructing ModelConfig directly and invoking _update_single_model_setting:

from codebase_rag.config import ModelConfig, cs
from codebase_rag.main import _update_single_model_setting

# Anthropic orchestrator

_update_single_model_setting(
    cs.ModelRole.ORCHESTRATOR,
    ModelConfig(
        provider="anthropic",
        model_id="claude-3-5-sonnet",
        api_key="sk-ant-test-key"
    )
)

# Ollama for Cypher

_update_single_model_setting(
    cs.ModelRole.CYPHER,
    ModelConfig(
        provider="ollama",
        model_id="llama3.2",
        endpoint="http://localhost:11434/v1"
    )
)

---

Complete Configuration Examples

Minimal Anthropic Setup

ORCHESTRATOR_PROVIDER=anthropic
ORCHESTRATOR_MODEL=claude-3-5-sonnet
ORCHESTRATOR_API_KEY=sk-ant-test-key

python -m codebase_rag.main query "Find all functions that call process_data"

Google Vertex AI

ORCHESTRATOR_PROVIDER=google
ORCHESTRATOR_MODEL=gemini-2.5-pro
ORCHESTRATOR_PROVIDER_TYPE=vertex
ORCHESTRATOR_PROJECT_ID=my-gcp-project
ORCHESTRATOR_REGION=us-central1
ORCHESTRATOR_SERVICE_ACCOUNT_FILE=/path/to/sa.json

from codebase_rag.main import run_query

result = run_query("Show the call graph for DataLoader.load")
print(result)

Ollama-Only Cypher Generation

CYPHER_PROVIDER=ollama
CYPHER_MODEL=llama3.2
CYPHER_ENDPOINT=http://localhost:11434/v1

from codebase_rag.main import run_query

result = run_query("List all tables that store user profiles", cypher_role="cypher")
print(result.cypher)  # generated locally

---

Summary

• Configuration hub: codebase_rag/config.py defines ORCHESTRATOR_* and CYPHER_* variables with Pydantic validation【/codebase_rag/config.py†L166-L173】
• Provider implementations: AnthropicProvider, GoogleProvider, and OllamaProvider live in codebase_rag/providers/base.py【/codebase_rag/providers/base.py†L70-L219】
• Google sub‑types: GoogleProviderType enum selects GLA versus Vertex【/codebase_rag/constants/providers.py†L36-L45】
• Mixed deployments: Cloud orchestrator + local Cypher provider validated in tests/test_provider_configuration.py【/codebase_rag/tests/test_provider_configuration.py†L17-L38】
• Programmatic control: ModelConfig + _update_single_model_setting bypasses environment variables entirely

---

Frequently Asked Questions

What happens if I omit ORCHESTRATOR_API_KEY for Anthropic?

The Pydantic validator in codebase_rag/config.py raises a validation error at startup because AnthropicProvider.create_model() requires the key to instantiate AnthropicModel【/codebase_rag/providers/base.py†L190-L219】.

Can I use the same Ollama endpoint for both orchestrator and Cypher?

Yes. Set both ORCHESTRATOR_PROVIDER=ollama and CYPHER_PROVIDER=ollama with identical or separate *_ENDPOINT values. The provider performs reachability checks via check_ollama_running【/codebase_rag/providers/base.py†L354-L357】.

How do I switch between Google GLA and Vertex?

Set ORCHESTRATOR_PROVIDER_TYPE=gla for the standard Google LLM API (API key only) or vertex for Vertex AI (requires project ID, region, and optionally a service account file)【/codebase_rag/constants/providers.py†L36-L45】.

Is there a default fallback provider?

No explicit default is hard-coded. If environment variables are absent, the Pydantic settings class raises validation errors. For testing, the internal test suite uses explicit ModelConfig construction to avoid environment dependencies【/codebase_rag/tests/test_provider_configuration.py†L17-L38】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}