What LLM Models Can I Use with Cognee? A Complete Provider Guide

Cognee supports nine LLM providers including OpenAI, Anthropic, Google Gemini, Mistral, AWS Bedrock, Ollama, Llama-CPP, and custom HTTP endpoints, configurable via environment variables in LLMConfig.

Cognee abstracts Large Language Model access behind a unified provider-model pair architecture. According to the topoteretes/cognee source code, the framework routes all LLM calls through LLMGateway while allowing you to swap between cloud APIs and local inference engines without changing application code.

Supported LLM Providers in Cognee

The LLMProvider enum in cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/get_llm_client.py defines nine supported providers:

• OpenAI – Cloud models including GPT-5-mini, GPT-4o, and GPT-4
• Anthropic – Claude 3 Opus, Sonnet, and Haiku models
• Ollama – Local models via Ollama server (Llama 3, Mistral, etc.)
• Gemini – Google Gemini 1.5 Pro and Flash
• Mistral – Mistral Large and open models
• Bedrock – AWS Bedrock including Claude and Llama 2
• Llama-CPP – Local GGUF models via llama.cpp
• Custom – Any OpenAI-compatible HTTP endpoint
• BAML – Optional structured output framework (install cognee[baml])

How Cognee's LLM Abstraction Works

The architecture centers on three core components that make provider switching transparent.

LLMConfig

cognee/infrastructure/llm/config.py contains the LLMConfig dataclass, which parses environment variables and sets defaults. The default configuration uses openai as the provider and openai/gpt-5-mini as the model.

Provider Routing

cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/get_llm_client.py implements the routing logic. The get_llm_client function maps the LLMProvider enum to specific adapter classes, returning a client that implements acreate_structured_output and create_transcript regardless of the underlying service.

LLMGateway

cognee/infrastructure/llm/LLMGateway.py serves as the public façade. Application code calls LLMGateway.acreate_structured_output() without knowing which provider is configured, enabling seamless provider switching at runtime.

Configuring Your LLM Provider

Cognee reads configuration from environment variables or a .env file at startup via LLMConfig.

Environment Variables

LLM_PROVIDER=openai          # Provider enum value

LLM_MODEL=openai/gpt-4o-mini # Model identifier

LLM_API_KEY=sk-...           # API key for cloud providers

LLM_ENDPOINT=                # Optional custom base URL

Local Inference with Ollama

LLM_PROVIDER=ollama
LLM_MODEL=ollama/llama3
LLM_API_KEY=dummy            # Ollama may not require authentication

AWS Bedrock

LLM_PROVIDER=bedrock
LLM_MODEL=bedrock/anthropic.claude-v2

# AWS credentials handled separately via boto3

Custom Endpoints

LLM_PROVIDER=custom
LLM_MODEL=my-company/awesome-llm
LLM_ENDPOINT=https://api.mycompany.com/v1/completions
LLM_API_KEY=my-secret-token

Code Examples

Generating Structured Output

import asyncio
from cognee.infrastructure.llm.LLMGateway import LLMGateway
from pydantic import BaseModel, Field

class Answer(BaseModel):
    summary: str = Field(..., description="A concise answer")
    confidence: float = Field(..., description="Confidence score 0-1")

async def ask(question: str) -> Answer:
    system_prompt = "You are a helpful assistant. Answer concisely."
    return await LLMGateway.acreate_structured_output(
        text_input=question,
        system_prompt=system_prompt,
        response_model=Answer,
    )

result = asyncio.run(ask("What is the capital of France?"))
print(result.summary)   # → "Paris"

print(result.confidence)  # → 0.99

Switching to Ollama at Runtime

import os
os.environ["LLM_PROVIDER"] = "ollama"
os.environ["LLM_MODEL"] = "ollama/llama3"
os.environ["LLM_API_KEY"] = "dummy"

from cognee.infrastructure.llm.LLMGateway import LLMGateway

async def translate(text: str) -> str:
    return await LLMGateway.acreate_structured_output(
        text_input=text,
        system_prompt="Translate to German, output only the translation.",
        response_model=str,
    )

Using Custom HTTP Endpoints

import os
os.environ.update({
    "LLM_PROVIDER": "custom",
    "LLM_MODEL": "my-company/awesome-llm",
    "LLM_ENDPOINT": "https://api.mycompany.com/v1/completions",
    "LLM_API_KEY": "my-secret-token",
})

from cognee.infrastructure.llm.LLMGateway import LLMGateway

async def simple_chat(message: str) -> str:
    return await LLMGateway.acreate_structured_output(
        text_input=message,
        system_prompt="You are a terse chatbot.",
        response_model=str,
    )

Summary

• Cognee supports nine LLM providers: OpenAI, Anthropic, Ollama, Gemini, Mistral, AWS Bedrock, Llama-CPP, Custom endpoints, and BAML.
• Configuration happens via environment variables parsed by LLMConfig in cognee/infrastructure/llm/config.py.
• The provider-model pair abstraction in get_llm_client.py routes requests to the correct adapter while exposing a unified interface through LLMGateway.
• You can switch providers at runtime by setting LLM_PROVIDER, LLM_MODEL, and LLM_API_KEY before importing LLMGateway.

Frequently Asked Questions

What is the default LLM model in Cognee?

The default configuration uses OpenAI as the provider and GPT-5-mini as the model. This is defined in cognee/infrastructure/llm/config.py at lines 44-45. You can override this by setting the LLM_PROVIDER and LLM_MODEL environment variables before starting your application.

Can I use local models with Cognee?

Yes. Cognee supports Ollama and Llama-CPP for local inference. For Ollama, set LLM_PROVIDER=ollama and LLM_MODEL=ollama/llama3 (or any model installed on your Ollama server). For Llama-CPP, use LLM_PROVIDER=llama_cpp and provide the model path via LLAMA_CPP_MODEL_PATH.

How do I switch LLM providers without changing my code?

Cognee's LLMGateway façade makes provider switching configuration-driven. Simply update your environment variables or .env file with the new LLM_PROVIDER, LLM_MODEL, and LLM_API_KEY values. The get_llm_client function in cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/get_llm_client.py handles the routing to the appropriate adapter automatically.

Does Cognee support AWS Bedrock and other enterprise providers?

Yes. Cognee includes native support for AWS Bedrock via the bedrock provider setting. You can use models like bedrock/anthropic.claude-v2 or bedrock/meta.llama2-13b-chat. Additionally, the custom provider allows integration with any OpenAI-compatible HTTP endpoint, making Cognee compatible with Azure OpenAI, vLLM, and other enterprise deployment patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}