# How to Configure PrivateGPT with Different LLM Providers: Ollama, OpenAI, Azure, SageMaker, and Gemini

> Configure PrivateGPT with Ollama OpenAI Azure SageMaker Gemini by setting llm.mode. Learn to integrate custom LLMs easily for enhanced local AI interactions.

- Repository: [Zylon/private-gpt](https://github.com/zylon-ai/private-gpt)
- Tags: how-to-guide
- Published: 2026-03-06

---

**Configure PrivateGPT with different LLM providers by setting the `llm.mode` field in a YAML profile and launching with the `PGPT_PROFILES` environment variable.**

PrivateGPT supports multiple large language model backends through a flexible, profile-based configuration system. Whether you want to run models locally with Ollama, use cloud APIs like OpenAI or Google Gemini, or deploy on AWS SageMaker, the repository's settings architecture allows you to switch providers without modifying application code.

## Understanding PrivateGPT's Configuration Architecture

PrivateGPT uses a hierarchical settings system defined in [`private_gpt/settings/settings_loader.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/settings/settings_loader.py). The `SettingsLoader` class merges multiple YAML configuration files into a single settings object at runtime.

The loading process follows this sequence:

1. **Profile Discovery** – The loader builds an `active_profiles` list starting with `default`, then appends any profiles specified in the `PGPT_PROFILES` environment variable (or `test` during test execution).

2. **File Loading** – For each profile, `load_settings_from_profile()` reads either [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml) (for the default profile) or `settings-<profile>.yaml` (for named profiles).

3. **Deep Merge** – The `merge_settings()` function recursively updates dictionaries, allowing later profiles to override values from earlier ones.

4. **Schema Validation** – The merged dictionary instantiates the `Settings` Pydantic model defined in [`private_gpt/settings/settings.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/settings/settings.py), which exposes the `llm.mode` field and provider-specific subsections like `ollama`, `openai`, `azopenai`, `sagemaker`, and `gemini`.

## How LLM Mode Selection Works

The `LLMComponent` class in [`private_gpt/components/llm/llm_component.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/components/llm/llm_component.py) acts as a factory that instantiates the correct LLM client based on the `settings.llm.mode` value.

The component uses a Python match statement to route to the appropriate implementation:

```python
match settings.llm.mode:
    case "ollama":
        # Instantiates Ollama client with settings.ollama parameters

    case "openai":
        # Instantiates OpenAI client with settings.openai parameters

    case "azopenai":
        # Instantiates AzureOpenAI client with settings.azopenai parameters

    case "sagemaker":
        # Instantiates SageMaker client with settings.sagemaker parameters

    case "gemini":
        # Instantiates Gemini client with settings.gemini parameters

    case "mock":
        # Uses MockLLM for testing

```

Each case reads its respective configuration subsection to configure API keys, endpoints, model names, and inference parameters.

## Configuring Ollama for Local Inference

Ollama allows you to run open-source models locally. To configure PrivateGPT with Ollama, create or modify [`settings-ollama.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-ollama.yaml):

```yaml
server:
  env_name: ${APP_ENV:ollama}

llm:
  mode: ollama
  max_new_tokens: 512
  temperature: 0.1

embedding:
  mode: ollama

ollama:
  llm_model: llama3.1
  api_base: http://localhost:11434
  keep_alive: 5m
  autopull_models: true
  tfs_z: 1.0
  top_k: 40
  top_p: 0.9

```

The `autopull_models: true` setting triggers automatic model downloading via the helper in [`private_gpt/utils/ollama.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/utils/ollama.py), which displays a progress bar during the pull operation.

Launch with:

```bash
PGPT_PROFILES=ollama python -m private_gpt

```

## Configuring OpenAI

For OpenAI's GPT models, use [`settings-openai.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-openai.yaml):

```yaml
server:
  env_name: ${APP_ENV:openai}

llm:
  mode: openai

embedding:
  mode: openai

openai:
  api_key: ${OPENAI_API_KEY:}
  model: gpt-4o
  api_base: https://api.openai.com/v1
  request_timeout: 30.0

```

You can override the API key via environment variable:

```bash
export OPENAI_API_KEY=sk-...
PGPT_PROFILES=openai python -m private_gpt

```

## Configuring Azure OpenAI

Azure OpenAI requires additional deployment-specific parameters in [`settings-azopenai.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-azopenai.yaml):

```yaml
server:
  env_name: ${APP_ENV:azopenai}

llm:
  mode: azopenai

embedding:
  mode: azopenai

azopenai:
  api_key: ${AZ_OPENAI_API_KEY:}
  azure_endpoint: https://my-resource.openai.azure.com/
  llm_deployment_name: gpt-35-turbo-deployment
  embedding_deployment_name: ada-embedding-deployment
  api_version: "2023-05-15"
  llm_model: gpt-35-turbo
  embedding_model: text-embedding-ada-002

```

Launch with:

```bash
export AZ_OPENAI_API_KEY=...
PGPT_PROFILES=azopenai python -m private_gpt

```

## Configuring Amazon SageMaker

For AWS SageMaker endpoints, create [`settings-sagemaker.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-sagemaker.yaml):

```yaml
server:
  env_name: ${APP_ENV:sagemaker}
  port: ${PORT:8001}

llm:
  mode: sagemaker

embedding:
  mode: sagemaker

sagemaker:
  llm_endpoint_name: my-llm-endpoint
  embedding_endpoint_name: my-embed-endpoint

```

Ensure your AWS credentials are configured via standard environment variables or the `~/.aws/credentials` file, then run:

```bash
PGPT_PROFILES=sagemaker python -m private_gpt

```

## Configuring Google Gemini

For Google's Gemini models, use [`settings-gemini.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-gemini.yaml):

```yaml
llm:
  mode: gemini

embedding:
  mode: gemini

gemini:
  api_key: ${GOOGLE_API_KEY:}
  model: models/gemini-pro
  embedding_model: models/embedding-001

```

Set your API key and launch:

```bash
export GOOGLE_API_KEY=...
PGPT_PROFILES=gemini python -m private_gpt

```

## Switching Between Providers Using Profiles

The `PGPT_PROFILES` environment variable controls which configuration files are loaded. You can specify multiple profiles separated by commas for inheritance:

```bash

# Load settings.yaml, then settings-dev.yaml, then settings-ollama.yaml

PGPT_PROFILES=dev,ollama python -m private_gpt

```

If you do not set a profile, PrivateGPT uses the default [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml). In this case, you must provide all required configuration via environment variables such as `OPENAI_API_KEY`, `AZ_OPENAI_API_KEY`, or `GOOGLE_API_KEY`.

## Summary

- **Profile-based configuration**: PrivateGPT uses YAML profiles loaded by `SettingsLoader` in [`private_gpt/settings/settings_loader.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/settings/settings_loader.py) and merged according to the `PGPT_PROFILES` environment variable.
- **Mode selection**: The `llm.mode` field in your YAML determines which LLM client is instantiated by `LLMComponent` in [`private_gpt/components/llm/llm_component.py`](https://github.com/zylon-ai/private-gpt/blob/main/private_gpt/components/llm/llm_component.py).
- **Supported providers**: Ollama (`ollama`), OpenAI (`openai`), Azure OpenAI (`azopenai`), AWS SageMaker (`sagemaker`), and Google Gemini (`gemini`).
- **Environment variables**: API keys and secrets should use the `${VAR:default}` syntax in YAML or be exported directly before running `PGPT_PROFILES=<profile> python -m private_gpt`.

## Frequently Asked Questions

### How do I switch between Ollama and OpenAI without editing code?

Set the `PGPT_PROFILES` environment variable to the desired provider name before launching. For example, `PGPT_PROFILES=ollama` loads [`settings-ollama.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-ollama.yaml), while `PGPT_PROFILES=openai` loads [`settings-openai.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings-openai.yaml). The `LLMComponent` automatically instantiates the correct client based on the `llm.mode` value in the loaded configuration.

### Can I use environment variables instead of YAML files for API keys?

Yes. PrivateGPT supports variable interpolation using the `${VAR:default}` syntax in YAML files. You can define `api_key: ${OPENAI_API_KEY:}` in your YAML and export the actual key in your shell: `export OPENAI_API_KEY=sk-...`. If you prefer not to use profiles, you can also set all configuration via environment variables and run with the default [`settings.yaml`](https://github.com/zylon-ai/private-gpt/blob/main/settings.yaml).

### What is the difference between Azure OpenAI and standard OpenAI configuration?

Azure OpenAI requires additional deployment-specific parameters including `azure_endpoint`, `llm_deployment_name`, `embedding_deployment_name`, and `api_version`. While standard OpenAI uses `api_base: https://api.openai.com/v1`, Azure OpenAI uses your custom endpoint URL (e.g., `https://my-resource.openai.azure.com/`). The mode value is also different: use `azopenai` instead of `openai`.

### Does PrivateGPT support running multiple LLM providers simultaneously?

No, PrivateGPT operates with a single active LLM mode at a time as determined by the `llm.mode` configuration field. The `LLMComponent` uses a match statement to instantiate exactly one client implementation. However, you can quickly switch between providers by stopping the server, changing the `PGPT_PROFILES` environment variable, and restarting, or by using different profiles for different deployment environments.