# How to Configure and Use Local Open-Source Models Like WizardCoder with gpt-engineer

> Learn to configure and use local open-source models like WizardCoder with gpt-engineer. Set environment variables and run the CLI for seamless local AI coding assistance.

- Repository: [Anton Osika/gpt-engineer](https://github.com/AntonOsika/gpt-engineer)
- Tags: how-to-guide
- Published: 2026-03-06

---

**To configure and use local open-source models like WizardCoder with gpt-engineer, set the `OPENAI_API_BASE` environment variable to point at a local llama-cpp-python server, export `LOCAL_MODEL=true`, and run the CLI with the `--lite` flag.**

gpt-engineer is an open-source AI coding assistant that generates entire software projects from natural language prompts. While it defaults to OpenAI's remote API, the codebase is designed to let you configure and use local open-source models such as WizardCoder by exposing them through an OpenAI-compatible HTTP endpoint, enabling fully offline, zero-cost code generation.

## Architecture Overview

gpt-engineer communicates with language models via standard HTTP requests to an OpenAI-compatible `/v1/chat/completions` endpoint. This design allows seamless swapping between remote APIs and local inference servers without modifying any Python code.

Key components in the repository handle this routing:

- **[`gpt_engineer/applications/cli/main.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/applications/cli/main.py)** – Detects local mode by checking `os.getenv("LOCAL_MODEL")` (lines 552-556). When this variable is present, the CLI reports **"Total api cost: $ 0.0 since we are using local LLM"** instead of calculating token costs.
- **[`gpt_engineer/core/ai.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/core/ai.py)** – Constructs the HTTP client using `OPENAI_API_BASE`, `OPENAI_API_KEY`, and `MODEL_NAME` environment variables to determine where to send requests.
- **[`docs/open_models.md`](https://github.com/AntonOsika/gpt-engineer/blob/main/docs/open_models.md)** – Provides the official user guide for running models like WizardCoder, CodeLlama, or Mixtral locally.

Because the client only requires a valid base URL and model name, you can route traffic to a local GPU or CPU server by changing environment variables alone.

## Step-by-Step Setup for WizardCoder

### 1. Install the Inference Backend

Install `llama-cpp-python` with the optional server extras to expose a local HTTP API:

```bash
pip install llama-cpp-python
pip install 'llama-cpp-python[server]'

```

### 2. Download GGUF Model Weights

Download a GGUF-quantized version of WizardCoder from Hugging Face (e.g., `TheBloke/WizardCoder-7B-V1.0-GGUF`). Save the file (e.g., `wizardcoder-7b-v1.0.Q4_K_M.gguf`) and note its absolute path as `$MODEL_PATH`.

### 3. Start the Local Server

Launch the server with hardware-specific optimizations. Adjust `--n_gpu_layers` based on your VRAM and `--n_batch` for throughput:

```bash
export MODEL_PATH=/path/to/wizardcoder-7b-v1.0.Q4_K_M.gguf

python -m llama_cpp.server \
    --model $MODEL_PATH \
    --n_batch 256 \
    --n_gpu_layers 30

```

The server listens on `http://127.0.0.1:8000/v1` and implements the OpenAI chat completions protocol required by gpt-engineer.

### 4. Configure gpt-engineer Environment Variables

In a separate terminal, export the variables that redirect gpt-engineer to your local endpoint:

```bash
export OPENAI_API_BASE="http://127.0.0.1:8000/v1"
export OPENAI_API_KEY="sk-local"  # any non-empty string; the local server ignores it

export MODEL_NAME="WizardCoder"
export LOCAL_MODEL=true

```

Setting `LOCAL_MODEL=true` is critical—it triggers the cost-free reporting logic in the CLI and ensures the tool handles local model limitations gracefully.

### 5. Run the gpt-engineer CLI

Execute your project with the `--lite` flag, which is recommended for local models because they handle very long instruction lists less gracefully than GPT-4:

```bash
gpte projects/example WizardCoder --lite --temperature 0.1

```

The tool will read the `prompt` file from `projects/example`, send it to your local WizardCoder instance, and generate the codebase while reporting zero API costs.

## Optimizing Local Inference Performance

For faster generation speeds, compile `llama-cpp-python` with hardware-specific acceleration before installing:

- **Linux (CUDA/BLAS):**  
  ```bash
  CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install --force-reinstall llama-cpp-python
  ```

- **macOS (Metal):**  
  ```bash
  CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall llama-cpp-python
  ```

These flags enable offloading computation to GPUs or Apple Silicon Neural Engines, significantly reducing generation time for large contexts.

## Summary

- **OpenAI-compatible API:** gpt-engineer accepts any endpoint implementing the `/v1/chat/completions` schema, making it compatible with `llama-cpp-python`, LocalAI, or LM Studio.
- **Zero-code configuration:** Routing to WizardCoder requires only environment variables (`OPENAI_API_BASE`, `LOCAL_MODEL`, `MODEL_NAME`) without touching source code in [`gpt_engineer/core/ai.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/core/ai.py).
- **Cost reporting:** The CLI automatically reports **$0.0** cost when `LOCAL_MODEL` is detected in [`gpt_engineer/applications/cli/main.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/applications/cli/main.py).
- **Performance flags:** Use `--lite` for local models to avoid context-length issues, and compile with `CMAKE_ARGS` for GPU/Metal acceleration.
- **Security:** Your code and prompts never leave the local machine when using this setup.

## Frequently Asked Questions

### What file formats are supported for local models?

gpt-engineer itself does not load model weights; it relies on an external inference server. When using the recommended `llama-cpp-python` backend, you must provide models in **GGUF** format (formerly GGML). WizardCoder, CodeLlama, and Mistral variants are widely available on Hugging Face in this quantized format.

### Why does the cost show $0.0 when using local models?

In [`gpt_engineer/applications/cli/main.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/applications/cli/main.py) (lines 552-556), the CLI checks `os.getenv("LOCAL_MODEL")`. If this variable exists, the application bypasses token counting and prints a fixed message indicating zero cost, since no external API billing occurs.

### Do I need a real OpenAI API key for local models?

No. The `OPENAI_API_KEY` environment variable must be set to any non-empty string (e.g., `sk-local`) because the HTTP client requires it, but the local `llama-cpp-python` server ignores the value entirely. Authentication is handled at the server level, not by gpt-engineer.

### How do I troubleshoot connection errors to the local server?

First, verify the server is running and accessible by testing `curl http://127.0.0.1:8000/v1/models`. If that succeeds, ensure `OPENAI_API_BASE` ends with `/v1` (not `/v1/` or the root path) and that `LOCAL_MODEL` is exported in the same shell where you invoke `gpte`. The application routes requests through [`gpt_engineer/core/ai.py`](https://github.com/AntonOsika/gpt-engineer/blob/main/gpt_engineer/core/ai.py), which constructs the full URL from these variables.