How to Configure and Use Local Open-Source Models Like WizardCoder with gpt-engineer

To configure and use local open-source models like WizardCoder with gpt-engineer, set the OPENAI_API_BASE environment variable to point at a local llama-cpp-python server, export LOCAL_MODEL=true, and run the CLI with the --lite flag.

gpt-engineer is an open-source AI coding assistant that generates entire software projects from natural language prompts. While it defaults to OpenAI's remote API, the codebase is designed to let you configure and use local open-source models such as WizardCoder by exposing them through an OpenAI-compatible HTTP endpoint, enabling fully offline, zero-cost code generation.

Architecture Overview

gpt-engineer communicates with language models via standard HTTP requests to an OpenAI-compatible /v1/chat/completions endpoint. This design allows seamless swapping between remote APIs and local inference servers without modifying any Python code.

Key components in the repository handle this routing:

• gpt_engineer/applications/cli/main.py – Detects local mode by checking os.getenv("LOCAL_MODEL") (lines 552-556). When this variable is present, the CLI reports "Total api cost: $ 0.0 since we are using local LLM" instead of calculating token costs.
• gpt_engineer/core/ai.py – Constructs the HTTP client using OPENAI_API_BASE, OPENAI_API_KEY, and MODEL_NAME environment variables to determine where to send requests.
• docs/open_models.md – Provides the official user guide for running models like WizardCoder, CodeLlama, or Mixtral locally.

Because the client only requires a valid base URL and model name, you can route traffic to a local GPU or CPU server by changing environment variables alone.

Step-by-Step Setup for WizardCoder

1. Install the Inference Backend

Install llama-cpp-python with the optional server extras to expose a local HTTP API:

pip install llama-cpp-python
pip install 'llama-cpp-python[server]'

2. Download GGUF Model Weights

Download a GGUF-quantized version of WizardCoder from Hugging Face (e.g., TheBloke/WizardCoder-7B-V1.0-GGUF). Save the file (e.g., wizardcoder-7b-v1.0.Q4_K_M.gguf) and note its absolute path as $MODEL_PATH.

3. Start the Local Server

Launch the server with hardware-specific optimizations. Adjust --n_gpu_layers based on your VRAM and --n_batch for throughput:

export MODEL_PATH=/path/to/wizardcoder-7b-v1.0.Q4_K_M.gguf

python -m llama_cpp.server \
    --model $MODEL_PATH \
    --n_batch 256 \
    --n_gpu_layers 30

The server listens on http://127.0.0.1:8000/v1 and implements the OpenAI chat completions protocol required by gpt-engineer.

4. Configure gpt-engineer Environment Variables

In a separate terminal, export the variables that redirect gpt-engineer to your local endpoint:

export OPENAI_API_BASE="http://127.0.0.1:8000/v1"
export OPENAI_API_KEY="sk-local"  # any non-empty string; the local server ignores it

export MODEL_NAME="WizardCoder"
export LOCAL_MODEL=true

Setting LOCAL_MODEL=true is critical—it triggers the cost-free reporting logic in the CLI and ensures the tool handles local model limitations gracefully.

5. Run the gpt-engineer CLI

Execute your project with the --lite flag, which is recommended for local models because they handle very long instruction lists less gracefully than GPT-4:

gpte projects/example WizardCoder --lite --temperature 0.1

The tool will read the prompt file from projects/example, send it to your local WizardCoder instance, and generate the codebase while reporting zero API costs.

Optimizing Local Inference Performance

For faster generation speeds, compile llama-cpp-python with hardware-specific acceleration before installing:

• Linux (CUDA/BLAS):

  CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install --force-reinstall llama-cpp-python

• macOS (Metal):

  CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall llama-cpp-python

These flags enable offloading computation to GPUs or Apple Silicon Neural Engines, significantly reducing generation time for large contexts.

Summary

• OpenAI-compatible API: gpt-engineer accepts any endpoint implementing the /v1/chat/completions schema, making it compatible with llama-cpp-python, LocalAI, or LM Studio.
• Zero-code configuration: Routing to WizardCoder requires only environment variables (OPENAI_API_BASE, LOCAL_MODEL, MODEL_NAME) without touching source code in gpt_engineer/core/ai.py.
• Cost reporting: The CLI automatically reports $0.0 cost when LOCAL_MODEL is detected in gpt_engineer/applications/cli/main.py.
• Performance flags: Use --lite for local models to avoid context-length issues, and compile with CMAKE_ARGS for GPU/Metal acceleration.
• Security: Your code and prompts never leave the local machine when using this setup.

Frequently Asked Questions

What file formats are supported for local models?

gpt-engineer itself does not load model weights; it relies on an external inference server. When using the recommended llama-cpp-python backend, you must provide models in GGUF format (formerly GGML). WizardCoder, CodeLlama, and Mistral variants are widely available on Hugging Face in this quantized format.

Why does the cost show $0.0 when using local models?

In gpt_engineer/applications/cli/main.py (lines 552-556), the CLI checks os.getenv("LOCAL_MODEL"). If this variable exists, the application bypasses token counting and prints a fixed message indicating zero cost, since no external API billing occurs.

Do I need a real OpenAI API key for local models?

No. The OPENAI_API_KEY environment variable must be set to any non-empty string (e.g., sk-local) because the HTTP client requires it, but the local llama-cpp-python server ignores the value entirely. Authentication is handled at the server level, not by gpt-engineer.

How do I troubleshoot connection errors to the local server?

First, verify the server is running and accessible by testing curl http://127.0.0.1:8000/v1/models. If that succeeds, ensure OPENAI_API_BASE ends with /v1 (not /v1/ or the root path) and that LOCAL_MODEL is exported in the same shell where you invoke gpte. The application routes requests through gpt_engineer/core/ai.py, which constructs the full URL from these variables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}